I've gone ahead and done the necessary porting to get memcached up and running on Windows. As we use it for our core platform at WhatsOnWhen in our java framework as a 'long term' serialization cache for JDO objects and general data cache, we rely on it heavily - not only for good database performance, but for good performance of the application, as it loads and manages thousands of JDO objects for our location hierarchy and related data. It runs out of process, so restarting Tomcat doesn't affect the cache, and that makes it easier for us to do our development work.
memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
Our live service uses a pool of them - on Tony, George, and John, three linux boxes which provide our application servers known internally as "The Coalition of the Willing". (Our databases, two Win32 terrorists named Osama and a currently non-functioning backup named Saddam, are the "Axis of Merge Replication". Droll, yet obvious.) Clients are available in a wide variety of languages - public APIs exist in Perl, PHP, Python, and Java. Internally, I've built a new Java-based client that does a better job of using larger connection pools and has more built-in parallelism through worker threads in the interests of making it happier in multithreaded environments (eat your heart out, perl/php folks) which I'll release once it's been through a complete test and deploy cycle here, and submit to the memcached clients page. It's a more adventurous client than the existing Java one, is larger, relies on Jakarta object pools and the concurrency library, and handles bulk loading (get_multi) much faster (and more safely) than the existing client (which you have to lock in order to ensure the safety of, as it uses its sockets in a pretty dirty fashion).
For those of you looking for more safety on the existing client, I've also gone through and retrofitted some concurrency locks in the existing client - however, it's important to note that large bulk load operations (we've got one that loads 3,000 objects out of cache pretty much in one request; that can take a while to fulfill) you'll want to have a pool of them handy if you're going to be using them in heavily multithreaded environments - hence the need for the rewrite, IMO, which looks pretty good at the moment.
Our development systems are diverse; we've got Win32-based developers (who historically used a build that was compiled pretty much automagically in Cygwin, I use Mac OS X 10.3, and one of our other developers is running RedHat Fedora. For the most part, for all but the Win32 developers, it worked very well.
That's all by the by. So, I hear you asking, why not just use Cygwin? On two of our systems, the cygwin version was crash-prone, and occasionally took up large amounts of CPU - needlessly, as it pretty much uses no cpu time on the rest of our systems. On one of the systems in question, it seems to be crash-prone enough to kill off the XP kernel - god knows how, as nothing in it does anything especially interesting with the kernel.
Someone has gone through the far more difficult job of porting the core library that memcached relies on, libevent, to Win32. The Win32 port of libevent is required to build this software.
I'll be pushing these changes back to the memcached team for incorporation, and we'll try to get an official build for Win32 out in the next version; at the same time, I might as well pick up the job of doing a dist for Mac OS X for those folks who don't have the compiler tools handy and want a binary, but there's little reason for a distribution for anyone other than Win32 guys, as the rest of the world has decent free compilers that do everything we need. :)
There is, however, one last remaining bug I can't seem to find. I don't know enough about Windows sockets to know what I'm missing, here - but libevent isn't catching the fact that there are events waiting for processing. It just loops, waiting on sockets, and never actually processes the incoming data. There's probably some trick to this - something to do with the way threading works on windows, or the way that sockets are created... anyways, I just can't find it. It compiles, it runs, it probably even works, except for this one last thing I just can't decipher. If anyone can help, please do.
To download the Win32 project, modified source, and MSVC++ 6.0 pre-built version of memcached-1.1.11, which, at the time of writing, is the current version of memcached: memcached-win32-1.1.11.zip (around 7.1 MB; I didn't bother doing a project clean or anything like that.).
Update: Please note that this is long since out of date; much of what's needed has already been prepared and pulled into the mainline codebase, or is maintained elsewhere.
Comments (2)
I've searched around this site a bit, I don't see an email address for you :(. So hopefully you'll get one from this submission. The reason memcached doesnt respond to socket events seems to be a problem with libevent. It plain doesn't work with sockets. Libevent seems to work fine with files/pipes/everything else, just no sockets. At least, thats what I've been encountering on win NT. So, I made some modifications to the win32 libevent library that seem to fix this problem.
Replace in win32.c:
res = WaitForSingleObject(ev->ev_fd, timeval_to_ms(tv));
With:
fd_set fds;
FD_ZERO(&fds);
FD_SET(ev->ev_fd, &fds);
res = select(0, &fds, 0, 0, tv);
if(res == 0) {
Now memcached responds to network events! But it's not out of the woods yet. There are still some problems with the memcache part of it that are preventing a successful transaction. I was hoping that you'd be able to make more progress armed with this information. Anyways, if you're interested in more detail or want my new binary of libevent, you can reach me at teniviki at gmail.com.
Thanks,
Aj
Posted by abommarito![[TypeKey Profile Page]](http://www.ctoforaday.com/nav-commenters.gif)
|
January 4, 2005 8:48 PM
Posted on January 4, 2005 20:48
I forgot to update this with the state of the universe.
1) Cygwin, at the time of compilation, just isn't reliable enough to use as a build of libevent/memcached. That may have changed since - users wishing to use libevent/memcached can try current cygwin to see if it serves their purposes.
2) libevent has moved on, and continues to do so - the Win32 port of libevent is pretty much unsupported, and hasn't been modified since the (broken) 0.7c at http://www.datanerds.net/~mike/libevent.html.
Until there is a proper and maintained port of libevent for win32, there is no point in making the modifications necessary to get memcached supported. It would be greatly appreciated if someone with knowledge and familiarity of Win32 to keep and maintain a port of libevent for Win32; until that happens, I shan't be reviving the win32 port of memcached.
Especially, if for no other reason, because I built this thing under VirtualPC on a mac. I don't run Windows, and I'm not a Windows guy; I'm happy to do the work if a windows guy provides guidance and testing, but I'm not willing to be a permanent maintainer of a port of something I can't use in an environment I can't test and don't want in production, in a dialect of C++ I don't like, with a compiler environment I find to be rubbish, and an OS I find to be just plain awful.
Posted by Gregory Block![[TypeKey Profile Page]](http://www.ctoforaday.com/nav-commenters.gif)
|
April 18, 2005 10:15 AM
Posted on April 18, 2005 10:15