This is a continuation of Maven 2, Or: How I Learned To Stop Worrying And Love The .pom - a post-mortem on what we expected, and what we got, from our new build system. Caveat: You may not like what you read. I'm being honest, but not necessarily nice about it.
First off - the Maven changeover has more or less ended now. We're done arm-wrestling with various problems, and have reached a stable build system. We're still stuck with the following issues:
- We're still getting multiple test runs, even though we should only technically need one.
- Having upgraded to Continuum 1.0.3, we're finding that while it's more functional, it feels less solid.
- Having finally worked out our dependency hell, we're now fully committed to running a repository of some of the stuff that's just plain badly managed by the central maven repository...
- And having watched our whole build system fall apart on ship day because Maven is dependent on snapshots available only in the maven snapshot server, hosted by Codehaus, which had an outage, we've just moved to maven-proxy.
- Then we had to go through and dig up which versions we were meant to be using when a whole lot of stuff went wrong as bits of what we thought were part of 2.0.4 got updated out from under us into a non-working state.
The maven-proxy is a godsend; if you haven't installed it already, do so - you can't afford not to, and it's just too much of a useful tool to skip. It's reduced more headache in less time than anything so far, and it's a wonder that it isn't included in some kind of “standard” documentation. The unfortunate problem, however, is that the server it's hosted on were the servers that went down: Codehaus. The standard, undocumented response from the mailing lists of 'what to do if a central server goes down', once a server has gone down, is “well, you should have been running maven-proxy”. Of course, the server that went down was the server that hosted the project that might have prevented you from having problems when the server went down. Life is full of little ironies, they say. The whole system depends on the uptime of two central servers; when either goes, your build is gone, unless you remembered to install and make use of a third-party, unconnected project that isn't in the Maven documentation and doesn't get much mention unless there's some huge outage. Nostradamus, I have no doubt, proudly predicted the death of the Codehaus servers and kept a list of the names of those who would, centuries later, predict the need to install an unrelated project to keep their build system running before the Codehaus's untimely death.
In retrospect, all of our problems, and all of the time our project took, was caused by three fundamental flaws in the Maven project: issues with the documentation, or lack thereof; issues with the centralisation of repositories and the dependency on single sites for the provision of a working system; and problems with the “collective” approach that Maven takes towards a 'plugin ecosystem' - or, to put it succinctly, “what the hell does Maven's version number actually mean”.
Problem 1: Documentation
Maven's problems haven't changed; its weakest link is its documentation. There's no way around this, and there's just no excuse. If Maven 2 is to become the world's build system, it needs proper documentation. Whatever process they think they have for managing documentation, however clever, just isn't good enough. If you want community submissions, then put up a system that encourages people to submit - don't assume that people who write documentation are dying to have endless arguments in your mailing list over semantic issues and feel like being a part of your commit repository. I'm not saying go Wiki - I'm saying have a process for writing and managing documentation that isn't about bolting it onto the software development process.
The problem with maven isn't its complexity: it's the complete lack of documentation on how to make that complexity work to your benefit, rather than to your detriment. You spend weeks learning how Maven thinks, why Maven has taken a given action, etc., - only to find that the Maven team ships some backend plugin and the next day your system stops building. What does the mailing list say? “Well, you should have known to hard-code your version numbers for the Maven plugins.” My arse. Where's that written? In the documentation they're hiding from the rest of the world? It's in their heads, one of those hard-learned lessons about the flaws of the system they've written that isn't going to seem obvious to any lay-person learning to use their software. What it really says is Don't Trust Updates - something which seems obvious, sure, but I didn't update my version of Maven; and there's no clear dividing line on which bits are a part of the version-number-I-download vs. “the rest of the soup”.
And to find out which versions I'm using? I've got to dig through tracelogs.
Riddle me this: If it's so damn important that I hardcode all the bits of maven in my POM to ensure my build doesn't break, and this is supposed to be something that everyone who uses Maven 'ought to be doing', why isn't there a report, generated at the time of the build, which lists the plugins it used to perform the build so I can compare it against previous, working builds and, without having to dig through the mailing list, have some vague idea of what went wrong?
Problem 2: Centralisation
It's second weakest link, the one that killed our time-critical deploy, is its centralisation strategy; when it fails to be up, you're stuffed. Running around building your own versions of packages that are just plain buggered on the central servers. Or missing. Lucene is a great example: 2.0 is out, but the version on the servers is out of date; it's out of date because Lucene guys wonder who the hell Maven people are, and why can't they just use the central download system like everyone else. Apache projects that don't get along are a wonder to the outside world - what's the purpose of the Apache organisation if not to encourage people to eat its own dog food? Maven 2 has been out for some time, and there's no excuse for the Apache team, which operates and manages the central repositories, to not sort out its own projects.
If it did, it probably would have resolved quite a few issues by now that affect its users; as is, the most vocal users are the people who are also the project developers. If the Apache developers drank the same kool-aid, we'd be seeing input from real, complex projects that will end up struggling with the same stupid issues everyone else does, but with the added benefit of being close enough to the process to actually do something about being heard.
How is this better than the chaos of having a central server which contains nothing but 'redirects' to the fragmented repositories scattered across the net, run by their software authors? If I'm going to be expected to run local repositories, and repository proxies that cache and obfuscate server downtime, how is this changed by distribution? Centralisation - the idea that it's all simple if it's in one place - has proven itself to be a lie time and time again in this project. POMs are in a state, the Jakarta groups' own teams either don't know about its existence or don't care, management is made overly-complex, and the updates don't happen. Meanwhile, the POMs remain buggered, and the Apache group, due to its restrictive software licensing, will always end up requiring me to run my own repository in order to contain all of the bits I have to download from the Sun website or anyone else that has fundamental licensing differences.
Decentralisation fixes that. I can get it from the Sun repository, because Sun can host it. If I need the JSR170 API, I can get it from Day's repository. Who does the Maven team think they're helping by retaining centralisation like this? If we're all running around making our own repositories of the bits the central system gets wrong, how many wasted hours need to happen before somebody realises that they'd save us all a lot of time by just not bothering. We're having to fend for ourselves in either scenario; the only difference is that the current approach prevents anyone else from helping find a solution to a niche problem they have domain expertise over. Day can't provide just the JSR170 bits without having everyone know about it - even though they're meant to be the authoritative source. Sun can't provide just the Sun bits - even though they're meant to be the authoritative source. If the central system can't manage to be authoritative over the stuff that is open source, and its licenses prevent it from being authoritative over the stuff that isn't open source, then what good is it?
Problem 3: It Doesn't Bloody Work, Does It
Dear Maven Development Team,
Either sort out the testing and packaging so that 2.0.4 works when you download it *and* a month after, or makes it clear which bits fell 'outside' of the standard system and might fail.
Don't tell me WAR packaging and JAR packaging fall outside the system, and don't tell me that I ought to expect that halfway through a ship cycle my working version of 2.0.4 might just stop working. Because that's not a reliable build system. It's also not a reasonable expectation.
If you are going to make that distinction, and tell me that the bits of the build system I think are “built in” aren't, then give me some way of managing the process, document the problem, build it into the documentation as part of the POM construction process, tell us all that we ought to expect having to list every single version every time and that auto-dependency-resolution can't be trusted as a result.
Or here's a shocking alternative: Ensure that 2.0.4's associated war plugin is locked to the 2.0.4 version by default, and doesn't just float free to the latest-and-greatest. Unless the quality of my build system doesn't matter, or it's my responsibility to do version management - because ultimately, those are your selling points, the one you're telling everyone you're solving with Maven. And if I AM supposed to be managing all these versions myself, build the reports that give me the information to let me know what ought to have been locked in stone. I get ginormous dependency graphs on the dependencies of my project, because it's expected that I manage the dependencies of my project. Note I get none on the build system - the inference is that I'm not expected to manage the dependencies of your project. If I am meant to manage the dependencies within the build system itself, start providing the tools for me to do so.
That's not unreasonable.
Moreover, keep in mind that by saying this, you're expecting us all to lock ourselves frozen in time - not because it's inherently dangerous to upgrade, but because the upgrade process itself is uncontrolled. It doesn't get more dangerous than that. So don't take it personally when, forced to supply manual versioning on everything we touch, we look at you with that smoldering flame reflection playing back over our pupils. Because you will have failed to live up to your own rhetoric, and we can rightly feel that you've got no control over what your own 'core' plugins are doing at any point in time - or for that matter, that any mistaken belief that the version numbers that apply to the Maven “thing” we downloaded have any semblance of true meaning other than a surface, minimalist level of coordination which provides almost no assistance to any reasonable real-world project's expected usage. To the real world, the version numbers on Maven mean nothing - they only mean something to the developers of Maven and the core when determining compatibility; to the real world, we're all still in manual-version-land, without any real idea of what version is meant to work with what - and not just what might work.
Lessons Learned: How Not To Marry Your Cousin
When you're building your own build system for your own projects with Maven, it's Caveat Emptor. Here are just a handful of things you can expect to be overheard saying while you're working on your Maven 2 build system:
- “But everything was working yesterday.”
You will find, at any given moment, that what works today fails tomorrow, unless you perform extreme action. You need to plan around this yourself by finding working versions and locking them down. This includes anything that 2.0.4 didn't ship with. Knowing what those things are may take some work. - “What the <insert expletive> is it doing?”
You must be prepared to dig through detailed logs to find the answers to important questions like “why do I have two copies of that library”. Do not expect it to be obvious; it's not designed by people who are casual users, and the only people who see the problems are the people setting up complicated projects - who have the problem once, while they're setting it up. Nobody's focusing on this in the development of Maven, they're focused on lots of other stuff. That makes setup hard; don't allow the mire of setup be enough to throw away the brilliance of the final running system - when it works, it's worth every minute. When it works. You need to do more than you might expect to help that be true, so be prepared to get your hands dirty. - “I can't find where I'm meant to do <insert trivial thing that everyone else seems to do>”
Nothing about the documentation is straightforward; nor is there anywhere central you can expect to find all the information you're looking for, unless where you're looking is in the minds of the people who wrote the software using your Orbital Mind Control Lasers. Trivial stuff, like adding documentation to your site, isn't trivially documented; even though the people likely to be doing it aren't going to want to know this much about your build system. Want to change the look and feel of your site? Prepare for obscurity - I gave up and stole ehcache's from fisheye and modified it to my intent, and even that changed between 2.0.3 and 2.0.4 to the point that I'm now having to reconsider how it was done. Nothing is obvious. Not to any of us. Don't feel like you're the only person who “doesn't get it” - none of us do. Everyone knows it's the single biggest problem, and nobody seems to know what to do about it. - “I can't believe I'm still doing this.”
If you think it's only going to take you a short time to get a build running, you're right. If you think it's only going to take you a short time to be done, you're wrong. In the end, one man spent weeks of time getting a build that worked across our three projects. Project A has 23 listed dependencies; many exist because we can't trust some of the dependencies we pull in from the central repository to say the right thing, or even the sane thing, in some cases. Expect to be in the same boat. Project B has 37 listed dependencies in the pom. Project C has 5 - but includes both project A and B in its list of dependencies, and so pulls in the complete, unified list of jars from those projects - 51 in total. Complain all you like, but managing 51 different dependencies is hard no matter what you're doing, in any build system - so don't expect miracles. - “But we don't want to be dependent on snapshots just to build the system.”
Tough. The underlying system is in enough flux that you're going to end up taking working code from wherever you can get it - and the further off the beaten path you go, the more true it's going to be. From time to time, to solve individual issues that your build environment raises, you're going to end up needing to use a snapshot of something to get the thing built. Ideally, you're not going to be shipping snapshots, but your build will probably end up being dependent on one at some time or another.
You're going to have problems. Be prepared to think creatively. Be prepared to take a route alternate to the mechanism you might consider to be “obvious”. Be prepared for the simple fact that you're still going to have some bubble gum and sticky tape in the system once you're done. Be prepared for the honest truth: it does an awful lot, but not necessarily some of the obvious stuff. Be prepared for the obvious - that no complicated project is going to have a simple build system; and that build systems are best chosen at project start.
Be prepared to spend some time.
Comments (1)
I couldnt agree more with your rant and rave about Maven. I've pretty much mastered m2 after about 2 weeks and using it avidly. Thanks for your column, really enjoyed reading.
Posted by Michael | January 5, 2007 8:37 PM
Posted on January 5, 2007 20:37