Pains, Trains, and Automobiles

How do you organize your development, so you can release a quality product?

This is a question that’s been plaguing man for centuries — well, okay, just a few decades now.  But the answer to that question has been changing a lot over the past several years, with newer, more powerful tools making it easier to adopt some of the more complicated release strategies.


Just ten years ago, the default answer I would find would still be to have everyone building off the same source control branch (usually, “trunk”!) and the software would be tagged and released when everyone was fairly happy with it.  Source control utilities at the time had started to handle branching more elegantly, but it was still fairly clunky and managing merges often resulted in complete disaster.  I still clearly remember the groans when we would find a bug after a release, and actually end up fixing the bug in both the branch we made for production, and the “trunk” (which wasn’t ready to go out yet).  Merging was such a headache, we’d actually manually apply the fix in both places (if we remembered to do it!) just to avoid dealing with that mess.

Times have changed, the tools have changed, and branching is a lot easier to manage now, particularly with such DVCS tools like git.  But now that branching is easy, the pendulum is starting to swing in the opposite direction — we’re dealing with a glut of branches, and nobody knows where they’re doing their work, which branches are ready for deployment and which aren’t.  Now that we’re no longer at the mercy of what our tools can support, we have actually think about how we want to have our teams do their development.

Two major patterns I’ve seen (if you distill some of the variants down to their core concepts) are based around software “trains”, and then around more continuous integration/deployment ideas (which I think nicely correlates to the ideas of “cars”).


The “train” pattern actually harkens back to the old school release model.  The idea is that you build up a “train” of cars, the train is then treated as a release, and the train goes out.  The next train might be getting built while the first one is in motion, but they will come out one after the other.  How branching is used may differ based on tastes and experience — often a branch will be made to represent that particular release, and developers may be working directly on that branch, or merge in personal branches when they’re ready.  The branch is tested and verified in totality, then merged into master (likely master hasn’t actually changed much in the meantime anyway, so the merge is very simple).  The train goes out, and a new one starts.

The train model has the benefit of fixed feature content — everyone knows what features compose the train, what has to be done before the train leaves the station, and you’re testing the finished product as an integrated solution.  This gives you a lot more confidence in the overall outcome, you can time the deployment with marketing and press releases, and you aren’t actually merging code all that often.

On the other hand, you are creating dependencies on features.  A single feature can hold up the entire train.  If you need to drop a car from the train, you need to rebuild the whole train, extricating the rejected content from the stuff you want to keep.


Since merging is so easy to do now, and continuous integration and deployment such a hot ticket, a lot of places are treating their code like a highway — with individual cars simply merging into traffic as it goes along.  In this model, every developer builds their feature as a separate branch, and their code is tested in isolation.  As soon as it’s validated, those changes get merged into master and the code goes off on its merry way (continuous deployment shops will immediately verify the build with automated testing and then put the code right into production, as soon as it finds changes).  There is no giant build-up of software to test, features can be rolled out regularly, bugs can be fixed almost as soon as they’re reported.

The automobile model is much more agile.  Features are released as soon as they’re ready, so you can start collecting your feedback — and then immediately respond to that feedback with iterations.  If a feature is held up (broken down on the side of the on-ramp), the other cars can just drive around it and still make it out to your production environment without delay.

On the other hand, your product can potentially become fragmented.  You end up testing a feature once in isolation, and then testing it again when it’s combined with everything else.  It’s also ever more important to make sure that features are truly isolated themselves, so when you merge that work in, it stands on its own, independent of whatever else is currently in progress.  If you’re using this model to deal with bugfixes, conflicts will arise if two people (or even the same person) attempts to twiddle the same bit of code two different ways in two different branches (because the same bit of code is responsible for two different symptoms).

So which is optimal?  That’s a very situation-dependent question.  If you’re developing a shrink-wrapped product, where you need to actually produce a “gold master” to ship, your preferred model may be very different from a website, where “distribution” is simply a matter of pushing the new code to a webserver.  If you’re working on code that is mission-critical, you probably will come up with a different answer than a casual-use website (probably opting for a train-based approach).

These models aren’t absolute, either — you can take the car-based approach for developing your train, having developers merge their work into the train branch, using continuous integration to test that train branch as it’s modified, but then basing your deployment strategy on the final results of that train.  Likewise, maybe you use the train model for features, but you’re using the car strategy for bugfixes — ensuring your bugs get fixed ASAP, but then scheduling when large features are released.

Jet Packs

I’m sure there are other variations out there.  Ten years from now, someone will probably be writing another blog entry, talking about how crazy we were to do this whole “branching thing” — when source code control is done by thought, and releases are managed by AI…  (and we’re all commuting to work with our jet packs!)

Paul Clegg

Director of Engineering, CloudForge at CollabNet, Inc.

Tagged with: , , , , , ,
Posted in CloudForge

Leave a Reply

Your email address will not be published. Required fields are marked *




You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

connect with CollabNet
sign up for emails
looking for something

CloudForge: @SmashedCrab_: Our apologies on the trouble you're experiencing. May we have your support ticket # so that we can escalate the issue?
Date: 8 April 2014 | 7:28 pm

CloudForge: Here's a great read to start your week! #CollabNet is included in the @sdtimes: April feature article!
Date: 7 April 2014 | 5:00 pm

CloudForge: @Flayra: We apologize for the inconvenience. May we please have your support ticket number so that we can escalate the issue?
Date: 3 April 2014 | 12:06 am