Making things go!

When I started this job, it took me about a week – just under, really – to figure out some ways I could make some very quick and very effective improvements. I’ve gone over some of those in varying details in previous posts, it’s time to detail one of them in particular that I just accomplished.

One of the pain points I identified was the distribution of sysadmin tools. Before I arrived, it had been done by scp’ing a directory to newly deployed servers. I think we can all see the problems with that — data divergence, lack of updates, far too easy to forget to update one or more systems when a given script changes… fun times. I decided Something Had To Be Done. And Quickly.

So I did something. I started out by building a package of those scripts and getting it distributed by our RHN Satellite. That was fairly easy – once I had the package, I just sign and push. Then I started to tackle the whole creating the RPM package bit, which was going to be a wee bit more difficult.

I started out with an empty SVN repository. I couldn’t figure out a clean way of keeping the specfile for the package in with the source tree, so I create two main directories in the repo – packages and specs. The specs directory just has the specfile, nothing more. The packages directory has all the fun stuff. Since I didn’t want packages to bleed through to each other, I then created a new directory for the first package, let’s call it “adminscripts” (no that’s not the actual name I used, I’m sanitizing things as I write).

Inside the adminscripts directory, I established the usual trunk-tags-branches structure so common to SVN projects. This turned out to make things much easier down the line, but I can’t claim any sort of prescience about it – I just did it out of habit and because that’s the way the smart people do things. I’ve got the usual src/ directory off the main project directory, and a Makefile at the top level, so no surprises there. Making commits to the project, updating the source tree, and all that jazz is now “industry-standard” – anyone can start contributing as long as they know how things are done in 90% of open-source projects.

Now comes the first challenge – how do I start with this SVN repository and extract a tar bundle of just the source code? Well, that’s sort of simple, just check out the code and get rid of the “.svn/” directories everywhere, then bundle it up – but I don’t necessarily want to build HEAD. Hmm. Okay, let’s use the tags/ directory and check out a specific tag. This also forces an extra step on the coders to tell the build system that a given revision is ready for packaging, not entirely a bad thing. So we tag it with the release and version we want the RPM package to be, and check out that tag.

Okay, so there’s at lease one important detail – the checkout needs to be renamed after removing the .svn/ directories and before being bundled, since the rpmbuild process expects a directory named %{NAME}-%{VERSION}. That’s just an ‘mv’ command, though.

So now I have a way to get a specific version-release, how do I figure out *which* version-release? Turns out that’s remarkably simple – just parse the specfile with a little “awk”. I think I mentioned in a previous post just how much I love my little friend ‘awk’… anyway. Once I have the bundle, it’s a simple process to move the bundle and specfile into place and launch an rpmbuild job.

But wait… I don’t want to keep rebuilding the same thing every night if there’s no need to. Which means I need to track the builds I’ve done – or at least the ones that have succeeded. I chose to use a PostgreSQL database to do so, though I could have just as easily used any other database – or probably even flat files. I also want to know who to email on build errors – oh and on successes as well, that would be cool – so I throw that into the database.

Without going into too much detail about the database layout, I log which package-version-release combinations are built and when, and also log which emails go with errors and successes for which packages. Then I glue them together with a script that parses the specfile for the “current” version-release of all packages, checks to see if a build has been done, and if not launches the build script.

So basically, in my first month-and-change, I’ve created an end-to-end automated CI build process that goes from source code check-ins to a package ready for signing and distribution. Sure, it’s small scale and systems-oriented rather than application-oriented, but it is a major accomplishment. Plus, it can be easily extended to build applications for deployment – I designed it to be extensible that way. Does it have some limitations? Sure – but for a company this size (~300 employees) in the IT industry (our primary focus is providing web and other IT based services), it’s a pretty hefty addition to the arsenal.

One Response to “Making things go!”