Setting up Hudson on Ubuntu 9.10 with Apache, Django, VirtualEnv, etc.

Hudson is the new testing hotness. It's a continuous integration server, like buildbot, except it doesn't take a ton of configuration. However, it can still take a little while to get your build system fully up and running (or at least it did for me) and I wanted to document how I did it in case other people found it useful.

First off, a bit about our system: we're a Python shop, we use virtualenv, we have a paver task to run our tests, the project is a Django project, and we use django-satprep as our test runner (mmm, nose). We use Postgres as our database, but we also use MongoDB. We also wanted to put this on a server that already had ReviewBoard on it and just have Hudson be a separate endpoint on that server. And, of course, we didn't want any random yahoo coming along and seeing how our builds were going. This is also the same server where we keep our central hg repo and while that makes some things easy, it's not required.

So, let's get going.

Installing Hudson

This was incredibly simple. Hudson has native packages for Ubuntu, so I just followed the instructions there and I was up and running. I could visit http://myserver:8080/ and see Hudson. Wow. Nice. That was it. Done.

Hooking Hudson up to Apache

Now, to get Hudson working at http://myserver/hudson. This took me a while, though it really shouldn't have. The wiki page on setting up Hudson behind Apache worked great for me. Basically, edit /etc/defaults/hudson to add "--prefix=/hudson" to HUDSON_ARGS (at the bottom of that file). I also edited it to run on port 8081 instead of 8080. Then, restart Hudson.

However, here's where the issue was: the restarting of Hudson didn't appear to really... do anything. I saw that the commands in ps aux were with the new information, but it just wouldn't take. After I killed all hudson processes with fire, and restarted Hudson, everything magically worked beautifully. So... that was fun.

At any rate, something like:

ProxyPass         /hudson  http://localhost:8081/hudson
ProxyPassReverse  /hudson  http://localhost:8081/hudson
ProxyRequests     Off

<Proxy http://localhost:8081/hudson*>
    Order deny,allow
    Allow from all
</Proxy>

and an Apache restart later, http://myserver/hudson should welcome you with open arms. Huzzah!

You can now go into your Hudson project, click on 'Manage Hudson', and give it your SMTP settings (if you want email notifications) and give it the final URL it lives at. These are both under 'Configure System'/E-Mail Notification.

Protecting Hudson

Hudson by default is kind of "everybody can do everything!" You can lock it down with user controls, but it still showed quite a bit of information that I didn't really want out there. It seemed like the easiest way to protect it would be to put it up behind basic auth in Apache. This was easily accomplished - in Apache 2.2 you can dd the authorization arguments to a definition, but I still stuck it in a block. The only trick here is that I had already enabled authentication in Hudson and the two conflicted. If you're doing auth with Apache, be sure to disable any security in Hudson beforehand. They don't play nice.

Install Plugins

Hudson comes pretty bare-bones, which is nice. The Plugin Manager is nifty - follow the 'Manage Hudson' link and then 'Manage Plugins'. I recommend the following plugins:

• Bruce Schneier Plugin
• Hudson Mercurial Plugin
• Green Balls

Two are purely cosmetic - the Mecurial plugin is the main one that you need. The install process should work fine, but sometimes seems to just take forever and/or hang. Kicking hudson via /etc/init.d/ will fix it.

Setting up your server

This might be the longest bit, depending on your project needs. I strongly recommend using the same database that you will in production (Postgres, in our case) and make this as true to what you're going to be live with as possible. We manage settings for different environments (development, staging, production, etc.) via different settings files copied to ourproject/local_settings.py, which is then imported at the bottom of our main settings.py file. So, I created a hudson_settings.py that didn't email us on errors, but also used production-like settings. I then tested for a bit in my home directory on that machine to make sure I could check out the project from Mercurial, run our bootstrap.py file to get started (which automatically creates a virtualenv for us, installs libraries, and does a couple of other basic set up tasks), run migrations, etc., and successfully run the tests on that box (if you get failures that aren't related to your setup, great!).

Testing with Hudson

Okay, now joy! We want Hudson to automatically checkout our project, run the bootstrap file, copy over our hudson_settings.py file to local_settings.py, perform any migrations on our main database, and then run our tests. Luckily this is pretty easy to do.

On your Hudson install, click on "New Job".

Give the job a name, then choose "Build a free-style software project."

You'll be taken to the job detail screen. We're doing a Mercurial project, and a local one at that, so once we choose "Mercurial" under the Source Code Management section we enter in the path to the repo on that box. If you don't need a specific branch, just leave it blank.

Next, under "Build", add an "Execute Shell" section. This is going to look something like this:

#!/bin/bash -ex
python bootstrap.py
rm -rf bin lib include
source ./bin/activate
cp myproject/hudson_settings.py myproject/local_settings.py
python myproject/manage.py migrate
paver test --with-xunit

Update: Before I didn't have that rm -rf in there - we want that to wipe out the virtualenv so we do a full bootstrap. This will catch any libraries that have suddenly stopped working. The real solution is to have your own version of pypi.

First off, Hudson uses sh to run these by default, so we put a shebang line in there to force bash (which we need to activate the virtualenv). We copy the hudson_settings over to local_settings, migrate the database (your database should have a decent amount of realish information in there to properly exercise this step), and then test. Our paver task "test" looks like this (in pavement.py):

@task
@consume_args
def test():
    """Run manage.py tests in a more friendly manner."""
    command_list = ['python', '-W', 'ignore::DeprecationWarning', '-W',
        'ignore::SyntaxWarning', 'myproject/manage.py', 'test', '--verbosity=0',
        '--noinput']
    if options.args:
        command_list.append('--')
        command_list.extend(options.args)
    subprocess.check_call(command_list)

As you can see, it's pretty basic - it's just a friendly wrapper around manage.py test, which in turn calls django-satprep's nose test runner. However, we do tell it to do xunit - this information gets output in a nosetests.xml file, which is useful for the next step.

After we set up the script to execute on each build, check the 'Publish JUnit test result report' and enter 'nosetests.xml' in the 'Test report XMLs' section that will appear. Don't worry if it gives you a warning about it not existing.

After that, we activate Bruce Schneier, and E-Mail Notification, telling it to email our whole list if anything goes wrong.

So, great! Wonderful!

Except right now, we have to manually activate the builds. We could have it poll Mercurial every minute or so, but it's really easy to set up a post-push commit hook to do that for us.

Automatic Builds on Push

It turns out that an authenticated HTTP GET to http://myserver/job/JobName/build will start the build process automatically. We just need to write a commit hook to do that for us.

Easy!

Here's an example script, notify_hudson.py (note that we use https, you might need just HTTPConnection):

import httplib

USERNAME = 'myuser'
PASSWORD = 'mypass'

auth_string = ('%s:%s' % (USERNAME, PASSWORD)).encode('base64')[:-1]

def hook(*args, **kwargs):
    h = httplib.HTTPSConnection('myserver')
    headers = {
        'authorization': 'Basic %s' % auth_string,
    }
    h.request("GET", "/hudson/job/JobName/build", headers=headers)
    resp = h.getresponse()

if __name__ == "__main__":
    hook()

Run this to see if your build runs. If it does, then you can hook it up to your repo.

In your repo's .hg/hgrc file, put something like this in:

[hooks]
changegroup.hudson = python:notify_hudson.hook

[extensions]
notify_hudson = /path/to/notify_hudson.py

And you're good to go. When you push upstream, your Hudson build will automatically clone the project, boostrap, run migrations, and run all tests. If something breaks, it'll notify you (and once something is fixed again, it'll do that too). We'll be able to see detailed history of our tests thanks to the junit/xunit integration. It's all awesome.