A Django Development Environment with zc.buildout

Setting up environments is a pain. Whether it's Django, Zope, ASP.NET, whatever - a typical web stack has often dozens of components with dependencies on each other and underlying libraries. How do you manage this? How do you make sure that the software you're running on your development environment is configured the same way, and is the same version that gets into your production environment? How do you make sure that the third-party Python library you've just started using is correctly deployed?

One answer is zc.buildout. Buildout is a tool for reliably creating reproducible software builds. It was originally developed by Zope Corporation, and is often used in Zope builds; however, there's no dependency on Zope. You can use it to build pretty much anything. And I'm going to show you how to get a Django build up and running using it.

I shall use PostgreSQL as the database in my examples, but there's nothing stopping you using MySQL or any other Django-supported database, if you wish.

You'll also need the standard development tools (gcc, etc.) available since we're going to be getting buildout to compile some binary eggs for us.

The Basics: Python and PostgreSQL

The only thing you need to get going is some version of Python installed. The system Python is probably fine, as long as it's version 2.3 or later. Get a database installed too: I'm using PostgreSQL here. (You can use buildout to install a database too; I'm not going to cover that here, though, since most people have their database installed through system packages.)

We are going to install two packages in the system python: setuptools and virtualenv.

(If you don't want to touch the system Python at all, that's fine; check out my earlier article on how to compile and install a local version of Python. You might want to do this if your system only offers an old version of Python. I do it as a matter of course, but then get shouted at by system admins who want to use a package manager to keep their Python up to date. Your mileage may vary.)

Download ez_setup.py, and run it as a user who can write to the Python's site-packages directory (root if you're using your system python):

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

Let's also take the opportunity to create a database for the project. This is for PostgreSQL; obviously, substitute whatever's appropriate for your platform:

createdb djangodevdb 

virtualenv

Before we get stuck into buildout, let's talk about virtualenv.

You're probably going to have more than one project on the go. You want to keep them separate from each other, and want to avoid polluting your system Python installation with application-specific third-party modules. This is what virtualenv does. It lets you create isolated sandboxes: modules installed in one virtualenv don't interfere with other virtualenv. Let's install virtualenv and create an environment for our Django environment:

easy_install virtualenv

That will download and install virtualenv. Next up, let's create ourselves a sandbox called 'djangodev' to work in:

hornet:2.4 dan$ virtualenv --no-site-packages djangodev
New python executable in djangodev/bin/python
Installing setuptools.............done.
hornet:2.4 dan$

Finally, we need to 'activate' the sandbox. This isolates the environment from the system Python, and ensures that any modules installed are local to this environment.

hornet:2.4 dan$ cd djangodev/
hornet:djangodev dan$ source bin/activate 
(djangodev)hornet:djangodev dan$ 

Note how the command prompt changes as a visual indicator that we're now working in a virtualenv. You can type 'deactivate' if you want to exit the virtualenv.

Initial Configuration

Getting going with buildout is very straightforward. Create a top-level directory to hold your application and  download the bootstrap.py file into it:

mkdir app
cd app
wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

Don't run this yet.

A Basic Buildout Configuration

Next, create a file in that same directory called buildout.cfg, and put the following into it:

[buildout]
parts =

This is pretty much the simplest buildout configuration you can start with.

Bootstrapping

The very first time you use buildout, you have to bootstrap it. This installs buildout itself, and generates scripts to run the buildout. Bootstrapping the buildout is simply a matter of running the bootstrap.py file you downloaded earlier. You should see output resembling this:

(djangodev)hornet:app dan$ python bootstrap.py 
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/bin'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/parts'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/develop-eggs'.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/buildout'.
(djangodev)hornet:app dan$ 

That just creates buildout's initial scripts and directory layouts. You don't have to run it again for this environment.

Installing Django

So, all that was boring. And it probably seemed fiddly: after all, couldn't we just have installed Django in the virtualenv's site-packages manually? Yes, we could; but then we'd have had to do that every time we deployed an environment. We can now start to use buildout to automate our builds. Let's install Django 1.0 with buildout.

Open up your buildout.cfg again, and change it so that it looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0

Now, go ahead and run buildout. This will download the Django 1.0 distribution, so may take a few minutes depending on your connection:

(djangodev)hornet:app dan$ bin/buildout 
Unused options for buildout: 'download-directory'.
Installing django.
django: Downloading Django from: http://www.djangoproject.com/download/%s/tarball/
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
(djangodev)hornet:app dan$

Buildout (via the 'djangorecipe' extension) has done a couple of things for us:

• It has created a script called bin/django to run the django management commands
• It has created an inital Django project (called, imaginitively, 'project') for us with some default settings
• It has installed Django
  

Note that buildout created a script called 'django' in the bin directory. This script it the exact equivalent of django-admin.py, or running python manage.py when manually installing Django; that is, you can run bin/django syncdb, bin/django sqlall, eveything you would expect. So let's go ahead and try to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Traceback (most recent call last):
  File "bin/django", line 20, in ?
    djangorecipe.manage.main('project.development')
  File "/Users/dan/.buildout/eggs/djangorecipe-0.13-py2.4.egg/djangorecipe/manage.py", 
    line 15, in main
    management.execute_manager(mod)
 [ ... snip ... ]
  File "/Users/dan/opt/virtual/2.4/djangodev/app/parts/django/django/db/
    backends/sqlite3/base.py", line 26, in ?
    raise ImproperlyConfigured, "Error loading %s module: %s" % (module, e)
django.core.exceptions.ImproperlyConfigured: Error loading pysqlite2 module: 
    No module named pysqlite2

Well - that didn't go so well!

The problem here of course is that we've installed Django, but we haven't specified the database to connect to, or installed the python module required to connect to the database. Let's do both now.

Installing Dependencies

Configuring Django to connect to a database is well covered in the Django documentation, so for now I'll just tell you to edit the project/settings.py file and change DATABASE_ENGINE to 'postgresql_psycopg2', and to set your DATABASE_NAME, DATABASE_USER etc. appropriately for your installation.

Installing the connector is more interesting. The connector for PostgreSQL is called psycopg2. Let's tell buildout to install that. Open up your buildout.cfg again, and change it so it now looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0
eggs = psycopg2

All we've done is add psycopg2 to the list of eggs to install with Django.

Now just rerun buildout:

(djangodev)hornet:app dan$ bin/buildout 
Uninstalling django.
Unused options for buildout: 'download-directory'.
Installing django.
Getting distribution for 'psycopg2'.
warning: no files found matching '*.html' under directory 'doc'
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: 
  warning: 'PyDateTimeAPI' defined but not used
psycopg/typecast.c:37: warning: 'skip_until_space' defined but 
  not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning: 
  'PyDateTimeAPI' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning: 
  'PyDateTimeAPI' defined but not used
zip_safe flag not set; analyzing archive contents...
Got psycopg2 2.0.8.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
django: Skipping creating of project: project since it exists

As you can see, buildout noticed that we'd specified an extra requirement of psycopg2 and so downloaded it from PyPI, compiled and installed it for us. What buildout did is essentially analagous to running 'easy_install psycopg2'. Now we should be able to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Validating models...
0 errors found

Django version 1.0-final-SVN-unknown, using settings 'project.development'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

It worked! You can now go ahead and check your buildout.cfg into source control. Anyone checking that out will get the same Django build as you.

PIL

PIL, the Python Imaging Library, is always a bit of a pain to install. Django requires it if you want to work with images. It's also not packaged with setuptools, let alone as an egg. How can we get this into our build?

Fortunately, Chris McDonough has repackaged PIL with setuptools, making it relatively straightforward to add to our build. Open up buildout.cfg again, and edit it so that is looks like this:

[buildout]
parts =
  PIL
  django

[django]
recipe = djangorecipe
version = 1.0
eggs =
  psycopg2
  markdown
  PIL

[PIL]
recipe = zc.recipe.egg
egg = PIL==1.1.6
find-links = http://dist.repoze.org/

Rerun buildout, as before, and PIL should be downloaded, compiled and installed.

That should be enough to get you going with buildout. You can find lots more on buildout and the Django recipe in the links below:

• http://buildout.zope.org/
• http://pypi.python.org/pypi/zc.buildout
• http://pypi.python.org/pypi/djangorecipe/
• http://plone.org/documentation/tutorial/buildout (Plone-oriented, but lots of useful background information there too)
• http://renesd.blogspot.com/2008/05/buildout-tutorial-buildout-howto.html
• http://wiki.python.org/moin/buildout/pycon2008_tutorial

Google found me those links, so I'm sure it can find more for you too! I'd particularly encourage you to read the djangorecipe documentation for more detail on how it can configure Django for you.

In future articles, I intend to talk about

• Starting Django applications as eggs
• Configuring applications as development eggs inside buildout
  
• Packaging applications and uploading them to PyPI, so that they're just an easy_install away
• Dealing with third-party Django applications which have not been packaged as eggs
• Using buildout to build non-python dependencies
• How all this works with source control