You are here: Home Blog A Django Development Environment with zc.buildout

A Django Development Environment with zc.buildout

by Dan Fairs last modified Nov 12, 2008 07:58 PM
This article will show you how to create a repeatable Django development environment from scratch using zc.buildout.

Setting up environments is a pain. Whether it's Django, Zope, ASP.NET, whatever - a typical web stack has often dozens of components with dependencies on each other and underlying libraries. How do you manage this? How do you make sure that the software you're running on your development environment is configured the same way, and is the same version that gets into your production environment? How do you make sure that the third-party Python library you've just started using is correctly deployed?

One answer is zc.buildout. Buildout is a tool for reliably creating reproducible software builds. It was originally developed by Zope Corporation, and is often used in Zope builds; however, there's no dependency on Zope. You can use it to build pretty much anything. And I'm going to show you how to get a Django build up and running using it.

I shall use PostgreSQL as the database in my examples, but there's nothing stopping you using MySQL or any other Django-supported database, if you wish.

You'll also need the standard development tools (gcc, etc.) available since we're going to be getting buildout to compile some binary eggs for us.


The Basics: Python and PostgreSQL

The only thing you need to get going is some version of Python installed. The system Python is probably fine, as long as it's version 2.3 or later. Get a database installed too: I'm using PostgreSQL here. (You can use buildout to install a database too; I'm not going to cover that here, though, since most people have their database installed through system packages.)

We are going to install two packages in the system python: setuptools and virtualenv.

(If you don't want to touch the system Python at all, that's fine; check out my earlier article on how to compile and install a local version of Python. You might want to do this if your system only offers an old version of Python. I do it as a matter of course, but then get shouted at by system admins who want to use a package manager to keep their Python up to date. Your mileage may vary.)

Download ez_setup.py, and run it as a user who can write to the Python's site-packages directory (root if you're using your system python):

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

Let's also take the opportunity to create a database for the project. This is for PostgreSQL; obviously, substitute whatever's appropriate for your platform:

createdb djangodevdb 


virtualenv

Before we get stuck into buildout, let's talk about virtualenv.

You're probably going to have more than one project on the go. You want to keep them separate from each other, and want to avoid polluting your system Python installation with application-specific third-party modules. This is what virtualenv does. It lets you create isolated sandboxes: modules installed in one virtualenv don't interfere with other virtualenv. Let's install virtualenv and create an environment for our Django environment:

easy_install virtualenv

That will download and install virtualenv. Next up, let's create ourselves a sandbox called 'djangodev' to work in:

hornet:2.4 dan$ virtualenv --no-site-packages djangodev
New python executable in djangodev/bin/python
Installing setuptools.............done.
hornet:2.4 dan$

Finally, we need to 'activate' the sandbox. This isolates the environment from the system Python, and ensures that any modules installed are local to this environment.

hornet:2.4 dan$ cd djangodev/
hornet:djangodev dan$ source bin/activate
(djangodev)hornet:djangodev dan$

Note how the command prompt changes as a visual indicator that we're now working in a virtualenv. You can type 'deactivate' if you want to exit the virtualenv.


Initial Configuration

Getting going with buildout is very straightforward. Create a top-level directory to hold your application and download the bootstrap.py file into it:

mkdir app
cd app
wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

Don't run this yet.


A Basic Buildout Configuration

Next, create a file in that same directory called buildout.cfg, and put the following into it:

[buildout]
parts =

This is pretty much the simplest buildout configuration you can start with.


Bootstrapping

The very first time you use buildout, you have to bootstrap it. This installs buildout itself, and generates scripts to run the buildout. Bootstrapping the buildout is simply a matter of running the bootstrap.py file you downloaded earlier. You should see output resembling this:

(djangodev)hornet:app dan$ python bootstrap.py 
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/bin'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/parts'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/develop-eggs'.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/buildout'.
(djangodev)hornet:app dan$

That just creates buildout's initial scripts and directory layouts. You don't have to run it again for this environment.


Installing Django

So, all that was boring. And it probably seemed fiddly: after all, couldn't we just have installed Django in the virtualenv's site-packages manually? Yes, we could; but then we'd have had to do that every time we deployed an environment. We can now start to use buildout to automate our builds. Let's install Django 1.0 with buildout.

Open up your buildout.cfg again, and change it so that it looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0

Now, go ahead and run buildout. This will download the Django 1.0 distribution, so may take a few minutes depending on your connection:

(djangodev)hornet:app dan$ bin/buildout 
Unused options for buildout: 'download-directory'.
Installing django.
django: Downloading Django from: http://www.djangoproject.com/download/%s/tarball/
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
(djangodev)hornet:app dan$

Buildout (via the 'djangorecipe' extension) has done a couple of things for us:

  • It has created a script called bin/django to run the django management commands
  • It has created an inital Django project (called, imaginitively, 'project') for us with some default settings
  • It has installed Django

Note that buildout created a script called 'django' in the bin directory. This script it the exact equivalent of django-admin.py, or running python manage.py when manually installing Django; that is, you can run bin/django syncdb, bin/django sqlall, eveything you would expect. So let's go ahead and try to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Traceback (most recent call last):
File "bin/django", line 20, in ?
djangorecipe.manage.main('project.development')
File "/Users/dan/.buildout/eggs/djangorecipe-0.13-py2.4.egg/djangorecipe/manage.py",
line 15, in main
management.execute_manager(mod)
[ ... snip ... ]
File "/Users/dan/opt/virtual/2.4/djangodev/app/parts/django/django/db/
backends/sqlite3/base.py", line 26, in ?
raise ImproperlyConfigured, "Error loading %s module: %s" % (module, e)
django.core.exceptions.ImproperlyConfigured: Error loading pysqlite2 module:
No module named pysqlite2

Well - that didn't go so well!

The problem here of course is that we've installed Django, but we haven't specified the database to connect to, or installed the python module required to connect to the database. Let's do both now.


Installing Dependencies

Configuring Django to connect to a database is well covered in the Django documentation, so for now I'll just tell you to edit the project/settings.py file and change DATABASE_ENGINE to 'postgresql_psycopg2', and to set your DATABASE_NAME, DATABASE_USER etc. appropriately for your installation.

Installing the connector is more interesting. The connector for PostgreSQL is called psycopg2. Let's tell buildout to install that. Open up your buildout.cfg again, and change it so it now looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0
eggs = psycopg2

All we've done is add psycopg2 to the list of eggs to install with Django.

Now just rerun buildout:

(djangodev)hornet:app dan$ bin/buildout 
Uninstalling django.
Unused options for buildout: 'download-directory'.
Installing django.
Getting distribution for 'psycopg2'.
warning: no files found matching '*.html' under directory 'doc'
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186:
warning: 'PyDateTimeAPI' defined but not used
psycopg/typecast.c:37: warning: 'skip_until_space' defined but
not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning:
'PyDateTimeAPI' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning:
'PyDateTimeAPI' defined but not used
zip_safe flag not set; analyzing archive contents...
Got psycopg2 2.0.8.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
django: Skipping creating of project: project since it exists

As you can see, buildout noticed that we'd specified an extra requirement of psycopg2 and so downloaded it from PyPI, compiled and installed it for us. What buildout did is essentially analagous to running 'easy_install psycopg2'. Now we should be able to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Validating models...
0 errors found

Django version 1.0-final-SVN-unknown, using settings 'project.development'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

It worked! You can now go ahead and check your buildout.cfg into source control. Anyone checking that out will get the same Django build as you.


PIL

PIL, the Python Imaging Library, is always a bit of a pain to install. Django requires it if you want to work with images. It's also not packaged with setuptools, let alone as an egg. How can we get this into our build?

Fortunately, Chris McDonough has repackaged PIL with setuptools, making it relatively straightforward to add to our build. Open up buildout.cfg again, and edit it so that is looks like this:


[buildout]
parts =
PIL
django

[django]
recipe = djangorecipe
version = 1.0
eggs =
psycopg2
markdown
PIL

[PIL]
recipe = zc.recipe.egg
egg = PIL==1.1.6
find-links = http://dist.repoze.org/

Rerun buildout, as before, and PIL should be downloaded, compiled and installed.

That should be enough to get you going with buildout. You can find lots more on buildout and the Django recipe in the links below:

Google found me those links, so I'm sure it can find more for you too! I'd particularly encourage you to read the djangorecipe documentation for more detail on how it can configure Django for you.

In future articles, I intend to talk about

  • Starting Django applications as eggs
  • Configuring applications as development eggs inside buildout
  • Packaging applications and uploading them to PyPI, so that they're just an easy_install away
  • Dealing with third-party Django applications which have not been packaged as eggs
  • Using buildout to build non-python dependencies
  • How all this works with source control

Until next time.

Filed under: ,
Bram
Bram says:
Sep 30, 2010 04:15 PM
Great article. I already use buildout for some time now and there was not that much new in here. I'm looking forward to your articles about distributing apps as an egg. The thing that I'm stil looking for is a way to easely deploy my apps to my testing/distribution server. I know that some things are possible with buildout and that there's fabric etc, but I didn't really work a strategy out yet.

It would be nice if the Django community gets more pythonic in distribution/deployment. More eggs, more apps on PyPi etc.
Dan Fairs
Dan Fairs says:
Sep 30, 2010 04:15 PM
Absolutely agree. For any third-party projects that I use that aren't at least setuptools-compatible (never mind on PyPI) I'm raising issues and attaching a setup.py file. Coming to Django after Zope and suddenly basically losing eggs does feel like a step backwards.

Distributing Django apps as eggs is actually straightforward, as I'll demonstrate in a future article. There's still some debate in the Django community about the best way to ship static media, and that's something I confess to have not solved myself yet!
tutuca
tutuca says:
Sep 30, 2010 04:15 PM
Hi fez, great article. I'm using this your paster templates to create a buildout for a project I'm working on, since it's regarding buildout more than paster I thought it worth posting here...

I'm early in the development of a new app and, although I've used django before (this project is about integrating a couple of "pet projects") it's the first time that I use buildout for a django development.

My common practice in plone is that buildout creates me an _instance_ to work with, and I keep the _eggs_ (usually a theme or policy product) in the src directory and maintain it versioned with EXTERNALS.

So the bit of the instance that I'm working on keeps versioned and buildout only fetch it for me.

The thing is that with the config that got into the paster template buildout will create a django project every time it's run, a project that I rather would not overwrite as i had set up my settings to get the proper INSTALLED_APPS, the database config and stuff.

I'm still eary in the process (as you might see), so it's sort of newbie question, but what would be your way of dealing with this situation?, how do you use django_buildout?

Kind regards
Dan Fairs
Dan Fairs says:
Sep 30, 2010 04:15 PM
Hi tutuca,

Well - I don't actually use django_buildout! I prefer djangorecipe, which is used in the article. One of the options that djangorecipe offers is 'projectegg'. This lets you specify one of your eggs as the project configuration egg, which contains all your config options. If that's specified, then the recipe won't generate a project egg.

As far as layout goes, I still have a src/ directory which all my Django develop eggs live in (be they actually checked in there, or managed via EXTERNALS.)

Tim Child
Tim Child says:
Sep 30, 2010 04:15 PM
Is it really needed to build psycopg2? These means that the linux box I tried it on needed libpq-dev and build-essentials installed and then it failed anyway.
Tim Child
Tim Child says:
Sep 30, 2010 04:15 PM
It seems that I was missing python-dev. Doesn't really make it that portable unless I install these things..
Dan Fairs
Dan Fairs says:
Sep 30, 2010 04:15 PM
Yeah, you need a bunch of -dev packages installed - you mentioned python-dev, but you'll also need the various dev versions of jpeg and whatever you want PIL to support. If you don't want to have those installed, then you could always install psycopg2 using your system package manager, and use that.

Personally I prefer to build it within the project, since I run apps with different Python versions on the same machine - so of course, I often end up with an individual Python for each app. This is mostly because the Zope 2 stuff I have required a Python no newer than 2.4.
MarcoBazzani
MarcoBazzani says:
Sep 30, 2010 04:15 PM
today dist.repoze.org doesn't work so I changed the PIL part like this:

[PIL]
recipe = zc.recipe.egg:custom
egg = Imaging
find-links = http://effbot.org/downloads/

it works like a charm and works even on environment with old version of setuptools (while the Pil version of dist.repoze.org doesn't)

on windows it doesn't work but on windows you don't need it because eggs=PIL is enough ( I tested it trust me :) )

MaxP
MaxP says:
Sep 30, 2010 04:15 PM
Just a little notice that PIL 1.1.7 is already out there and it's installable via easy_install/pip, so there's no need for an extra receipe.
Kevin Williams
Kevin Williams says:
Sep 30, 2010 04:15 PM
At the risk of sounding like a jerk and reviving a dead thread, this has been a great resource for me for the past couple months -- because I keep forgetting the steps.

I finally translated them into a little bash script that does everything completely automated with the only interaction being the original provision of a name for the project folder. It's not great (no error checking or anything) so I won't post it here, but it suffices to say you can pretty much copy-paste the instructions here, substitute `$1` for the folder name in the first couple steps, and `cat << EOF > buildout.cfg` when you need to edit buildout.cfg. I set up my buildout.cfg with a few commonly-used eggs as well (like postgres/mysql connectors).

Thanks!
Matt L.
Matt L. says:
Sep 30, 2010 04:15 PM
Very nice post. I've used buildout before, but not effectively and was kind of lost. Maybe I also need a better strategy.

I agree with one of the comments above, it would be helpful if Django gets more pythonic in distribution and dep.
Add comment

You can add a comment by filling out the form below. Plain text formatting.

Stereoplex is sponsored by Fez Consulting Ltd