You can use GenericForeignKeys for this, and these are slightly more flexible in that each model instance foreign key may point to a different model. However, there is a performance cost associated with them, and joining can be problematic.

(The project actually rarely uses django-dfk directly - instead, it uses it as a basis for abstract foreign keys, which have a greater awareness of the application environment - however, that's a topic for another day.)

Before we go much further though - rather than using this package, a better long-term investment would be to look at the app-loading branch that Arthur Koziel and Jannis Leidel have been working on - testing it, and helping them to get it into a state to be merged into trunk. I think that should provide a more holistic approach to solving this kind of problem. Until then...

Deferred foreign keys are useful in applications where you know that a model will require a foreign key, but don't know what the target will be at the time you're writing the application. Taking an example from the project that django-dfk was created for, let's say you have a Django app that is an online game. Users are entered into the game by way of an 'Entry' model, and the Entry contains a foreign key back to a model which contains user information.

However, you will have several instances of this game deployed, and each game may have its own source of player data - hence, its own Player model.

Let's say that there are two applications involved: one called 'core', which contains the core game logic, and one called 'mygame1', which contains models and logic specific to an individual game deployment.

core/models.py:
from django.db import models

class Entry(models.Model):
    created = models.DateTimeField(defaults=datetime.datetime.now)
    player = models.ForeignKey( …. uh, what do I type here?

Remember - the core application is going to be deployed in several places, and there are several possible places that FK might need to point.

One approach solving this would be to introduce a model which relates an entry to the game-specific Player model, resulting in models that look something like this:

core/models.py:
from django.db import models

class Entry(models.Model):
    created = models.DateTimeField(defaults=datetime.datetime.now)


mygame1/models.py:

from django.db import models

class MyPlayer(models.Model):
    name = models.CharField(max_length=50)
    entry = models.OneToOneField(Entry)

This works fine - however, there are some games which share player data with each other. This means that the key needs to live on the Entry model - but, we don't know which player model to point the FK at, as this model might be deployed in multiple places.

We can solve this using django-dfk.

core/models.py:
from django.db import models
from dfk.models import DeferredForeignKey

class Entry(models.Model):
    created = models.DateTimeField(defaults=datetime.datetime.now)
    player = DeferredForeignKey(unique=True)


mygame1/models.py:

from django.db import models
from dfk import point
from core import Entry

class MyPlayer(models.Model):
    name = models.CharField(max_length=50)

point(Entry, 'player', MyPlayer)

The first thing to notice is that our Entry model now sports a DeferredForeignKey instance. It's important to realise that this isn't a real field, it's just a placeholder. Any arguments (except for the special 'name' argument, more on this below) are simply stored.

The action happens during the call to 'point', in mygame1's models.py. As the name implies, this points the DFK called 'player' on Entry to the MyPlayer class. (Actually, under the hood, it simply replaces the DeferredForeignKey instance with a real ForeignKey instance complete with the arguments which were originally passed to the DFK). Note that we do this at the module level in models.py - all your pointing (and repointing) needs to be done before the application is ready to use to ensure that syncdb outputs the correct SQL.

After all this is done, the definition of Entry will effectively look like this (although of course your code won't have changed!):

class Entry(models.Model):
   created = models.DateTimeField(defaults=datetime.datetime.now)
   player = models.ForeignKey('mygame1.MyPlayer', unique=True)

Other game applications (say, mygame2 and mygame3) would point the foreign key to the Player model that is appropriate for their game.

It's quite common to need to point a number of these keys at once - there might be several models which refer to a player. Rather than writing lots of 'point' statements (and having to add to them if a new key is added), django-dfk allows deferred foreign keys to be named:

core/models.py
class Entry(models.Model):
   created = models.DateTimeField(defaults=datetime.datetime.now)
   player = DeferredForeignKey(name='Player', unique=True)

class StatusUpdate(models.Model):
   text = models.CharField(max_length=140)
   player = DeferredForeignKey(name='Player')


mygame1/models.py:

from django.db import models
from dfk import point
from core import Entry

class MyPlayer(models.Model):
   name = models.CharField(max_length=50)

point_named('core', 'Player', MyPlayer)

Roughly translated, this means 'point all the deferred foreign keys in all models in the core app which have the name 'Player' to the MyPlayer model'. This will affect both the 'Entry' and 'StatusUpdate' models above.

Finally, django-dfk also provides a 'repoint' function. This is a big hammer, and is not to be used lightly. 'point' only works on DeferredForeignKey instances, by design - it's meant to prevent you making mistakes. 'repoint' works on regular foreign keys too. It's useful if you've used 'point' to change the destination of a DFK, but later need to change it. (However, if you find yourself in this position, you should probably refactor your code to just use 'point'). Both 'point' and 'repoint' take care of cleaning up various internal Django caches to ensure things life filtering on related fields work properly after a point operation.

django-dfk can be found on PyPI, and the source is on github - forks, bug reports and patches with docs and tests are welcome. django-dfk is in production on several high-volume sites.

What's a REST API?

There are probably as many definitions of what a REST API is as there are implementations (so naturally I'll add my own), but they all share some common characteristics:

• They recognise that the web is composed of resources, and are structured around that
• They use the underlying HTTP methods (GET, POST, PUT, DELETE) to interact with resources
• Representations of those resources are what actually flow back and forth between REST API servers and clients
• URIs (usually URLs, which I'll use for the remainder of the article) are used to identify application state - particularly information on what the client can do next - andis contained within the representations received from the server.

Take a look at the Wikipedia article on REST for one description of what it means to be RESTful.

Resources

When you design a REST API, the first task is usually to consider what resources you want to expose to web clients. This is often pretty straightforward: a reasonable approach is often to use the same arrangement as your underlying data model (be that tables in a relational database, document types in a document management system, and so on). This doesn't have to be the case, of course - in particular, you'd be unlikely to expose a link table that facilitates a many-to-many relation in a relational database as an individual resource, for example.

Let's say we're building an API for a pizza shop. We're probably going to have a Pizza resource, a Topping resource, and an Order resource.

Resources tend to be arranged hierarchically, probably due to the nature of the URLs used to address them. It's therefore common to have resources in an API that work as collections of other resources. The URL for a topping resource might be `http://pizza.com/toppings/cheese`. The URL of the collection resource for all toppings would be `http://pizza.com/toppings/`. (The presence of absence of a trailing slash is meaningless, but for the purpose of this article I'll use the convention of collection resources having a trailing slash.)

Representations

Representations are how the client and server talk about a resource. Prior to the advent of machine-usable APIs, HTML was the most common form of representation. These days, XML and JSON representations are commonly used for machine-readable representations. PDF, JPEG, and MP3 are also all perfectly good representations of a resource as is, of course, good old HTML.

Keep in mind that a resource may have more than one representation. You might be able to fetch a resource as HTML (useful for humans) and JSON (useful for machines). It's also important to realise that representations are not simply content types: there's no reason a resource can't have multiple JSON representations, for example. This idea becomes important a little later on.

Once you've defined your resources, therefore, thinking of representations is usually the next step. The default choice of a simple JSON representation, with a key per resource attribute, is a common choice. That will do for now. A representation of our Topping resource might therefore look like this:

{
 "calories": 100, 
 "name": "Cheese"
}

Resources may need to refer to other resources. This should be done in a self-describing way: the client should not need to have any knowledge of the server application to build its own URLs. Seeing keys like 'foo_id' in a JSON representation is usually a sign of this design error. For example, this is a reasonable representation:

{
 "favourite_topping": "/toppings/cheese"
}

This isn't so good:

{
 "favourite_topping_id": "36"
}

That 'favourite_topping_id' is meaningless to the client - it has to know how to construct topping resource URLs to be able to use that data.

Incidentally, note that the examples in this article only include URL paths, rather than full, absolute URLs. Either is fine; as long as the client can resolve them.

Interactions

The next piece of the usual REST API design process is to consider the operations available for each resource, and what they mean. There are four key operations provided by HTTP - GET, POST, PUT and DELETE. (Actually, HTTP does provide more, but this quartet is what is most normally used in RESTful API design.)

These are often compared to the core SQL-based relational database operations of SELECT, INSERT, UPDATE and DELETE. This is slightly misleading. SELECT and GET are fairly similar, as are the two DELETEs; POST and PUT are different beasts though. POST is used for a write operation on a resource that has side-effects. PUT writes to a resource, but has no side-effects.

Put another way, PUT is idempotent - if you do the same PUT twice (and there's no other state changes in between) then the system state will be the same. POST carries no such guarantee.

A good example might be to compare the creation of a new Topping resource with the creation of an Order for a pizza. Creating a Topping would probably consist of something like the following:

PUT http://pizza.com/toppings/jalapenos
{
  "calories": 25,
  "name": "Jalapeños"
}

This would create the Topping resource at http://pizza.com/toppings/jalapenos. Re-running the request would not make any difference. Changing the 'calories' field in the JSON to a new value would replace the existing resource with the new one. So - PUT has a 'create-or-update' semantic.

The response for this request would probably simply be an HTTP 200 response, with an empty body (or more strictly, a 204, which tells the client to maintain its view of the representation.)

Contrast this with what creating an order might look like:

POST http://pizza.com/orders/
{
 "toppings": [
   "/toppings/cheese", 
   "/toppings/jalapenos"
 ], 
 "card": "1234567890"
}

We've made this a POST request because it has side-effects: it bills your credit card, and sets off the process of making a delicious pizza. Making that same request twice would bill your card twice, and get you two pizzas. POST is not idempotent.

The HTTP response here would probably be 201 Created, with the representation of some Confirmation resource in the body, perhaps looking like this:

201 Created
{
 "order": "/orders/432544"
}

Note once more how the response contains a self-describing URI, rather than some opaque order ID, which is not meaningful out of context.

Reality Bites

So we've identified our resources, the representations of them, and what operations the HTTP verbs actually correspond to. We're good to go, right?

Well, yes, basically. It'll work. But you'll almost certainly run into some problems. Before we get to the meat of selecting resource representation though, let's take a minute to consider a couple of real-world implementation problems you're likely to encounter.

Aside 1: Bad HTTP clients

There are some broken HTTP clients out there. I've run into one: Flash (circa 2009). Flash gets upset if it doesn't receive a 200 response from the server. In particular, if your server returns a 4xx HTTP code in an API response, Flash will not even pass the response to the Flash application.

On the Django project on which we ran into this, we ended up writing a custom middleware that looked for the presence of a magic query string parameter on the URL and, if it was found, replaced the status code with a 200 and put the real status code on the first line of the body. The Flash app then parsed out the response code from the response body. Ugly, but workable.

Flash also (at the time, it may have changed) was unable to perform PUT or DELETE requests. Our solution was similar: the Flash application would always perform a POST when it actually wanted to do a PUT or DELETE, and the real intended method went into another magic query string parameter. The aforementioned middleware would then rewrite the HTTP method on incoming API requests that carried this flag.

Aside 2: Rich Error Handling

The standard approach to expressing errors in a REST API is to use HTTP status codes. As is often the case with REST, this works fine for simple systems, but is simply too limiting for more sophisticated systems, particularly those which might submit a JSON or XML document to describe a POST request. If a client does submit such a rich request, which perhaps does not validate on the server, it is useful to be able to provide more than just a 400 Bad Request error.

Since I primarily use Django these days, and Django uses forms and formsets for validation, I have found that providing a standardised JSON representation of form and formset errors works well. The format can be specified in advance, and allows the server to inform the client of rich validation errors (down to field-level validation, with decent error messages) even over an API. I hope to write more on this in a future post.

Normal service resumes...

Normalised resource representations

OK, let's say we've got an iPhone app which uses our REST API to place orders and display the calorific content of the toppings we added. Let's look at the response that it might receive from a GET request to our Order resource:

GET /orders/432544

200 OK
{
 "toppings": [
   "/toppings/cheese", 
   "/toppings/jalapenos"
 ]
}

That's cool - the client can see that there are a couple of toppings there, so it fetches each one to get the calorific content:

GET /toppings/cheese
200 OK

{
 "calories": 100, 
 "name": "Cheese"
}

And then:

GET /toppings/jalapenos

{
 "calories": 25, 
 "name": "Jalapeños"
}

That's great. Our iPhone app can tell our user that the pizza they ordered has 125 calories.

What's not so great is that our iPhone app has had to make three separate requests. This works fine in development, across the office wifi network to the dev server. It doesn't work so well when a user's ordering a pizza on the train home from work, and the train goes into a tunnel halfway through this multi-request conversation (and the user was outside a 3G signal anyway).

Nesting resource representations

The natural response (and probably the right response) is simply to extend the representation of an Order resource to include the required representations of our toppings. This means that our Order resource representation now looks like this:

GET /orders/432544

200 OK
{
    "toppings": [
    {
        "calories": 100, 
        "name": "Cheese"
    },
    {
        "calories": 25, 
        "name": "Jalapeños"
    }
    ]
}

That's cool. Our iPhone app now only needs to make one request, and it gets all the information on the toppings as well. We've traded the brevity of the original representation of the Order resource for not having to make multiple requests. The individual Topping representations are still available, of course.

To (ab)use database parlance, we've denormalised our Order representation, trading size for performance (that is, a smaller representation will be quicker to download).

Now, wind the clock forward a few months. We've extended our iPhone app and the supporting REST API to cover table reservations, meal pre-ordering, and so on. We've run into the problem described above, so we've heavily optimised our API responses to minimise HTTP round trips. Life is good. Right?

We're approached by a company who have developed an Android app that can use our API, wondering if we'd be happy to make it the official Android app. It's an awesome piece of software. They've thought about the user experience in a totally different way, and it works well on Android (though the current iPhone app approach works best on iPhone). The only real problem is that some of the API requests they make download a ton of data that they simply don't need; and they have to make lots of other smaller requests to make other parts of their app work as they want.

In other words, they want a different set of optimisation choices for the API.

What do we do? Add a new API version with different optimisations (even though it's still dealing with the same set of resources) in the representations? That doesn't sound so great, as we'd be maintaining two APIs. It doesn't scale, either - what happens when someone writes an app for the TV, which needs another set of tradeoffs?

A Possible Solution: Choosing Representations

(Note that this section outlines a possible solution - I don't have an implementation for this yet.)

The key insight is that the applications do not require different resources. They merely need different representations of those resources. Some frameworks allow this to be done by specifying an extension on the end of the URL. If we did that, we'd end up with:

http://pizza.com/toppings/cheese.xml
http://pizza.com/toppings/cheese.json

While not strictly wrong, this isn't ideal when thinking in resources. Different URLs mean different resources. Just because both XML and JSON representations are available doesn't double the number of resources - it's all the same cheese topping. We need some way of expressing what representation we want for a given resource. Ideally, we shouldn't change the URL.

Fortunately, HTTP gives us the tools to do this: the HTTP Accept header. It's usually used for specifying content types: text/html, application/json, and so on. However, the specification allows for extra parameters. Let's take a look (from http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html):

Accept         = "Accept" ":"
#( media-range [ accept-params ] )

media-range    = ( "*/*"
 | ( type "/" "*" )
 | ( type "/" subtype )
 ) *( ";" parameter )
accept-params  = ";" "q" "=" qvalue *( accept-extension )
accept-extension = ";" token [ "=" ( token | quoted-string ) ]

That means that a client could send a request like this:

GET /orders/432544
Accept: application/json;q=1;depth=1,application/json

Here, we've added a depth parameter to the acceptable types. This depth parameter works a little like the numeric depth parameter to a Django queryset's `select_related` method call - it asks the server to traverse one level deep into related resources. The standard also specifies that more specific content types take precedence over less specific types - which means that we have a mechanism for the client to allow fallback to a normal JSON representation of the resource. If the server cannot provide a representation that fulfils any of the values in the Accept header, it would return a 406 Not Acceptable.

Aside: q?

What's that 'q' parameter? It's what appears to be a historical quirk of the spec, where q is used to specify a 'quality' - used to allow the choice of a variety of representations based on degrading quality, eg. sample rate for sound. It's baked into the accept-params definition, so we include it here.

Example Requests

Let's compare two requests, one before using the proposed Accept convention, and one after:

GET /orders/432544
Accept: application/json

200 OK
{
 "toppings": [
   "/toppings/cheese", 
   "/toppings/jalapenos"
 ]
}

This is exactly the same request as we saw before, but with an explicit Accept header, specifying that the client is prepared to accept the normal form of the response. Let's see what might happen if a depth is specified:

GET /orders/432544
Accept: application/json,application/json;q=1;depth=1

200 OK
{
    "toppings": [
    {
        "calories": 100, 
        "name": "Cheese"
    },
    {
        "calories": 25, 
        "name": "Jalapeños"
    }
    ]
}

We can now see the client expressing that it can accept resource representations nested to a single level - and the server responds in kind. Note that the client also specified that it could accept the normal form - so it would be valid for the server to respond with the first, normal form even to this second request.

What if the server could not fulfil the request? Perhaps the backend is unable to perform the join required to provide the nested data, and the client specified that it could only accept that nested representation:

GET /orders/432544
Accept: application/json;q=1;depth=1

406 Not Acceptable

It might be sensible for the server to provide the normal representation in the body anyway, in case the client was able to process it.

In particular, this response might be used to prevent (or mitigate) DoS attacks; depending on the application, the server might impose a depth limit of 2 or 3 levels.

Thoughts: Advantages, Limitations and Questions

The approach outlined above would afford clients some flexibility in 'denormalising' the data that they receive on request, avoiding both the need for developers to create custom code to nest related data and the tendency for APIs to become over-specialised to the needs of one particular API client over time.

It's also cacheable - the Accept header should form part of the cache key, and provides a graceful degradation for servers which cannot perform the data nesting request.

However,it's not a silver bullet: in particular, it's not clear how one might implement an analogue to Django's .select_related('foo', 'bar') form, where the API consumer could specify which resources it wished to be nested in the main resource requested. Instead, all the client can specify is a simple depth, and may therefore receive far more nested information than it might actually need.

As far as I can see, there are two problems to solve before being able to implement this more specific second form:

• How to provide a self-describing way for a server to indicate that this facility is available for linked resources
• How a client can tell the server which individual resource links to expand
• Perhaps including an XPath-like parameter in the Accept header might work here (tweaked suitably to apply to JSON documents too); but how would the client know which paths were expandable without requesting the normal form to start with?

Implementation details aside, I do think that setting a convention (even of a limited form, such as the depth approach described above) would increase the usefulness of APIs. Perhaps I'll even get around to a Tastypie/Piston extension to implement it!

Comments on the approach, improvements and especially criticisms, are welcome in the comments.

I'd seen some efforts to solve this problem before. Most seemed to involve stashing the data for some predetermined part of the website somewhere (often in the session) and then reconstituting it into real application data when the user eventually bites the bullet and signed up. This worked OK, but you had to write it anew for every web site, as clearly the data you'd want to store would change from site to site. You also ended up effectively developing a miniature version of your site that would work with some limited data set.

This didn't really seem good enough.

So I started wondering - what if we just created a real user for every person who visited the site? Django already has support for creating users with unusable passwords - so if we just create a user with an unusable password every time a new person comes along, log them in, and then at some future point (presumably once they've fallen in love with your site) they can set themselves up with a real username and password. And as a bonus, all that data that they created while messing about with the site sticks around, and carries over into their 'real' user.

This is, in essence, how django-lazysignup works.

Let's take a look in a bit more detail. You can grab an official release from PyPI, or clone it on GitHub.

What's in the package?

Once you've installed django-lazysignup (I'll let you read the docs to see how to do that), you've got a few tools to play with:

• An authentication backend
• A user conversion view
• The allow_lazy_user view decorator
• The is_lazy_user template filter
• The user agent blacklist
• Custom user models

Authentication backend

The authentication backend needs to be installed for django-lazysignup to work. This backend is required so that we can authenticate the temporary user accounts without a password. We refer to these temporary users as 'lazy' users - they haven't bothered to sign up yet.

The user conversion view

In many cases, you're going to want your users to sign up eventually. The package includes a view to allow you to do this, converting a lazy user to a real user. In practice, this simply involves the user setting a username and password for their temporary user account. This approach means they get to keep all the data that they've already created in your application.

The allow_lazy_user decorator

The temporary user creation process is potentially an expensive one (and on a high-traffic site, may cause contention on the user table). django-lazysignup therefore provides a decorator that allows the developer to specify which views can trigger this process. Sites I've developed that use this package tend to apply the decorator to the views that do interesting things, but exclude the homepage and any static pages on the site.

The is_lazy_user template filter

It's often useful - particularly in templates - to find out whether the current user is a lazy (ie. temporary) user or not. In particular, you may want to show a link to the convert view if they're a lazy user. This template filter is provided for this purpose. Note that lazy users will appear as authenticated (ie. is_authenticated() returns True). For now, has_usable_password also returns False for lazy users, though this should not be relied on. The canonical way of detecting a lazy user is through the is_lazy_user filter (and its associated function in the utils module)

The user agent blacklist

You probably don't want every request to a view that's opted in to lazy user creation to actually create a user. Principally, you probably don't want search engines to do this. The user agent blacklist is a crude way of filtering out such robots.

Note that this means that views that have the allow_lazy_user decorator won't be guaranteed to always have an authenticated user. You still need to make sure that your views work with unauthenticated user (or mark them with the login_required decorator or similar).

Custom user models

As of version 0.7, django-lazysignup also has limited support for custom user models. Just set the LAZYSIGNUP_USER_MODEL setting appropriately (by default, it's auth.User to support contrib.auth out of the box). As alluded to before, support for custom users is basically predicated on the user model looking pretty much like a normal Django user - especially in that it's stored in the database.

I anticipate that this mechanism will change in the distant future, if Django were to adopt some form of pluggable model (or at least, a pluggable user model).

Restrictions

django-lazysignup was built with Django's own contrib.auth application in mind. If you're not using this for authentication, then integrating django-lazysignup will be more of a challenge. (If you do, and there are some changes to the package that would help you, please let me know!). In particular, it expects that you will have a user model stored somewhere in the database.

The Future

Honestly, I'd expected to have a 1.0 release out the door by now, but people keep suggesting great features that I'd like to get in before committing to API stability. Currently on the list are:

• Global view opt in, with opt-out decorator (suggested by Rob Hudson)
• Support for deferred user validation - for example, to support email address validation (suggested by Alex Ehlke)

In addition to Rob and Alex, thanks also to Luke Zapart for suggesting and providing an initial implementation and tests for the custom user model feature.

If you have a play with the app, and there's something you'd like to see, then do get in touch - or just fork it on GitHub, add the feature (with docs and tests, preferably!) and send me a pull request.

And that application that django-lazysignup was originally built for? Well, I still want to do it. Someday.

I researched this a little bit a while ago, and noticed that it had been reported as a bug in the MySQL tracker. At the time, there were no fixes or workarounds.

A recent update, however, has revealed the use of the skip-sync-frm option. Put it in your MySQL config file in the [mysqld] section for a quick speedup:

[mysqld]
default-table-type=innodb
transaction-isolation=READ-COMMITTED
default-character-set=utf8
skip-sync-frm=OFF

Of course, nothing in this life is free, as Daniel Fischer explains in a comment:

The reason why it's slower on Mac OS X than on Linux is that on Mac OS X, fcntl(F_FULLFSYNC) is available, and mysqld prefers this call to fsync(). The difference is that fsync() only flushes data to the disk - both on Linux and Mac OS X -, while fcntl(F_FULLFSYNC) also asks the disk to flush its own buffers and blocks until the data is physically written to the disk.

In a nutshell, it's slower because it's safer.

So, we're trading data integrity for performance - but this is a development machine, so trashing and recreating databases (or the MySQL installation for that matter) is fine, if necessary.

Et tu, Linux?

My colleague was having similar problems on the latest Ubuntu, 10.10. The tweak above helped him too, but his test runs were also painfully slow. He'd already added the 'noatime' option to fstab.

It turns out that the newest Ubuntu ships with ext4 as the default file system. By default, ext4 makes absolutely sure that all data has been written out to the filesystem journal before writing the journal commit record. This is done through the use of filesystem barriers. Again - this is done to prefer data integrity over performance. Since this is a dev machine, it's disposable, and performance is more important. So, we can turn this off in /etc/fstab:

/dev/sda3 on / type ext4 (noatime,rw,errors=remount-ro,barrier=0)

Read more about the barrier setting on Kernelnewbies.org.

Just to reiterate - it's probably best not to do this on a machine that's important without thinking about it carefully. Those settings have conservative defaults for a reason!

Here are the models that the examples will work with. They're abbreviated and slightly modified versions of some models from the Swoop project I'm currently working on:

class Area(models.Model):
    title = models.CharField(max_length=100)
    area = models.MultiPolygonField(blank=True, null=True)
 
class Trip(models.Model):
    title = models.CharField(max_length=100)
    area = models.ForeignKey(Area)
 
class Landmark(models.Model):
    title = models.CharField(max_length=100)
    point = models.PointField()
 
class MountaineeringInfo(models.Model):
    trip = models.ForeignKey(Trip)
    area = models.ForeignKey(Area, blank=True, null=True)
    base_camp = models.ForeignKey(Landmark, blank=True, null=True)

As you can see, we're using GeoDjango here - I'm not going to talk much about that here, but it should be obvious what's going on when we get it it. Note that these examples assume Django 1.2.

Filtering a form's ModelChoiceField

Consider this form:

class MountaineeringForm(forms.ModelForm):
    class Meta:
        model = MountaineeringInfo

This'll generate a simple form for us, including dropdowns with options for every Landmark, Trip and Area we have defined. Let's look at the area foreign key first. Note how the area attribute of the Area model is nullable. Let's say we only wanted to be able to select Areas from our MountaineeringForm which had a valid area attribute set - put another way, we want to filter out those records which are null.

This is pretty straightforward, and is in fact covered in the docs. And there are, in fact, two ways to do it:

class MountaineeringForm(forms.ModelForm):
    area = forms.ModelChoiceField(queryset=Area.objects.exclude(area=None))
    class Meta:
        model = MountaineeringInfo

This is probably the simplest way, and works well whenever the filtering you need to do does not depend on any request-specific or context-specific information.

The other option we have is to allow the ModelForm base class to do its usual thing, and then modify the fields that were generated directly.

class MountaineeringForm(forms.ModelForm):
    class Meta:
        model = MountaineeringInfo

    def __init__(self, *args, **kwargs):
        super(MountaineeringForm, self).__init__(self, *args, **kwargs)
        self.fields['area'].queryset = Area.objects.exclude(area=None)

Now, this is slightly more verbose, and to my eye, not so clear as the first version. However, this approach of modifying the form after the fields have been constructed is a pattern we'll see in the coming examples.

Filtering a Django Admin Dropdown

Now, let's say that we want to edit MountaineeringInfo instances in the Django admin. At the moment, we just have this in our admin.py:

admin.site.register(MountaineeringInfo)

This generates a form much as we had previously with our simple ModelForm definition. However, we still want to filter out those Area instances which don't have an area set. We do this by providing a custom ModelAdmin sublcass, and overriding the formfield_for_foreignkey method:

class MountaineeringInfoAdmin(admin.ModelAdmin):
    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        if db_field.name == 'area':
            kwargs['queryset'] = Area.objects.exclude(area=None)
admin.site.register(MountaineeringInfo, MountaineeringInfoAdmin)

This is pretty straightforward, and is indeed documented in the Django docs. Note that the request is passed into this method, so it's easy to perform some filtering based on some aspect of the request - the currently logged-in user, for example.

Filtering an inline's dropdown based on the inline instance

This is slightly more complicated. Let's say we're editing a Trip, and we're editing MountaineeringInfo instances by way of an inline. In code terms, we've got something like this:

class MountaineeringInfoInline(admin.TabularInline):
    model = MountaineeringInfo

class TripAdmin(admin.ModelAdmin)
    inlines = [MountaineeringInfoInline]

Now, referring back to our models, let's say we want to filter the available Landmarks for an inline depending on what Area is selected. There are two cases we have to consider: what happens when the inline is displayed but blank (ie. it's not bound to a MountaineeringInfo instance); and then, once we've got a MountaineeringInfo instance to bind to.

This can be solved using custom inline formsets. Let's extend the MountaineeringInfoInline class:

class MountaineeringInfoInline(admin.TabularInline):
    model = MountaineeringInfo
    formset = MountaineeringInfoInlineFormset

Note we've defined an extra attribute, specifying a custom formset. Let's go ahead and define that formset:

class MountaineeringInfoInlineFormset(BaseInlineFormSet):
    def add_fields(self, form, index):
        super(MountaineeringInfoInlineFormset, self).add_fields(form, index)
        landmarks = Landmark.objects.none()
        if form.instance:
            try:        
                area = form.instance.area    
            except Area.DoesNotExist:
                pass   
            else:  
                landmarks = Landmark.objects.filter(point__within=area.area)
        form.fields['base_camp'].queryset = landmarks

Here, we override the inline formset's add_fields() method which - unsurprisingly - is called to generate all the fields that will appear in the inline formset. Note that the form is passed in as an argument. Since this is a ModelForm, the underlying instance (which will be a MountaineeringInfo instance, remember) is available using the instance attribute on the form. Now, if Django is generating a new, blank inline formset, then form.instance will be None. In this case, we don't want any landmarks to display - we want the user to have chosen an area first. Hence, we assign an empty QuerySet to the base camp field on the form.

On the other hand, if instance is set, then we have an existing MountaineeringInfo instance to work with. In this case, we get the area associated with it (note that area is nullable, so we have to wrap the access in a try/except to guard against the possibility that no area has been set) and create a QuerySet of all landmarks whose point lies within the area. So, when a user selects an area from the dropdown and presses Save, the landmarks contained in the base camp dropdown filter themselves to only those within the specified area.

Depending on your app, you might want the default value for landmarks to be Landmark.objects.all() rather than none() as per the example above - if so, remember that all() is the default, so you could eliminate some of that code.

The only thing to be aware of with the above code is if a user selects an area and a landmark, then changes the area so that the selected landmark is no longer valid for the area, the old landmark will remain set in the database. Of course, the base_camp dropdown would be blank, and would be reset to None when Save was pressed. If this were a problem, it would be possible to set the queryset to be Landmark.objects.filter(pk=form.instance.base_camp.pk).

Filtering an inline's dropdown based on the parent

OK, confession first - this feels like a hack. But I haven't found a cleaner way to do it - let me know if you know how to!

Note the Trip model has an Area foreign key as well. How might we go about filtering the 'area' dropdown in new MountaineeringInfo instances to only contain areas that are within the parent Trip's area?

Well, there are two parts to this:

1. Figuring out what the area of the parent Trip is
2. Filtering our own area dropdown depending on this values

We've actually done most of the work necessary to understand how to do the second part already. So let's do that part first. The key thing to know is that BaseInlineFormset-derived instances, like ModelAdmin instances, have a formfield_for_dbfield method. We can therefore override this to restrict the queryset used in fields contained within the inline. Let's extend our existing definition:

class MountaineeringInfoInline(admin.TabularInline):
    model = MountaineeringInfo
    formset = MountaineeringInfoInlineFormset
    
    def formfield_for_dbfield(self, field, **kwargs):
        if field.name == 'area':
            # Note - get_object hasn't been defined yet
            parent_trip = self.get_object(kwargs['request'], Trip)
            contained_areas = Area.objects.filter(area__contains=parent_trip.area.area)
            return forms.ModelChoiceField(queryset=contained_areas)
        return super(MountaineeringInfoInline, self).formfield_for_dbfield(field, **kwargs)

As you can see - very similar to what we've seen before. We use the get_object call to extract the Trip that this MountaineeringInfo instance is (or will be) related to, and find all Area instances which are contained by that parent Trip's area.

So, what does that get_object() method look like?

    def get_object(self, request, model):
        object_id = request.META['PATH_INFO'].strip('/').split('/')[-1]
        try:
            object_id = int(object_id)
        except ValueError:
            return None
        return model.objects.get(pk=object_id)

This clearly isn't ideal, as it depends on the URL structure used by the Django admin: it extracts the object ID by stripping off slashes, splitting on slashes, and taking the last element. It then looks up the appropriate object using the model class passed on.

class MountaineeringInfoInline(admin.TabularInline):
    model = MountaineeringInfo
    formset = MountaineeringInfoInlineFormset

    def formfield_for_dbfield(self, field, **kwargs):
        if field.name == 'area':
            # Note - get_object hasn't been defined yet
            parent_trip = self.get_object(kwargs['request'], Trip)
            contained_areas = Area.objects.filter(area__contains=parent_trip.area.area)
            return forms.ModelChoiceField(queryset=contained_areas)
        return super(MountaineeringInfoInline, self).formfield_for_dbfield(field, **kwargs)

    def get_object(self, request, model):
        object_id = request.META['PATH_INFO'].strip('/').split('/')[-1]
        try:
            object_id = int(object_id)
        except ValueError:
            return None
        return model.objects.get(pk=object_id)

(In the real app code, that get_object() is in a base class, for easier reuse - hence the parameterisation of the model.)

Can you do better?

This kind of filtering is often required in non-trivial applications: be it filtering on security (which is relatively easy, as the request is usually present in most ModelAdmin APIs) or filtering on other data values - which seems a lot trickier than you might want. However, once you understand how the various ModelAdmin classes, inlines and formsets fit together, it's not too bad.

I'm keen to hear how you're tackling this in your Django apps - and whether this can be simplified!

Swoop Travel is a GeoDjango site using PostgreSQL and PostGIS, although very little GIS functionality is in the public-facing site at the moment - it's mainly in the backend.

We built this from scratch in about a month (while juggling other projects too!) are are pretty pleased with how it's turned out. This is just the first iteration and we're looking forward to expanding the site.

GeoDjango is pretty awesome. As we work on the site and get more experience, I'll post some more about some of the innards: there's quite a lot of interesting stuff in the backend, particularly admin customisations like filtering dropdowns based on landmarks within geographic regions, and so on. I'll also try to talk about the production server configuration, as GeoDjango needs a slightly different WSGI configuration to the standard run-of-the-mill Django site.

Buildout

Buildout, developed originally by Jim Fulton at Zope Corporation, was an attempt to address this problem. zc.buildout 1.0.0, released back in January 2008 (after over a year of betas) has become the accepted way of managing the hundreds of packages that make up Zope, BlueBream and Plone. Buildout isn't restricted to Zope, of course - it's usable for any setuptools-aware Python software.

Buildout is based around configuration files, most commonly a single file called buildout.cfg. Buildout itself doesn't do very much; most functionality is provided by add-on modules. These modules are called 'recipes'. They're easy to write, and there are dozens of custom recipes to perform specialised tasks. However, the key piece of functionality that buildout offers is to allow the developer to simply list the packages (eggs) required by their application, and optionally their versions. Minimum, maximum and ranges of acceptable versions can be specified, and buildout will do its best to satisfy version requirements.

Buildout is based on setuptools (which is in turn based upon distutils, part of the Python standard library). It uses setuptools to do the heavy lifting of package search and installation. Buildout environments are isolated from the system Python: packages installed via buildout don't end up anywhere near system Python's site-packages directory.

pip and virtualenv

Pip and virtualenv both come originally from the ever-productive Ian Bicking. pip is a replacement for easy_install, part of setuptools (as indeed pip is now): it offers a number of advantages over easy_install, **expand this**. Like buildout, pip also offers the ability to install a set of Python packages of a given version, using a requirements file. The requirements file lists which eggs to install, their versions, editable eggs checked out of source control, alternative mirrors for the location of packages, and so on.

pip installs into the current Python environment - which by default, will be the system Python. Pip is therefore commonly used with virtualenv, which isolates the Python environment.

buildout dependency resolution - build can fail halfway through due to version conflicts

A hybrid approach? gp.recipe.pip

Would be nice if it could provide data to feed mr.developer [sources], and [versions]

Conclusion

Choose pip - simplicity

Choose buildout - sophistication

File "/Users/dan/.eggs/djangorecipe-0.20-py2.6.egg/djangorecipe/recipe.py", line 271, in install_release
    os.listdir(extraction_dir)[0]
OSError: [Errno 2] No such file or directory: '/Users/dan/.downloads/django-archive'

After a bit of poking around, I found that this is to do with a corrupted Django tarball. In my case, this is usually because I've interrupted a download with Ctrl-C. Unfortunately it seems that the tarfile module in the Python standard library (at least as invoked by setuptools) treats broken a tar.gz files as an empty archive, without throwing an exception. Since there's no exception, djangorecipe assumes everything was uncompressed without problems, and is therefore rather surprised when the unpacked Django package isn't where it expected it to be.

The short term solution is to delete the bad Django archive from your download cache. This will likely be a 'downloads' directory in your buildout, or you may have a global one (as I do). When you next run buildout, the tarball will be freshly downloaded.

When I get a moment I'll see if I can modify djangorecipe to notice this condition and not proceed with the build.

Fast-forward to the start of 2010, and things have moved on. Plone's moved on, for sure. Plone 4 is just around the corner, and there's some really, really cool stuff in there: Dexterity, a new content types framework, finally looks like it'll make content type creation as easy as it should be, and simple types and behaviours can be created through the web. Deco, slated (last time I heard) for Plone 5, is quite literally going to make publishers wet themselves. And Deliverance has got great potential in helping to unify the many disparate systems which makes the typical corporate user's daily life such a grind. Thing is, I'm not a big corporate user; a lot of what makes Plone great for that environment is overhead for me and my little blog.

The big change for me personally, though, is that more and more of my work is now Django. About two thirds of 2009 was Zope 2/Five, and a third Django; 2010 is looking like it'll be the other way around. So, having decided that my blog was looking a little tired, and seemed to be a bit of a spam magnet, I decided to bite the bullet and do a complete rebuild in Django.

I didn't want to write yet-another-blog from scratch, so I picked what seemed to have the most buzz around it at the time - Django Mingus - and started from there. And this is how I did it.

Oh, before I get stuck in, all the code for this site is open source. Fortunately, there's not much of it. There's no documentation as such (this is open source, after all) because the target audience is, well, me. I've stayed true to my Zope roots and used buildout; the buildout and Django project can be checked out from GitHub:

Stereoplex buildout and project

Onwards.

Planning

There's more to moving a blog than just installing your blog software and bashing out new articles. I had to consider a number of migration issues, specifically:

• Obviously, I need to move all the content over from my old blog. That's not just articles: that's images, comments, the whole shebang.
• I needed to either keep all the old article URLs from my previous blog working, or have them redirect to the new URLs.
• Similarly, there are lots of RSS URLs out in the wild; I've handed a couple of them out to the Django community aggregator and the Planet Plone aggregator. There's also the main site feed.
• I decided not to move user accounts over, as I'd already decided that I wasn't going to require a login to comment. I was going to go with a ReCaptcha integration.

So, the first step was obviously going to be to extract all the content from my old blog.

Getting the content out of the ZODB

Plone stores data in the ZODB. The ZODB is amazing. It was years ahead of its time, and provides a really natural way to store and interact with data in document-oriented systems. It's only really in the last year or so that we've seen the rise of broadly similar data stores, most of which talk HTTP and don't have the fine-grained transaction control that ZODB provides. The only thing to remember with the ZODB is that you can only access it directly with Python. That meant that I had to write a Python script to dump out all my content into a platform-neutral form. (I could have written a script which read the ZODB directly and created Django models, I guess. Using an XML intermediate format seemed easier though, and was probably quicker to work with - loading up the Plone 2.5 codebase is pretty slow.)

Most Plone applications, SimpleBlog included, stored its data using the Archetypes content type framework. Archetypes (often just AT) is a schema-driven approach to creating content. This is a pretty handy approach (even if AT's implementation was a bit awkward in places) as it's really simple to write code that can introspect that content. This was invaluable for writing an export script.

The result is on github: Simple-AT-XML-Dump. I should note that AT does have support for native XML dump and load. Thing is, I've got an ancient version of AT, and SimpleBlog used CMF (a layer underneath Plone) comments. I had no idea if XML marshalling would work properly, so I just did my own custom XML format. It's an instance script. It should work pretty well on any folderish Archetypes types. Invoke it like this:

bin/instance run Simple-AT-XML-Dump/run.py -p /zodb/path/to/plone/content -o out.xml

/zodb/path/to/plone/content is the physical path in the ZODB to dump. This needs to be the root folderish AT item. out.xml is the output file.

I won't go into the specifics of how the script works (if there's anything you're interested in especially, mail me or post a comment below). In a nutshell though, it'll dump out AT types by schema (including ImageFields with base64-encoded data) and any CMF discussions that are associated with them.

Starting with Django Mingus

Many years of pain have taught me to automate my build. The XML dumper instance script above is pretty much the smallest thing I'll do without a build (and even now, I feel a twinge of guilt at not having packaged it properly.) For me, therefore, the first thing to do was to get a build up and running.

The system du jour seems to be to use pip and virtualenv. Pip (a replacement for easy_install) lets you specify requirements files, which define which Python packages you application uses, what versions of them, and lets you install directly from the major source control systems. (This ability seems to have caused lots of requirements files with github links in them to spring up in 'released' packages, especially in the Django world. I regard this as a Bad Thing; but that's an opinion piece for another time). However, grizzled Zope veterans tend to reach for buildout in such a circumstance so, wanting to restrict New Fangled stuff to learning how Mingus was put together, I stuck with that. Buildout predates pip and virtualenv, and so does stuff that you'd normally just do with virtualenv, like creating an isolated environment. Buildout config files are also a touch more verbose than requirements files. That said, I'm still pretty sure that buildout is the more extensible system, with a vast array of custom recipes; and, in common with a lot of software from the Zope world, the narrative documentation is atrocious.

But I know buildout, and I'm not giving up these scars so easily, so that's what I went with, basing my buildout config on the requirements file that comes with Mingus. This process was actually pretty simple:

1. Set up a standard Django buildout using djangorecipe
2. Pop mr.developer in as an extension, add [sources] for all of the editable (-e) eggs in the pip requirements file, and add them all as auto-checkout
3. Put all the non-editable eggs from the requirements file into the eggs section of the buildout config
4. Use the versions supplied in the requirements file to create a [versions] section in the buildout config

This gives me a buildout.cfg file that mirrors the Mingus requirements file, but will also:

• Create the django management script and django WSGI file automatically
• Create a project and select an appropriate settings file for me

The end result of this is the GitHub project I linked to above, stereoplex-buildout.

Stereoplex

I created another egg, called stereoplex, to contain all my site-specific customisations and scripts. I used another package of mine, fez.djangoskel, to create the basic layout. Specifically, it has:

• A Django management command to import the XML file created with Simple-AT-XML-Dump
• A Django ModelAdmin subclass (actually a basic.blog.admin.PostAdmin subclass) to let me use TinyMCE as my editor
• A ReCaptcha Django form field, widget, and custom comment form
• An single extra view, which returns all items posted on the blog
• A URLConf which brought together Mingus' URLs plus those for the extra view, and for TinyMCE
• And of course, all the template overrides and CSS, JavaScript and images required for the new Stereoplex look and feel

Importing the Data

The next step was to write a data importer. This had to do a number of things (data migrations are never simple!):

• Import all the images in the XML data file, creating basic.media.models.Photo instances for each of them
• Rewrite all image links in the body text of posts to contain <inline> elements used by Mingus
• Create basic.blog.models.Post instances for each blog post in the file
• Create django.contrib.comments.models.Comment instances for every comment, and associate them with the appropriate post
• Create django.contrib.redirect.models.Redirect objects for each imported post, to allow existing inbound links to be redirected to the new location.

Automated content import is one of those things that you tell clients is usually impossible. And when they're migrating from a legacy CMS platform (or indeed, hand-maintained HTML), then that's usually right. The inevitable gigabytes of hand-rolled HTML are at best poorly formed, and at worse represent content which needs throwing away anyway.

I was more fortunate. I didn't have that much content to migrate - 60-odd posts, and a few images - and Plone 2.5's default editor Kupu is actually pretty good at producing good HTML. I was able to directly use the existing HTML, and only needed to replace the <img> tags with the appropriate <inline> expected by Mingus.

Changes to Mingus packages

I did make some modifications to Mingus' packages. These were as follows:

django-mingus

• Improve the way subscription links are generated
• On a category page, include the RSS link for that category in the page header
• Hide teaser-related markup if there's no teaser

django-basic-apps

• Fix a bug where a template tag library was not loaded

django-sugar

• Enhance the pygmentize filter to not be restricted to <code>

All really very minor.

Server Configuration

Apache

The Plone instance used a standard small setup: a single ZEO client talking to a ZEO server, fronted by Apache with the magic RewriteRule. The Zope configuration had been tweaked slightly to be usable on a small (by Plone's standard!) 512MB RAM host, but I hadn't had time to do any other optimisations (for example, serving static files from Apache) that I would normally do.

Django forces you to do at least some of these. Static files are always served by an external web server (unless you are really determined to sail through all the warnings in the documentation). I also finally got around to turning on gzip compression.

One remaining task that I planned to do in the Apache configuration was to provide redirects for my RSS feeds. This wasn't as easy as I might have liked, since the old feed URLs had query string elements; and indeed, it was some of those values that I needed to formulate a correct redirect. I ended up with the following:

RewriteEngine On
RewriteCond %{query_string} ^(.*)?EntryCategory=(.*)?&(.*)
RewriteRule ^/search_rss /feeds/categories/%2/ [R=permanent,L,NC]

This simply declares three match groups in the RewriteCond regex, the second of which (hence %2) is the category slug. The RewriteRule then issues a permanent redirect to the new URL. We have to use RewriteCond, because RewriteRule regexes won't match a query string, only the URL path.

Next up is the mod_wsgi configuration. I use a fairly standard configuration, which is generally as follows:

WSGIScriptAlias / /var/websites/www.stereoplex.com/bin/django.wsgi
WSGIDaemonProcess stereoplex user=stereoplex group=stereoplex processes=3 threads=25 maximum-requests=1000 stack-size=524288
WSGIProcessGroup stereoplex

This runs the Stereoplex web application in its own process group, and with its own user and group membership. This is a safety net: if the site is compromised or (to be honest, more likely) I screw up and the app tries to write to the filesystem, its rights are limited by the host's file access control. The WSGI script file is generated by buildout. Probably the most interesting parameter here is the stack size. This is set lower than Linux's default value; for this sort of app, it doesn't need to be as large as the default. Setting this value to lower than the default led to a massive memory saving (remember, this is only a 512MB host; and this site is one of around half a dozen running on the same machine).

Memcached

Mingus makes extensive use of Django's caching support. Other Django sites on the same host use a Memcached instance, so I just pointed Stereoplex at that one. Memcached is essentially in a default configuration, except only bound to the localhost IP address, rather than the public IP address. It's configured to use a maximum of 64MB of RAM. This doesn't sound a great deal, but even with two or three websites using it, I haven't seen it go over 40MB.

So... All Done?

Nearly.

If I'm honest, the content editing experience isn't as nice as Django as it was in Plone. This is expected. Plone is a CMS, and has an administrative interface that's focussed on the business of managing content. Django isn't a CMS, it's a more general web framework. The experience is more like editing content in the ZMI.

That said, there are advantages. I've found third-party Django software much easier to integrate and customise than third-party Plone products ever were. I'm not fighting reams of configuration all the time. There's a strong mindset of developing apps to be reusable in the Django world; Mingus itself is little more than some UI glue (as, really, is Stereoplex). If I were tasked with delivering a large CMS with flexible authentication and authorisation, workflow, and so forth: I'd go for Plone in an instant. But for this job, many of Plone's strengths simply don't apply, and somthing more simple and lightweight was more appropriate.

Anyway - I'm quite pleased with the results. There are still a few kinks to be worked out (pygementize only seems to be being applied on the home page, not individual post pages, for example) but I'll get there over the next couple of weeks.

If there's anything you'd like to have more information on, then leave a comment (click on the article heading - yes, a proper link to comments is on the list!) or of course, just go and grab the code at Github.

Enjoy!

OK, let's take a step away from text for a moment. I want you to think of a number between one and ten. Got one? Great - now, grab a pen and paper, and write it down.

What number did you think of? Well, I thought of the number six. And when I wrote it down, it looks like this:

Of course, if I were an ancient Roman (or possibly a clockmaker), I could have written this:

They all mean the same thing - the number six. But we've written them in different ways. In other words, we've 'encoded' our idea of the number six in our head in three different ways - three different encodings.

The separation of the idea of 'the number six' from its actual representation is basically all Unicode is. The Unicode Character set (UCS) defines a set of things (loosely, a set of letters) that we can represent. How we represent each of those letters is called an encoding. There's only one Unicode, but there are many encodings. In Unicode parlance, each of those 'things' (letters) are known as 'code points'. Unicode separates the characters' meaning from their representation.

For historical reasons, the most common encoding (in Western Europe and the US, anyway) is ASCII. This is also Python's default encoding.

Let's think about ASCII for a moment. It's an encoding that uses 7 bits, which limits it to 128 possible values. That's enough to represent all the characters that Western Europe and the US use (letters in both cases, the numbers, punctuation, a few characters with diacritics). Therefore, Unicode strings that only include code points that are in these 128 ASCII characters can be encoded as ASCII. Conversely, any ASCII encoded string can be decoded to Unicode.

It's worth reiterating that terminology, as you come across it a lot: the transformation from Unicode to an encoding like ASCII is called 'encoding'. The transformation from ASCII back to Unicode is called 'decoding'.

    Unicode  ---- encode ----> ASCII
    ASCII    ---- decode ----> Unicode

Non-ASCII encodings

Most people don't live in the US or Western Europe, and therefore have a requirement to store more characters than can be represented with ASCII. What those folk need to represent *is* part of the Unicode set (Unicode is massive!) - so a different encoding is required. Common encodings have familiar names: UTF-8 and UTF-16. UTF-8, for example, uses a single byte for encoding all the ASCII values, then variable numbers of bytes to encode further characters. (The ins and outs of these encodings are beyond the scope of this article - check out their respective Wikipedia entries for the gory details.)

The fact that the first byte of UTF-8 isthe same as ASCII is important, since it means that the encoding is backwards-compatible with ASCII. However, it can mask problems in software. We'll come to this shortly.

Some terminology

Unicode-related terminology can get confusing. Here's a quick glossary:

• To encode
• Encoding (the verb) means to take a a Unicode string and produce a byte string
• To decode
• Decoding (the verb) means to take a byte string and produce a Unicode string
• An encoding
• An encoding (the noun) is a mapping that describes how to represent a Unicode character as a byte or series of bytes. Encodings are named (like 'ascii', or 'utf-8') and are used both when encoding (verb!) Unicode strings and decoding byte strings.

In other words, when you encode or decode, you need to specify the encoding that you're using. This will become clearer shortly.

Python, bytes and strings

You've probably noticed that there seems to be a couple of ways of writing down strings in Python. One looks like this:

  'this is a string'

Another looks like this:

  u'this is a string'

There's a good chance that you also know that the second one of those is a Unicode string. But what's the first one? And what does it actually mean to 'be a Unicode string'?

The first one is simply a sequence of bytes. This byte sequence is, by convention, an ASCII representation (ie. encoding) of a string. The whole Python standard library, and most third-party modules, happily deal with strings natively in this encoding. As long as you live in US or Western Europe, then that's probably fine for you.

The second one is a representation of a Unicode string. This can therefore contain any of the Unicode code points. It's possible that whatever you're using to edit the Python code (or just view it) might not be able to display the entire Unicode character set - for instance, a terminal usually has an encoding that it assumes data it's trying to display is in. There's a special notation, therefore, for representing arbitrary Unicode code points within a Python Unicode string: the \u and \U escapes. These will be followed by four or eight hex digits; there's some subtlety here (see the Python string reference for further information) but you can simply think of the number after the \u (or \U) representing the Unicode code point of the character. So, for example, the following Python string:

  u'\u0062'

represents LATIN SMALL LETTER B, or more simply:

  u'b'

To summarise then: the Unicode character set encompasses all characters that we may wish to represent. Individual encodings (ASCII, UTF-8, UTF-16, etc.) are representations of all or some of that full Unicode character set.

Encoding and Decoding

Byte strings and Unicode strings provide methods to perform the encoding and decoding for you. Remembering that you *encode* from Unicode to an encoding, you might try the following:

>>> u'\u0064'.encode('ascii')
'd'

As you'd expect, the Unicode string has an 'encode' method. You tell Python which encoding you want ('ascii' in this case, there are lots more supported by Python - check the docs) using the first parameter to the encode() call.

Conversely, byte strings have a decode() method:

>>> 'b'.decode('ascii')
u'b'

Here, we're telling Python to take the byte string 'b', decode it based on the ASCII decoder and return a Unicode string.

Note that in both these previous cases, we didn't really need to specify 'ascii' manually, since Python uses that as a default.

UnicodeEncodeError

So, we've established that there are encodings which can represent Unicode, or more usually, a certain subset of the Unicode character set. We've already talked about how ASCII can only represent 128 characters. So, what happens if you have a Unicode string that contains code points that are outside that 128 characters? Let's try something all too familiar to UK users: the £ sign. The Unicode code point for this character is 0x00A3:

>>> u'\u00A3'.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' 
in position 0: ordinal not in range(128)

Boom. This is Python telling you that it encountered a character in the Unicode string which it can't represent in the requested encoding. There's a fair amount of information in the error: it's giving you the character that it's having problems with, what position it was at in the string, and (in the case of ASCII) it's telling you that the number it was expecting was in the range 0 - 127.

How do you fix a UnicodeEncodeError? Well, you've got a couple of options:

• Pick an encoding that does have a representation for the problematic character
• Use one of the error handling arguments to encode()

The first option is obviously ideal, although its practicality depends on what you're doing with the encoded data. If you're passing it to another system that (for example) requires its text files in ASCII format, you're stuck. In that case, you're left with one of the other two options. You can pass 'ignore', 'replace', 'xmlcharrefreplace' or 'backslashreplace' to the encode call:

>>> u'\u0083'.encode('ascii', 'ignore')
''
>>> u'\u0083'.encode('ascii', 'replace')
'?'
>>> u'\u0083'.encode('ascii','xmlcharrefreplace')
'ƒ'
>>> u'\u0083'.encode('ascii','backslashreplace')
'\\x83'

If you choose one of those options, you'll have to let the eventual consumer of your encoded text know how to handle these.

UnicodeDecodeError

This one is probably more familiar to most developers. A UnicodeDecodeError occurs when you ask Python to decode a byte string using a specified encoding, but Python encounters a byte sequence in that string that isn't in the encoding that you specified (phew!). This one probably benefits from an example.

Consider once more the ASCII encoding. Being a 7-bit representation, ASCII only has 127 characters, represented by the numbers 0 - 127. So let's imagine the ASCII-encoded string below:

'Hi!'

In terms of ASCII numbers, that is:

72 105 33

Or in actual Python:

>>> s = chr(72) + chr(105) + chr(33)
>>> s
'Hi!'
>>> s.decode('ascii')
u'Hi!'

That's all great. But what happens if we add a byte that's not in the ASCII range?

>>> s = s + chr(128)
>>> s.decode('ascii') 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 
in position 3: ordinal not in range(128)

Boom. Python is saying that it encountered a character 0x80 (which is 128 in hex, the one we added) which was at position 3 (counting from zero) in the source byte string which was not in the range 0 - 127.

This is normally caused by using the incorrect encoding to try to decode a byte string to Unicode. So, for example, if you were given a UTF-8 byte string, and tried to decode it as ASCII, then you might well see a UnicodeDecodeError.

But why only might?

Well, remember what I mentioned before - UTF-8 shares the first 127 characters with ASCII. That means that you can take a UTF-8 byte sequence, and decode it with the ASCII decoder, and *as long as there are no characters outside the ASCII range* it will work. *Only* when that byte string starts featuring characters which don't exist within the ASCII encoding do errors start being thrown.

ASCII - the default codec

Lots of Python programmers (well, US and Western European ones) can get quite a way into their Python careers converting byte strings to unicode like this:

>>> print unicode('hi!')
u'hi!'

What's going on here? Well, Python uses the ascii codec by default. So, the above is equivalent to:

>>> 'hi!'.decode('ascii')
u'hi!'

And, because most US/European test data is composed of this byte string:

  'test'

... nobody notices the problem until the Japanese office complains the intranet is broken.

Unicode Coercion

If you try to interpolate a byte string with a Unicode string, or vice-versa, Python will try and convert the byte string to Unicode using the default (ie. ascii) codec. So:

>>> u'Hi' + ' there'
u'Hi there'
>>> u'Hi %s' % 'there'
u'Hi there'
>>> 'Hi %s' % u'there'
u'Hi there'

These all work fine, because all the strings that we're working with can be represented with ASCII. Look what happens when we try a character which can't be represented with ASCII though:

>>> u'Hi ' + chr(128)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 
in position 0: ordinal not in range(128)

Python sees we're trying to combine a Unicode string with a byte string, so tries to decode the byte string to Unicode using the ASCII codec. Since character 128 (the Euro symbol, as it happens) can't be represented in ASCII, Python throws a UnicodeDecodeError.

In my experience, Unicode coercion is often where UnicodeDecodeErrors manifest themselves. The programmer has a Unicode string (probably a template) into which they're trying to put some data from a database. Relational databases tend to supply byte strings. Usually the encoding is a property on the database connection. Often, however, developers simply assume it's ASCII (or don't do anything special at all, which in Python amounts to the same thing). They try to stick the data from the database (perhaps in UTF-8 or ISO-8859-1) into a Unicode string using the %s format specifier, Python tries to decode the byte string using the ascii codec, and the whole thing falls flat on its face.

Why do Python byte strings have an encode() method?

The sharp-eyed amongst you will have noticed that byte strings have an encode() method as well as a decode() method. What does this do? Quite simply, it does a decode-then-encode. The byte string is decoded to Unicode using the default (ascii) encoding, and is then encoded to the target encoding specified in the call to encode() using the appropriate encoding. As you'd expect, fun and games ensue if the original byte string isn't actually encoded in ASCII at all.

Avoiding Unicode Errors

So - this is really what you care about, right? How do you avoid these Unicode problems? Well, there are three simple rules:

• Within your application, always use Unicode
• When you're reading text in to your application, decode it as soon as possible with the correct encoding
• When you're outputting text from your application, encode at that point and do it explicitly

What does this mean in practice? Well, it means:

• Whenever you're writing string literals in code, always use u''.
• Whenever you read any text in, call .decode('encoding') on the byte string to obtain Unicode
• Whenever you're writing text out, pick an appropriate encoding to handle whatever Unicode you're outputting - remember that ASCII can only represent a very limited subset

There are more places than you probably realise that text can get into your application. Here's some:

• An incoming request from a web browser
• Some text read in from a data file on disk
• A template file read in from disk
• Some user's input from a form
• Some data from a database
• Data returned from a web services call

Frameworks help a lot here. Many frameworks handle the common encoding and decoding cases (usually the template encoding, and data encoding from a database) for you, and just pass you back Unicode strings. Watch out for web request variables - many of those may be plain byte strings. Also watch out for web service responses; you might need to inspect the response headers to find out the encoding. And even then be careful; I've come across situations with in-house apps where declared encoding were simply wrong, leading to unexpected UnicodeDecodeErrors.

Figuring out which encoding to use

When you're faced with a byte string, how do you know which decoding to use? The answer is, unfortunately, simple: you don't. Some environments (such as the Web) may help you - HTTP requests and responses contain headers which specify the encoding used within them. You can inspect those, and if they're wrong - well, at least you've got someone else to blame.

If you're lucky, you know the byte string is encoding some XML. XML is gets a lot of flack, but one of the things it does right is to specify explicitly a default encoding that's actually useful (UTF-8) and provide a mechanism to declare a different encoding. So with XML, you can scan the first few bytes of the file, decode using UTF-8, and look for the magic encoding declaration. If there isn't one, then you can safely decode the rest of the file using UTF-8. If there is one, then switch encoding. Of course, your XML library of choice will do all this for you, and should give you Unicode text back once you've read your XML in.

If you're unlucky, then you've got two more options. First off, you can talk to the people who run your source (or destination) system - find out what encodings they're using, or accept, and use those.

The final, last resort option is to simply have a range of common encodings to try. A list I often use is ASCII, ISO-8859-1, UTF-8, UTF-16. Keep trying to decode with each of those in turn until one works. Which encodings you pick of course depends on what kind of files you're expecting to see. You may also run into problems of course if you have a byte string in encoding X which also happens to be valid when decoded using encoding Y - in this case, you'll just get garbage data. This is the cause of many of the 'funny character' bugs you see in web applications: byte strings being decoded using an encoding which happened to work, but was in fact not the original encoding used to create the byte string.

Python 3

I'm not going to talk too much about Python 3, since I haven't actually used it yet.

But - you rarely hear .NET or Java programmers complaining about Unicode errors. This is simply because both .NET and Java define a string to *be* Unicode in the first place. Anything involving the String class (in either runtime) is Unicode anyway; the developer sees encoding problems much less frequently as it's much less common for unexpected byte data to creep into applications. This doesn't mean the problems don't exist, of course: at the end of the day, text is still being encoded to and from byte strings; it's just done explicitly. (The fact that the default encoding on MS Windows, the OS on which many of these systems run, is UTF-16 helps here too - many more characters can be encoded in UTF-16 than ASCII).

My understanding is that Python 3 takes this general approach. Python 2's 'str' type is gone. In its place is the 'unicode' type (equivalent to Java and .NET's String class), and the 'bytes' type. String operations are done on 'unicode' instances.

Coding in a Unicode world

Unicode is here to stay. The days of writing software that would only need to work in American universities, where the only language and script used was US English in Latin text are long gone. There's no magic to Unicode and the various encodings, and once you understand what's going on, there's no reason to have that sick feeling in the pit of your stomach the next time you see a UnicodeDecodeErrror. Just remember these rules:

• Decode on the way in
• Unicode everywhere in your application
• Encode on the way out

I've been working on a buildout to get at least most of the steps done for you. There are a couple of manual steps at the end, which I hope to automate when I next have time to work on this.

The buildout installs the following items:

• PostgreSQL
• PostGIS
• GDAL
• Proj
• GEOS
• psycopg2
• Django

It should also perform initial setup of the PostGIS database template, loading some sample SQL files, and sets up some convenience symlinks for the PostgreSQL command-line programs.

It's not finished - in particular, it just assumes that the user running the buildout is to be used as the database owner and such like. Anyway, here it is:

[buildout]
parts =
  postgresql
  postgis
  gdal
  init-pgsql
  pgsql-symlinks
  django
  
eggs =
    psycopg2

[postgresql]
recipe = zc.recipe.cmmi
url = http://wwwmaster.postgresql.org/redir/198/h/source/v8.3.7/postgresql-8.3.7.tar.gz
extra_options =
  --with-readline
  --enable-thread-safety
  
[postgis]
recipe = hexagonit.recipe.cmmi
url = http://postgis.refractions.net/download/postgis-1.3.5.tar.gz
configure-options =
    --with-pgsql=${postgresql:location}/bin/pg_config
    --with-geos=${geos:location}/bin/geos-config
    --with-proj=${proj:location}

[proj]
recipe = zc.recipe.cmmi
url = http://download.osgeo.org/proj/proj-4.6.1.tar.gz

[geos]
recipe = zc.recipe.cmmi
url = http://download.osgeo.org/geos/geos-3.0.3.tar.bz2

[gdal]
recipe = zc.recipe.cmmi
url = http://download.osgeo.org/gdal/gdal-1.6.0.tar.gz
extra_options = 
    --with-python
    --with-geos=${geos:location}/bin/geos-config

[init-pgsql]
recipe = iw.recipe.cmd
on_install = true
on_update = false
cmds = 
    ${postgresql:location}/bin/initdb -D ${postgresql:location}/var/data -E UNICODE
    ${postgresql:location}/bin/pg_ctl -D ${postgresql:location}/var/data start
    sleep 30   
    ${postgresql:location}/bin/createdb -E UTF8 template_postgis
    ${postgresql:location}/bin/createlang -d template_postgis plpgsql
    ${postgresql:location}/bin/psql -d template_postgis -f ${postgis:location}/share/lwpostgis.sql
    ${postgresql:location}/bin/psql -d template_postgis -f ${postgis:location}/share/spatial_ref_sys.sql
    ${postgresql:location}/bin/psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
    ${postgresql:location}/bin/psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"
    ${postgresql:location}/bin/pg_ctl -D ${postgresql:location}/var/data stop

[pgsql-symlinks]
recipe = cns.recipe.symlink
symlink_target = ${buildout:directory}/bin
symlink_base = ${postgresql:location}/bin
symlink =
    clusterdb
    createdb
    createlang
    createuser
    dropdb
    droplang
    dropuser
    ecpg
    initdb
    ipcclean
    pg_config
    pg_controldata
    pg_ctl
    pg_dump
    pg_dumpall
    pg_resetxlog
    pg_restore
    postgres
    postmaster
    psql
    reindexdb
    vacuumdb

[django]
recipe = djangorecipe
version = 1.0.2
project = project
eggs =
    ${buildout:eggs}

Note that running this will actually attempt to start up and shut down the database server, as it needs to be running in order for some of the initialisation scripts to run. That 'sleep 30' in the middle is to allow the database server to start, and (if you're on OS X and running it) to give you a change to enter your username and password for the firewall!

There are still some manual steps to be taken (which I'd like to automate in due course). These are the fairly standard things that you do when starting any Django project, plus an extra step for bootstrapping PostGIS.

Create your database

From the command line, you'll need to create the database for you application. You need to specify the PostGIS template, so use something like:

$ bin/createdb -T template_postgis <db name>

Change the settings for your application

Edit the settings.py for your application, and make sure that you're using 'postgresql_psycopg2' as the database engine. Set the database name as appropriate for your application. You should also add 'django.contrib.gis' to your INSTALLED_APPS setting, and you'll also need to add the following two lines to your settings.py:

GDAL_LIBRARY_PATH = '/path/to/buildout/parts/gdal/lib/libgdal.dylib'
GEOS_LIBRARY_PATH = '/path/to/buildout/parts/geos/lib/libgeos_c.dylib'

Add Google projection

I'll confess: I'm only doing this because the GeoDjango docs say you should! I don't know enough about GeoDjango yet to understand why. But you should do the following:

$ bin/django shell
>>> from django.contrib.gis.utils import add_postgis_srs
>>> add_postgis_srs(900913)
>>> ^D
$

If you get an error when importing add_postgis_srs, then double check you got the GDAL_LIBRARY_PATH and GEOS_LIBRARY_PATH correct, and that the files specified were built. (I'm on Mac OS X - I suspect the exact file name may change depending on platform.)

Done!

Once all that's done, you should hopefully be able to bin/django syncdb, start a new app (using fez.djangoskel, of course!) and start using GeoDjango.

I shall refine the above process over time (in particular, there are some modifications I'd like to make to djangorecipe to remove the manual steps at the end), and I'll post extra parts when I've done that.

The easiest way to get this is, as usual, with easy_install. (See my previous posts on how to set this up):

easy_install fez.djangoskel

Once that and its dependencies have installed, you should fine that paster can now create Django projects and apps:

$ paster create --list-templates
Available templates:
  basic_package:   A basic setuptools-enabled package
  django_app:      Template for a basic Django reusable application
  django_project:  Template for a Django project
  paste_deploy:    A web application deployed through paste.deploy

It's standard paste from here on in: to create a project, use:

paster create -t django_project

To create an app, use:

paster create -t django_app

Paster will then ask you a bunch of questions (the most crucial one being the name of your app/project!) and generate the file layout for you.

I plan to do another release later this week which includes additional templates for creating Django eggs using namespace packages, as well as improved documentation.

One answer is zc.buildout. Buildout is a tool for reliably creating reproducible software builds. It was originally developed by Zope Corporation, and is often used in Zope builds; however, there's no dependency on Zope. You can use it to build pretty much anything. And I'm going to show you how to get a Django build up and running using it.

I shall use PostgreSQL as the database in my examples, but there's nothing stopping you using MySQL or any other Django-supported database, if you wish.

You'll also need the standard development tools (gcc, etc.) available since we're going to be getting buildout to compile some binary eggs for us.

The Basics: Python and PostgreSQL

The only thing you need to get going is some version of Python installed. The system Python is probably fine, as long as it's version 2.3 or later. Get a database installed too: I'm using PostgreSQL here. (You can use buildout to install a database too; I'm not going to cover that here, though, since most people have their database installed through system packages.)

We are going to install two packages in the system python: setuptools and virtualenv.

(If you don't want to touch the system Python at all, that's fine; check out my earlier article on how to compile and install a local version of Python. You might want to do this if your system only offers an old version of Python. I do it as a matter of course, but then get shouted at by system admins who want to use a package manager to keep their Python up to date. Your mileage may vary.)

Download ez_setup.py, and run it as a user who can write to the Python's site-packages directory (root if you're using your system python):

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

Let's also take the opportunity to create a database for the project. This is for PostgreSQL; obviously, substitute whatever's appropriate for your platform:

createdb djangodevdb 

virtualenv

Before we get stuck into buildout, let's talk about virtualenv.

You're probably going to have more than one project on the go. You want to keep them separate from each other, and want to avoid polluting your system Python installation with application-specific third-party modules. This is what virtualenv does. It lets you create isolated sandboxes: modules installed in one virtualenv don't interfere with other virtualenv. Let's install virtualenv and create an environment for our Django environment:

easy_install virtualenv

That will download and install virtualenv. Next up, let's create ourselves a sandbox called 'djangodev' to work in:

hornet:2.4 dan$ virtualenv --no-site-packages djangodev
New python executable in djangodev/bin/python
Installing setuptools.............done.
hornet:2.4 dan$

Finally, we need to 'activate' the sandbox. This isolates the environment from the system Python, and ensures that any modules installed are local to this environment.

hornet:2.4 dan$ cd djangodev/
hornet:djangodev dan$ source bin/activate 
(djangodev)hornet:djangodev dan$ 

Note how the command prompt changes as a visual indicator that we're now working in a virtualenv. You can type 'deactivate' if you want to exit the virtualenv.

Initial Configuration

Getting going with buildout is very straightforward. Create a top-level directory to hold your application and download the bootstrap.py file into it:

mkdir app
cd app
wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py

Don't run this yet.

A Basic Buildout Configuration

Next, create a file in that same directory called buildout.cfg, and put the following into it:

[buildout]
parts =

This is pretty much the simplest buildout configuration you can start with.

Bootstrapping

The very first time you use buildout, you have to bootstrap it. This installs buildout itself, and generates scripts to run the buildout. Bootstrapping the buildout is simply a matter of running the bootstrap.py file you downloaded earlier. You should see output resembling this:

(djangodev)hornet:app dan$ python bootstrap.py 
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/bin'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/parts'.
Creating directory '/Users/dan/opt/virtual/2.4/djangodev/app/develop-eggs'.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/buildout'.
(djangodev)hornet:app dan$ 

That just creates buildout's initial scripts and directory layouts. You don't have to run it again for this environment.

Installing Django

So, all that was boring. And it probably seemed fiddly: after all, couldn't we just have installed Django in the virtualenv's site-packages manually? Yes, we could; but then we'd have had to do that every time we deployed an environment. We can now start to use buildout to automate our builds. Let's install Django 1.0 with buildout.

Open up your buildout.cfg again, and change it so that it looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0

Now, go ahead and run buildout. This will download the Django 1.0 distribution, so may take a few minutes depending on your connection:

(djangodev)hornet:app dan$ bin/buildout 
Unused options for buildout: 'download-directory'.
Installing django.
django: Downloading Django from: http://www.djangoproject.com/download/%s/tarball/
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
(djangodev)hornet:app dan$

Buildout (via the 'djangorecipe' extension) has done a couple of things for us:

• It has created a script called bin/django to run the django management commands
• It has created an inital Django project (called, imaginitively, 'project') for us with some default settings
• It has installed Django
  

Note that buildout created a script called 'django' in the bin directory. This script it the exact equivalent of django-admin.py, or running python manage.py when manually installing Django; that is, you can run bin/django syncdb, bin/django sqlall, eveything you would expect. So let's go ahead and try to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Traceback (most recent call last):
  File "bin/django", line 20, in ?
    djangorecipe.manage.main('project.development')
  File "/Users/dan/.buildout/eggs/djangorecipe-0.13-py2.4.egg/djangorecipe/manage.py", 
    line 15, in main
    management.execute_manager(mod)
 [ ... snip ... ]
  File "/Users/dan/opt/virtual/2.4/djangodev/app/parts/django/django/db/
    backends/sqlite3/base.py", line 26, in ?
    raise ImproperlyConfigured, "Error loading %s module: %s" % (module, e)
django.core.exceptions.ImproperlyConfigured: Error loading pysqlite2 module: 
    No module named pysqlite2

Well - that didn't go so well!

The problem here of course is that we've installed Django, but we haven't specified the database to connect to, or installed the python module required to connect to the database. Let's do both now.

Installing Dependencies

Configuring Django to connect to a database is well covered in the Django documentation, so for now I'll just tell you to edit the project/settings.py file and change DATABASE_ENGINE to 'postgresql_psycopg2', and to set your DATABASE_NAME, DATABASE_USER etc. appropriately for your installation.

Installing the connector is more interesting. The connector for PostgreSQL is called psycopg2. Let's tell buildout to install that. Open up your buildout.cfg again, and change it so it now looks like this:

[buildout]
parts = django

[django]
recipe = djangorecipe
version = 1.0
eggs = psycopg2

All we've done is add psycopg2 to the list of eggs to install with Django.

Now just rerun buildout:

(djangodev)hornet:app dan$ bin/buildout 
Uninstalling django.
Unused options for buildout: 'download-directory'.
Installing django.
Getting distribution for 'psycopg2'.
warning: no files found matching '*.html' under directory 'doc'
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: 
  warning: 'PyDateTimeAPI' defined but not used
psycopg/typecast.c:37: warning: 'skip_until_space' defined but 
  not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning: 
  'PyDateTimeAPI' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
./psycopg/config.h:63: warning: 'Dprintf' defined but not used
/Users/dan/opt/python-2.4.5/include/python2.4/datetime.h:186: warning: 
  'PyDateTimeAPI' defined but not used
zip_safe flag not set; analyzing archive contents...
Got psycopg2 2.0.8.
Generated script '/Users/dan/opt/virtual/2.4/djangodev/app/bin/django'.
django: Skipping creating of project: project since it exists

As you can see, buildout noticed that we'd specified an extra requirement of psycopg2 and so downloaded it from PyPI, compiled and installed it for us. What buildout did is essentially analagous to running 'easy_install psycopg2'. Now we should be able to run the Django development server:

(djangodev)hornet:app dan$ bin/django runserver
Validating models...
0 errors found

Django version 1.0-final-SVN-unknown, using settings 'project.development'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

It worked! You can now go ahead and check your buildout.cfg into source control. Anyone checking that out will get the same Django build as you.

PIL

PIL, the Python Imaging Library, is always a bit of a pain to install. Django requires it if you want to work with images. It's also not packaged with setuptools, let alone as an egg. How can we get this into our build?

Fortunately, Chris McDonough has repackaged PIL with setuptools, making it relatively straightforward to add to our build. Open up buildout.cfg again, and edit it so that is looks like this:

[buildout]
parts =
  PIL
  django

[django]
recipe = djangorecipe
version = 1.0
eggs =
  psycopg2
  markdown
  PIL

[PIL]
recipe = zc.recipe.egg
egg = PIL==1.1.6
find-links = http://dist.repoze.org/

Rerun buildout, as before, and PIL should be downloaded, compiled and installed.

That should be enough to get you going with buildout. You can find lots more on buildout and the Django recipe in the links below:

• http://buildout.zope.org/
• http://pypi.python.org/pypi/zc.buildout
• http://pypi.python.org/pypi/djangorecipe/
• http://plone.org/documentation/tutorial/buildout (Plone-oriented, but lots of useful background information there too)
• http://renesd.blogspot.com/2008/05/buildout-tutorial-buildout-howto.html
• http://wiki.python.org/moin/buildout/pycon2008_tutorial

Google found me those links, so I'm sure it can find more for you too! I'd particularly encourage you to read the djangorecipe documentation for more detail on how it can configure Django for you.

In future articles, I intend to talk about

• Starting Django applications as eggs
• Configuring applications as development eggs inside buildout
  
• Packaging applications and uploading them to PyPI, so that they're just an easy_install away
• Dealing with third-party Django applications which have not been packaged as eggs
• Using buildout to build non-python dependencies
• How all this works with source control
  

Until next time.

Django applications tend to have views, in a views.py file. These are generally looked up by URL. URLConfs (various urls.py files) are used to map request URLs to views.

This can lead to a problem from a testing perspective. Not all applications have urls.py files; and for those that do, there's nothing stopping a project wiring up different URLs to those views through a custom urls.py. It therefore becomes difficult to use Django's test client to invoke a view using a GET or a POST because you don't know how that URL has been configured. After all, you run python manage.py test from your project, and the project's configuration is used.

Let's say I have a views.py that looks like this:

from django.shortcuts import render_to_response
from django.template import RequestContext

def say_hi(request):
    return render_to_response('app/hi.html', context=RequestContext(request))

I want to use Django's test client to ensure that when this view is called, it returns an HTTP 200 OK response. However, my app doesn't have a urls.py - the person using it is meant to wire this view into whatever URL they want to. How do I test the view? 

The solution is deceptively simple: inject a test URLs module.

Create a base test case that looks something like this:

from django.conf import settings
from django.test import TestCase

class TestUrlsTestCase(TestCase):

    def setUp(self):
        self._old_root_urlconf = settings.ROOT_URLCONF
        settings.ROOT_URLCONF = 'app.testurls'

    def tearDown(self):
        self.ROOT_URLCONF = self._old_root_urlconf

Remember that setUp() is run just before each test, and tearDown() is run just after. What we're doing is replacing the URLConf just for the duration of the test. We have to put it back in the tearDown(), else we'll end up with test state 'leakage' into other tests.

You then just have to add a testurls.py file to the 'app' application (the one I'm testing) which contains known URLs  for the views I want to invoke. For example, my testurls.py might look like this:

from django.conf.urls.defaults import *

urlpatterns = patterns('app.views',
    (r'^foo/',  'say_hi'),
)

I can then write standard test client code to check that my views are working as expected:

class RealTest(TestUrlsTestCase):
                
    def testGet(self):
        response = self.client.get('/foo/')
        self.assertEqual(200, response.status_code)

It now doesn't matter how the project integrator has configured their urls.py. When the tests are run, the URLConf that I have specified will always be used.

Unit tests are supposed to test your code, and just your code. However, once you're in a framework environment (be that Zope and Plone, Django, or anything else) then testing how your code integrates with that framework is vital. Zope and Plone provide unittest.TestCase subclasses (ZopeTestCase and PloneTestCase respectively) which provide a lot of scaffolding for you to be able to run integration tests. Part of that scaffolding is automatic transaction management. This hooks into Zope's transaction API to roll back the transaction after each test runs.

I wanted to do something similar for my Django test cases; I was finding 'state pollution' between my unit test runs, since data created by one test method isn't automatically cleaned out.

Django's transaction handling is much simpler than Zope's: it cares only about the one database transaction that the current request has, and only if the transaction support middleware is installed. This means that we can pretty easily crib the code from that middleware and use it in a test case base class:

from django.db import transaction

class TransactionalTestCase(unittest.TestCase):
    
    def setUp(self):
        super(TransactionalTestCase, self).setUp()
        
        transaction.enter_transaction_management()
        transaction.managed(True)
        
    def tearDown(self):
        super(TransactionalTestCase, self).tearDown()

        if transaction.is_dirty():
            transaction.rollback()
        transaction.leave_transaction_management()

UPDATE: Fixed an error in the call to the base class' tearDown() method, which caused open transactions to hang around and (among other things) prevented the test database being cleanly dropped at the end of the test run.

After this, you can simply derive your test fixture classes from TransactionalTestCase, and make sure that you call the base setUp() and tearDown() methods if you do need to override them to perform your own setup and teardown.

My next spare time (hah!) project will be to integrate Django's transaction management into repoze.tm (which is Zope's transaction management suitably WSGI-fied). This would let a Django application participate in transactions with other transaction-aware components, making integration at the WSGI layer much more straightforward.