Speeding up Django unit test runs with MySQL
When I'm developing Django sites, my database of choice is usually PostgreSQL. However, lots of clients use MySQL. And there lies a problem: table creation on MySQL seems to be an order of magnitude slower on Mac OS X than on Linux. This makes repeated unit test runs extremely painful.
I researched this a little bit a while ago, and noticed that it had been reported as a bug in the MySQL tracker. At the time, there were no fixes or workarounds.
A recent update, however, has revealed the use of the skip-sync-frm option. Put it in your MySQL config file in the [mysqld] section for a quick speedup:
Of course, nothing in this life is free, as Daniel Fischer explains in a comment:
The reason why it's slower on Mac OS X than on Linux is that on Mac OS X, fcntl(F_FULLFSYNC) is available, and mysqld prefers this call to fsync(). The difference is that fsync() only flushes data to the disk - both on Linux and Mac OS X -, while fcntl(F_FULLFSYNC) also asks the disk to flush its own buffers and blocks until the data is physically written to the disk.
In a nutshell, it's slower because it's safer.
So, we're trading data integrity for performance - but this is a development machine, so trashing and recreating databases (or the MySQL installation for that matter) is fine, if necessary.
Et tu, Linux?
My colleague was having similar problems on the latest Ubuntu, 10.10. The tweak above helped him too, but his test runs were also painfully slow. He'd already added the 'noatime' option to fstab.
It turns out that the newest Ubuntu ships with ext4 as the default file system. By default, ext4 makes absolutely sure that all data has been written out to the filesystem journal before writing the journal commit record. This is done through the use of filesystem barriers. Again - this is done to prefer data integrity over performance. Since this is a dev machine, it's disposable, and performance is more important. So, we can turn this off in /etc/fstab:
/dev/sda3 on / type ext4 (noatime,rw,errors=remount-ro,barrier=0)
Read more about the barrier setting on Kernelnewbies.org.
Just to reiterate - it's probably best not to do this on a machine that's important without thinking about it carefully. Those settings have conservative defaults for a reason!