Document Actions
Two Voices
Microsoft and Open Source - there and back again, a software engineer's tale.
- Creating a Python 2.4, Plone and Zope Development Environment on Mac OS X Leopard by Dan Fairs — posted on 2008-05-07 16:49 — 0 comment(s) —
-
Compiling Python, Zope and Plone on Leopard isn't as easy as it is on Linux. Here's a walkthrough of the process, from a bare Leopard install right through to having a working Plone 3 development environment, using paster and buildout.
If you're new to Zope and Plone on the Mac, and just want to get up and running, stop reading right now! Instead, go and download the unified Plone installer for Mac OS X, Linux and Solaris on Plone.org. This article is for people who need more control over the installation, and need to build the core pieces from source.
Still reading? Good.
Note that the information here has been drawn together from a number of places on the web. However, it's not been presented as an end-to-end process. That's what I aim to do here.
Install development tools
Development tools (compilers and headers, basically) aren't installed by default on Leopard. Dig out your Mac OS X DVD, and install Xcode. This will give you everything you need. You should be able to do something like this:
$ gcc
i686-apple-darwin9-gcc-4.0.1: no input filesYou can see that the gcc compiler is installed. If instead you see 'command not found', then you don't have gcc installed.
Compiling Python
Python is the first hurdle. Leopard ships with Python 2.5, but Zope 2 still requires 2.4. Unfortunately, Python 2.4 doesn't compile out of the box with Leopard. You'll see something like this:
hornet:Python-2.4.5 dan$ ./configure --prefix=/Users/dan/tmp/opt
[snip lots of output]
hornet:Python-2.4.5 dan$ make
[compile, compile, explode]
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3
-Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c
-o Modules/signalmodule.o
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3
-Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/posixmodule.c
-o Modules/posixmodule.o
./Modules/posixmodule.c: In function 'posix_setpgrp':
./Modules/posixmodule.c:3145: error: too few arguments to function 'setpgrp'
make: *** [Modules/posixmodule.o] Error 1Ouch.
Fortunately the solution for this is actually on the Python.org site. (If you scroll down a bit, which I didn't initially). Simply modify your configure line to include an extra define:
./configure MACOSX_DEPLOYMENT_TARGET=10.5
You should fine Python now compiles. You can then make install.
Oh - but the fun isn't over yet! Have a play with your new Python. At best, you'll find your arrow keys don't work. At worst, you'll experience odd crashes with the helpful generic Bus Error message.
Fortunately, Andreas Jung has the answer here: a broken or outdated version of the readline library. You can compile your own, or do what Andreas recommends and install one from DarwinPorts. Once that's installed, you simply need to do:
$ sudo port -d selfupdate
$ sudo port install readlineYou should now have a working readline. Next up, you need to configure your Python build environment to pick this up (otherwise it'll just pick up the broken OS X readline again.) I actually did this the way Andreas describes, a manual hack in the Makefile:
CC = gcc -I/opt/local/include -L/opt/local/lib
I suspect that you'd be able to pass --includedir=/opt/local/include and --libdir=/opt/local/lib to the configure script, and that would be a cleaner way of doing it. Once you've down that, run make and make install, and you should have a working Python, with working readline support, that doesn't crash.
Remember to set your PATH up to use your new Python binary in preference to the system one - something like:
export PATH=/Users/dan/opt/bin:$PATH
What's next?
setuptools
A lot of stuff these days is installed using setuptools. Download ez_setup.py to somewhere convenient, and run it using your new python:
python ez_setup.py
This will download and install setuptools into your Python 2.4's site-packages directory, and configure it so that it's usable.
To check it's installed correctly, type:
which easy_install
The path should be within your Python 2.4 install. If it's not then you've managed to install it into your system Python. First - why are you running as root; and second, check your PATH.
Note that from now on we're going to be using a package called virtualenv to isolate any further package installs. You've gone to a lot of effort to get a working 2.4 install - let's keep it tidy!
virtualenv
virtualenv is a fantastic tool which creates a Python 'sandbox' to work in. This means that you can install packages in a virtualenv to your heart's content, and it won't pollute the main Python installation. It creates a new python binary, and lib/python24/site-packages structure to facilitate this.
Installing it is dead easy:
easy_install virtualenv
virtualenv will be downloaded and installed. Let's immediately create a virtualenv to work in:
$ mkdir -p ~/virtual/2.4
$ cd ~/virtual/2.4
$ virtualenv plone3
New python executable in plone3/bin/python
Installing setuptools............done.
$ cd plone3
$ source bin/activate
(plone3)$Note how your prompt has changed - this is a visual clue that you're now working in a virtual environment. Look at the output for:
which python
You should see that the Python from the virtualenv is used. This of course means that any further easy_installs you do will be into thsis virtualenv.
As above, I find it useful to put my virtualenvs for Python 2.4 and Python 2.5 in separate directories. But that's just me - running source bin/activate in each virtualenv will do the right thing.
PIL
For a while, Plone has depended on the Python Imaging Library. It's not quite as simple as just doing easy_install PIL, unfortunately. You've got a couple of options:
(plone3)$ easy_install --find-links http://www.pythonware.com/products/pil/ Imaging
... may work for you. Otherwise, just download the PIL source kit manually from the distribution site, and install it using the standard distutils technique:
(plone3)$ tar -xzf Imaging-1.1.6.tar.gz
(plone3)$ cd Imaging-1.1.6
(plone3)$ python setup.py build_ext -i
(plone3)$ python setup.py install(You wouldn't believe how many times I've missed the last step, and wondered why my PIL wasn't working.)
Almost there! But... I feel some more build tools coming along. Since this is a Plone 3 install, we're going to be using buildout and paster. Fortunately there's a package that makes the setup of this easy.
ZopeSkel
ZopeSkel is a package which will install paster, buildout, and some commands to help create Plone development instances (amongst other things, including Silva instances, Plone themes, etc.) Installing it is easy:
(plone3)$ easy_install ZopeSkel
This will whir for a while. Once it's installed, you should find that you have a paster installed:
which paster
... will show you that it has, as with everything else, been installed into your virtualenv.
Creating a Plone 3 development environment
We're on the final leg of the journey now!
paster and buildout do most of the heavy lifting for us here. First of all, we use paster to create a buildout for us:
paster create -t plone3_buildout p3
This, again, will whir for a while. plone3_buildout refers to the template to be used to create p3. If you're interested, you can see what other templates were installed:
paster create --list-templates
Of course, paster create --help will give you more information on available options.
Once we have our buildout, we're ready to go. Let's finally build Zope out, configured for Plone. During this process, you'll be asked for an initial administrative username and password for Zope:
(plone3)$ cd p3
(plone3)$ python bootstrap.py
(plone3)$ bin/buildoutNow go and make a cup of tea - that last step can take a while.
What's happening is that buildout is using the buildout.cfg to locate and download all the eggs and parts which make up a Zope 2 and Plone 3 installation. It will then compile and install those for you. If you run buildout with multiple -v options (eg. -vvvv) then it'll give you a lot more information about what it's doing.
When this finishes, you should be able to run up your Zope instance:
(plone3)$ bin/instance fg
Browse to http://localhost:8080 (assuming you kept the port the same) and you should see the familiar Zope Quick Start page.
Creating a development egg
It's likely that all new development that you'll be doing in Plone 3 will be egg-based. However, it's not immediately obvious how you get started.
Development eggs live in the src/ directory of your buildout. As with the main Plone 3 buildout, we use paster to create the egg for us:
(plone3)$ cd src/
(plone3)$ paster create -t plone egg.name
You'll get asked some questions. The namespace and package you should set appropriate for your project (I'll assume you answered 'egg' and 'name' respectively); and say 'no' to the Zope 2 product and zip-safe questions. Paster will then go ahead and create the file structure for your egg in src.
Finally (and I mean it this time) we have to tell buildout about the development egg. Go back up one level of directory, above src, and you should see a file called buildout.cfg. Edit this file, and add the following:
[buildout]
eggs =
egg.name
...
develop =
src/egg.name
...
[instance]
zcml =
egg.nameOnce this is saved, then run buildout again in offline mode:
bin/buildout -o
This will look at your buildout.cfg and modify the buildout to include your development egg.
And after that?
If you've got to here, well done! You've now got a development environment. The final stage, once you've developed your egg, is to package it. This is pure setuptools. From within your egg:
python setup.py bdist_egg
This will create a finished egg, placed in the dist/ directory, ready to use.
There's a lot more to learn around this process - but I hope that's clarified the basic procedure, and workarounds for some of the issues specific to Mac OS X Leopard.
- import this.NET by Dan Fairs — posted on 2008-04-15 16:28 — 0 comment(s) —
-
The Zen of Python doesn't just apply to Python.
PEP 20 is one of the most important documents that you read as a Python programmer. For the lazy, here's the meat:
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way
to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!The interesting thing about this PEP, of course, is that it doesn't really apply to Python. It applies to the programmer. Therefore, it applies equally well to most software development; be it in Python, C#, SQL, or whatever. Whenever I find myself faced by a design decision where I'm unsure of the best path, applying the guidelines above usually helps me decide.
Try it yourself. You'll be surprised how the above cuts through a lot of design cruft.
(Having said that, I'm not sure the one about being Dutch really applies to .NET.)
- Shared Development Databases by Dan Fairs — posted on 2008-04-14 12:11 — 0 comment(s) —
-
One of the big culture shocks moving to a .NET development team was the amount of shared development infrastructure - particularly on the database side. In this post, I share some of my experiences and suggest ways that you can work around some of the impositions of shared infrastructure.
First of all - long time no postee! Sorry about that. As well as starting to arrange my own wedding, I've been helping a couple of friends out with some interview questions and another ASP.NET application. All this adds up to very little time left over for blogging.
Back to your scheduled transmission.
I've been wearing the Enterprise .NET/SQL Server/Oracle developer hat for a little over a year now, and it still isn't very comfortable. One of the particularly uncomfortable aspects is the shared development database that we have to use.
Yes, that's right - everyone develops on a single instance of Oracle or SQL Server. People are making structural and procedural changes all the time as we develop the software. Chaos ensues.
Well - actually, less chaos than you would expect. We have some carefully orchestrated dances which minimise the impact of this restriction. The given reasons for this restriction are of varying levels of convincingness (new word made up today - check):
- Centralised access control
- Centralised maintenance and patching
I can understand those arguments. However, for me, the massive inconvenience and danger of losing work significantly outweighs the benefits (especially considering the access control procedures are laughable - but that's a story for another day).
The rhyme and reason aside, what do you do?
Make sure you have scripts for everything
If you're not physically sharing a database, then you need to be able to share changes. If you don't have this, take a snapshot of your current schema now and check it into source control.
You should then add a table to your database to track schema version (optionally add fields to track who made changes and when, if you like).
Finally, only make changes to the database schema via scripts. These scripts should update the database version table once they've run; they should probably sanity check that the database is at the expected version before running. If you use an appropriate naming convention (perhaps <version>.sql) then it becomes pretty easy to automate running these.
Get Your Own Database Schema
Even if you're stuck on shared kit and database instance, getting your own database schema (or even better, permission to create your own schemas) is the next best thing. You should be using the schema scripts to communicate change. Having an automated database build is really critical here. Wiring that into a continuous integration process (or at the very least, a nightly integration run) will also help you spot any conflicts.
This works best if you're using a branch-for-feature (or branch-for bug) strategy in source control. That way, when your piece of work is done, you drop your development database, rebuild it from trunk code, then merge your scripts into trunk and run them. This merge step gives you the opportunity to reconcile any conflicts.
Of course - it always helps to talk to your colleagues! Even with the best branching and merging strategy in the world, conflicts are going to be difficult to resolve if two of you have created a column with the same name but different types.
Suck it up
Unfortunately, the above doesn't seem to be possible at The Day Job. We have to share schemas. We can't each have our own because disk is expensive.
(Well, we all know that disk is actually cheap. But The Day Job is fully bought into the idea that you need an expensive enterprise - there's that word again - storage solution. The upshot of this is that the amount of storage that's in my Xbox would cost thousands at my place. And it certainly doesn't translate into fast or flexible provisioning, and doesn't seem to offer much more in the reliability stakes than a standard RAID configuration. But I'm just a lowly software engineer - what would I know?)
So we all share a single development schema (for each application), on a single database instance.
If you're like this, then you're in 'cooperative multitasking' territory - that is, you have to agree with the rest of the team that, at all times, you work on different parts of the schema and different stored procedures. (It's even slightly worse on Oracle - tools tend to steer you towards scripting whole packages, so the 'atomic unit of work' tends to be much larger on Oracle than SQL Server.)
You can (just about) make this work. It's painful and error-prone though - and one misstep will blow everyone's work away at once.
- Django Redirects (Updated) by Dan Fairs — posted on 2008-03-07 18:50 — 0 comment(s) —
-
This is a quick-n-dirty app to let you set up redirects on your Django site through the admin interface. (Updated to reflect my own oversight of django.contrib.redirects, which, er, does all this for you...)
Update: A reader has kindly pointed out to me that django.contrib.redirects already exists, which does what I describe below! D'oh. I'm sure that wasn't there when I last looked. Don't use my code - use the Django one, of course. Consider this... another talked-through example of middleware. And a lesson in reading before you write.
There are loads of ways to tell your audience that their content has HTTP 301: Moved Permanently. A line in your Apache/lighttpd/whatever configuration perhaps; or you could even wire in Django's redirect_to generic view in your urls.py file. These are the recommended approaches: directly configuring your web server is likely to be the most performant solution (as requests will never touch your app server cluster), and the urls.py method will even let you pick apart the incoming URL and plug arguments into the URL to be redirected to. All good stuff.
Sometimes, however, you need to get a redirect up NOW. Perhaps a high-profile site has linked to an article of yours, but they've messed up the link, and it's 2am and you're in the UK and all your techs who would normally do this for you are in bed (or the pub) and all that potential advertising revenue is bouncing off your 404 page.
Here's a tiny app (based heavily on flatpages) that lets you quickly and easily add a redirect to your site. You can download it from the Software section. Here's how you install it:
- Unpack the file and drop it somewhere in your PYTHONPATH, so Django can import it.
- Edit your settings file, and add 'stereoplex.flatredirects' to your INSTALLED_APPS
- While you're add it, add 'stereoplex.flatredirects.middleware.FlatredirectFallbackMiddleware' to your MIDDLEWARE_CLASSES
- Resync your database, and you should be good to go.
It's pretty simple to use. You should see the 'Flatredirects' application appear in your admin interface, and it'll let you add Redirect objects.
Your 'from url' should be the location which is currently 404'ing (ie. the incoming, broken link). It works in the same way as flatpages - so it's relative to the root of your site, and it should have a leading and trailing slash.
The 'to url' is sent back to the browser - it can be relative, or absolute.
Once you've saved that, try it - you should find that going to yoursite.com/redirectfrom (where redirectfrom is what you put in the 'from url' box) will now redirect you appropriately.
How does it work?
Flatredirects uses a very simple Django middleware (not to be confused with WSGI middleware) to do its work. I've talked about Django middleware in a previous post. In that example, I implemented a process_request() middleware, since I needed to augment the request before it was passed on to.
This time, we're dealing with the response, so we provide a process_response() method. Take a look at the files middleware.py and views.py in stereoplex/flatredirects.
The logic is pretty simple: if the response is going to be a 404, then hand processing off to the flatredirect view. This view looks the URL up in the database, and uses Django's own redirect_to generic view to perform the redirection if there is a match. If there isn't, 404 is raised once more back to the middleware; and in this case, the middleware returns the original 404.
Is this suitable for doing all your redirects?
This product makes adding redirects very easy. However, there are some caveats to this approach:
- In this implementation at least, there's no ability to dynamically generate the redirect URL.
- More importantly, you should consider the performance implications of all your potential 404 accesses causing a database hit.
So it's probably OK as a quick band-aid if nobody's around to fiddle with your web server configuration - but don't blame me if your database melts!
Note that this is an alpha release, but feel free to let me know of any bugs, or features that you'd like to see.
Get it at:
- Using output parameters with the Enterprise Library by Dan Fairs — posted on 2008-02-25 09:13 — 0 comment(s) —
-
Trying to get stored procedure output parameters working when you're using the Enterprise Library 3 can be frustrating. Here's a couple of tips to help you along.
It's fairly common to return a IDataReader from a stored procedure call to iterate through the results set. It gives the underlying data source the opportunity to return results lazily, and can potentially avoid your application having a whole data set in memory at once. This is clearly useful if you're dealing with very large data sets.
Typically, code to get one of these data readers looks something like this:
using (DbCommand cmd = db.GetStoredProcCommand("s_sProcName"))
{
db.AddInParameter(cmd, "@p_sFoo", DbType.String, "Bar");
}
return db.ExecuteReader(cmd);This works as you'd expect - the stored procedure s_sProcName executes, and the method returns an IDataReader from which you can fetch results.
However, adding an out parameter doesn't work as you'd expect:
using (DbCommand cmd = db.GetStoredProcCommand("s_sProcName"))
{
db.AddInParameter(cmd, "@p_sFoo", DbType.String, "Bar");
db.AddOutParameter(cmd, "@po_iOut", DbType.Int32, 4)
}
IDataReader reader = db.ExecuteReader(cmd);
int x = db.GetParameterValue("@po_iOut");
return reader;If you try this, you'll find that x is zero — and in fact, the call to GetParameterValue returned null.
It seems that all the row data has to already have been returned by the DbCommand object before out parameters from the stored procedure actually become available. To maintain your interface, you therefore have to do something like this to get at your data:
DataTable table = db.ExecuteDataSet(cmd).Tables[0];
IDataReader reader = table.CreateDataReader();
int x = db.GetParameterValue("@po_iOut");
return reader;ExecuteDataSet() fetches all available data from the DbCommand object. Once this has been done, output parameters become available for use. Of course, this approach negates one of the key advantages to using IDataReader at all - that you don't pull all the data into memory at once.
I'd love to hear from anyone with a better way of doing this. It's occurred to be that multiple result sets might be a way forward, though I don't know if multiple IDataReaders are supported in that scenario.
I guess I'll just have to try it to find out.