Packaging reuseabe & testable Django apps with virtualenv, pip, and Fabric

As someone noted the other day on one of my Facebook posts, I've been doing a lot of Python development. I've moved almost entirely to Python for development, web and otherwise. Instead of PHP, I reach for Django when I need to prototype an application quickly.

One of the things I've been struggling with is how to build re-usable applications that are testable without having the entire Django stack running. Until recently, I've used buildout to handle this. There's a djangorecipe for creating a Django repository. I include that, a sample project, the necessary requirements in a buildout.cfg and away we go.

All was well, until I included two project that had a sample project/ directory. The base Django project couldn't figure out what was what, problems abound. There are other solutions. I could have set the project (doesn't this seem like the Misses Bennett in Pride and Prejudice?) variable to change it on a per-app basis, but that still left me with some problems.

Namely, I don't like the default layout of djangorecipe's Django project. I wanted to change it, but after some digging around in buildout's internals, I realized it wasn't going to be a solution I could live with long term. I'd heard a lot of people (by that, I mean James) state their preference for virtualenv and pip. The separation of concerns (one application for isolation, another for installation) instead of the all-in-one approach of buildout felt better to me, so I started exploring.

And I came up empty.

A lot of people talk about using virutalenv and pip together; pip documents how to install into a virtual environment; but no one talks about how to use everything together with Django. Specifically, no one mentions what to include in your repository. Until now. :-)

Setting up the repository

The most important part of this for me was what to store in the repository. It's simple enough, really. First, you need a requirements.txt file. For most simple Django apps, it contains one line. For example, this is what the requirements.txt file for d51.django.apps.tagging looks like:

Django

The next thing I need is a simple .gitignore file. My mantra is to not commit anything that I can generate. This means all of the files generated by virtualenv and pip need to be ignored. I also ignore the swap files created by Vim (hey, I'm not the only one at the company who uses it, so might as well ignore it) and I ignore all .pyc and .pyo files. The resulting .gitignore file looks like:

bin/*
include/*
lib/*
.*.swp
*.py[co]

Now we're ready. Of course, you need to have virtualenv and pip installed, but once you've done that, running tests are pretty simple. First, you have to initialize the environment:

prompt> git clone git://github.com/domain51/d51.django.apps.tagging.git
prompt> cd d51.django.apps.tagging
prompt> virtualenv .
prompt> pip install -E . -r requirements.txt

The observant might notice my call to virtualenv. I've left out the parameter --no-site-packages. Two reasons. First, I don't keep things like Django installed at the site-packages level. Second, the things I do install, tools like Fabric, I want access to them while in the virtual environment without having to re-install them.

Now that the virtualenv has been initialized, now you need to activate the virtual environment:

prompt> source ./bin/activate
(d51.django.apps.tagging)prompt>

Notice that the prompt changes. It's prefixed with the name of the directory you're in to signify that you're inside virtualenv. Now running the tests are dead simple:

(d51.django.apps.tagging)prompt> python ./run_tests.py

Testing Django apps inside virtualenv

I need audio that plays when you get to this line. That screeching record player coming to halt. The visual question mark. What's this ./run_tests.py file you ask? The secret sauce.

Django wants to be setup in order to run. Normally that's requires a project, settings.py, and a partridge in a pear tree. Unless you call settings.configure. You can use that to mimic the normal Django settings, tweaking the settings to match your needs for testing.

For d51.django.apps.tagging, the settings are pretty simple. I need to make sure that my app is available along with django.contrib.contenttypes since I make use of the generic relationship code. There's also some cargo culting required, as Django won't run without a DATABASE_ENGINE specified. The end result looks like this:

from django.conf import settings
from django.core.management import call_command

def main():
    # Dynamically configure the Django settings with the minimum necessary to
    # get Django running tests
    settings.configure(
        INSTALLED_APPS=(
            'django.contrib.contenttypes',
            'd51.django.apps.tagging',
        ),
        # Django replaces this, but it still wants it. *shrugs*
        DATABASE_ENGINE='sqlite3'
    )

    # Fire off the tests
    call_command('test', 'tagging')

if __name__ == '__main__':
    main()

Running without activating virtualenv

This works, but requires that you always have virtualenv activated. For example, if you deactivate virtualenv and try to run the test, you get an ImportError:

(d51.django.apps.tagging)prompt> deactivate
prompt> python run_tests.py 
Traceback (most recent call last):
  File "run_tests.py", line 1, in <module>
    from django.conf import settings
ImportError: No module named django.conf

You can programmatically activate virtualenv, however, by including this snippet of code in a .py file located in the root of your repository:

execfile('./bin/activate_this.py',
         dict(__file__='./bin/activate_this.py'))

You can add that line to the top of the file and execute run_tests.py without needing to activate the virtualenv before hand. The line needs to go before the from django.conf line to make sure that Python knows where to find Django and any other requirements of the test.

This requires that you activate the virtual environment prior to running the test, or have Django installed at the system level. This can be further simplified and remove the need to activate and deactivate the environment prior to test runs by executing the bin/activate_this.py file that virtualenv ships with.

Making this reusable

There's a lot of code in that run_tests.py file that is going to be duplicated for every project. Actually, there's only three lines, well, two lines and one variable in a line, in it that are unique: the two line of INSTALLED_APPS and the app name to test in the call_command line.

To keep from repeating that for every single project, I created a simple harness for initializing tests. Introducing d51.django.virtualenv.test_runner, a very small package for running Django tests inside virtualenv.

Using this, run_tests.py now looks like:

try:
    from d51.django.virtualenv.test_runner import run_tests
except ImportError:
    print "Please install d51.django.virtualenv.test_runner to run these tests"

def main():
    settings = {
        "INSTALLED_APPS": (
            "django.contrib.contenttypes",
            "d51.django.apps.tagging",
        ),
    }
    run_tests(settings, 'tagging')

if __name__ == '__main__':
    main()

The first four lines give the user some input when they run the tests without test harness. That's optional, depending on how user friendly you want to be. After that, all I do is call run_tests with a settings dictionary and the name of the app I want to test.

There is one downside, though. You have have it installed outside of your virtualenv in order to run your tests. Personally, I'm not to worried about it, as I have it installed, but if you're really paranoid, you could include it in your requirements.txt file, which would require that the user be inside the virtualenv to run your tests.

Wrapping it all up in a Fabric cloth

The final step is to make all of this bullet proof is creating a Fabric file that handles all of the initialization and running of the tests for me. For good measure, it should also be capable of cleaning up after itself. I don't need a bazillion copies of Django laying around, afterall.

The end result looks something like this (using Fabric 1.0a):

from fabric.api import local

def test():
    """
    Run tests for d51.django.apps.schedules
    """
    local("python ./run_tests.py")

def init():
    """
    Initialize a virtualenv in which to run tests against this
    """
    local("virtualenv .")
    local("pip install -E . -r requirements.txt")

def clean():
    """
    Remove the cruft created by virtualenv and pip
    """
    local("rm -rf bin/ include/ lib/")

Now you can run the initialize the environment, run the tests, and clean up after yourself with three commands:

prompt> fab init
prompt> fab test
prompt> fab clean

One drawback to this method, Fabric's local command swallows the output of the test. This isn't a problem until you have a failure. local does contain a capture parameter, but it doesn't display the output from an failed command. That's fixable, but for the time being, my recommendation is to use Fabric as your quick sanity check, but rely on straight python ./run_tests.py for your real testing.

Conclusion

That brings us to the end of our quick tour. Hopefully this provides you with the information you need to get started using virtualenv and pip with Django. It's not that complicated. Actually, once you have your bearings, it's downright easy. The problem is more that people who have trodden down this path haven't documented their way. Hopefully, this post helps serve as a rough map.

Thanks

I'd like to thanks James Bennett for illuminating a few pieces of this puzzle for me (in particular, pointing me toward settings.configure) and his preview a draft of the article before I posted. I'd also like to thank Jeff Triplett for his comments on a draft and pointing out an unexplained inconsistency with the rest of the world's examples of virtualenv. And, as always, my good buddy Roder for his constant encouragement.