http://mathematism.com/2009/07/30/presentation-pip-and-virtualenv/
Screencasts, diagrams, and links from my presentation for July’s django-district.
I put together a walkthrough of pip and virtualenv for the django-district July meeting. It was a slide-less presentation, focusing mostly on talking points and command-line walkthroughs. This post provides a recap of what I covered.
virtualenv
virtualenv is a tool to create isolated Python environments. It was created by Ian Bicking to provide a solution to a consistent problem.
The Problem
To illustrate the problem let’s pretend you have a Web server with two Web sites on it: mysite.com and anothersite.com. Each of your Web sites are sharing the global site-packages directory in /usr/lib/python2.x/site-packages.
The global site-packages directory has Django 1.0.2, Pinax 0.5 and PIL 1.1.6 among other things. The sites are purring along, living in complete harmony.
One day, the client for anothersite.com decides they’d like to take advantage of some of the new features in Django 1.1. Now we have a problem. How do we upgrade anothersite.com’s dependencies without affecting mysite.com?
The Solution
anothersite.com and mysite.com need the ability to have their own set of dependencies. The way to achieve this is to create isolated Python environments using virtualenv.
virtualenv creates an environment that has its own installation directories and isolates itself from other virtual environments. The environment contains a site-packages directory, installs setuptools and a Python interpreter that is aware of its environment.
This allows us to install a different set of dependencies for each of our sites. Now anothersite.com can upgrade to Django 1.1 and Pinax trunk without affecting mysite.com at all.
Installation
virtualenv is available on PyPI. Run the following command to install:
easy_install virtualenv
Working with virtualenv
While I could go into detail about the commands available with virtualenv, I honestly never use the virtualenv commands directly. I use suite of conveniences for virtualenv called virtualenvwrapper.
virtualenvwrapper
Doug Hellmann was kind enough to create virtualenvwrapper, which is best described on PyPI:
virtualenvwrapper is a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.
Screencast
I put together a quick screencast to go over the wrappers and hooks that virtualenvwrapper provides as stated below. You can also view the source codeon Bitbucket. Doug’s code is well commented and I found out about several of these wrappers only after taking a look at the source.
Wrappers
The wrappers provided by virtualenvwrapper (that I know of) are:
- mkvirtualenv (create a new virtualenv)
- rmvirtualenv (remove an existing virtualenv)
- workon (change the current virtualenv)
- add2virtualenv (add external packages in a .pth file to current virtualenv)
- cdsitepackages (cd into the site-packages directory of current virtualenv)
- cdvirtualenv (cd into the root of the current virtualenv)
- deactivate (deactivate virtualenv, which calls several hooks)
Hooks
One of the coolest things about virtualenvwrapper is the ability to provide hooks when an event occurs. Hook files can be placed in ENV/bin/ and are simply plain-text files with shell commands. virtualenvwrapper provides the following hooks:
- postmkvirtualenv
- prermvirtualenv
- postrmvirtualenv
- postactivate
- predeactivate
- postdeactivate
Installation
virtualenvwrapper is available on PyPI. Run the following command to install:
easy_install virtualenvwrapper
Add two lines to your .bashrc to set the location where the virtual environments should live and the location of the script installed with this package:
export WORKON_HOME=$HOME/.virtualenvs source /usr/local/bin/virtualenvwrapper_bashrc
You should also take a look at Doug’s blog post on virtualenvwrapper, which provides a great overview of why he created virtualenvwrapper and provides detailed setup instructions.
pip
Another tool created by Ian Bicking, pip stands for pip installs Python packages. It is a replacement for easy_install that provides some great improvements including requirements files and support for version control systems.
Requirements files
Requirements files are plain text files that contain a list of packages to be installed. These text files allow you to create repeatable installations. As illustrated in this example file, there are several ways to specify a required package.
Package name with version requirements:
Django>=1.1 Pinax==0.5
Direct URL to a tarball containing a setup.py script:
http://effbot.org/downloads/Imaging-1.1.6.tar.gz
Editable checkouts from VCS repositories:
-e svn+http://svn.myproject.org/svn/MyProject/trunk#egg=MyProject -e git+http://git.myproject.org/MyProject/#egg=MyProject -e hg+ssh://[email protected]/MyProject/#egg=MyProject -e bzr+https://bzr.myproject.org/MyProject/trunk#egg=MyProject
Editable checkouts from VCS repositories with revision information:
-e svn+http://svn.myproject.org/svn/MyProject/trunk@2019#egg=MyProject -e git://git.myproject.org/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709#egg=MyProject -e hg+http://hg.myproject.org/MyProject/@2019#egg=MyProject -e bzr+https://bzr.myproject.org/MyProject/trunk/@2019#egg=MyProject
View my sample requirements file for more examples of checking out with revision information including branches, tags and dates.
Usage
Once you have a requirements file, Installing a package using pip is as simple as the following command:
pip install -r /path/to/requirements.txt
I should note that you don’t have to create a requirements file. You can simply issue any of the commands listed in the requirements file directly from the command line like so:
pip install Django>=1.1 pip install http://effbot.org/downloads/Imaging-1.1.6.tar.gz pip install -e svn+http://myrepo/svn/MyApp#egg=MyApp
pip freeze
pip doesn’t just install packages. It also provides a few other commands, including freeze:
pip freeze > /path/to/requirements.txt
This will inspect the current environment and generate a requirements files that contains explicit version number for each of the installed packages. I generally use this as a starting point for creating a stable requirements file.
There are other commands as well. I’d encourage you to take a look at thedocumentation for pip to learn more about the various options.
Installation
pip is available on PyPI. Run the following command to install:
easy_install pip
You can also check out a copy from bitbucket. Run one of the following commands:
wget http://bitbucket.org/ianb/pip/raw/0.4/pip.py
OR
hg clone http://bitbucket.org/ianb/pip/
Conclusion
Using virtualenv, virtualenvwrapper and pip together can help you create isolated, repeatable environments with ease. We use these tools on a daily basis at Discovery Creative and it has saved us countless hours in addition to providing documentation for requirements for our applications and sites.
Showing 26 comments
yes, keep spreading the virtualenv love, nice article.
great stuff! thanks for putting the work in and sharing this informative piece.
Absolutely! I love putting these together – let me know if you have any other walkthroughs/screencasts you’d like to see.
Question: What happens to my $PYTHONPATH within virtualenv?
echo $PYTHONPATH
returns nothing. Is it now assumed that all directories within the current virtualenv are automatically on the path? Looks like it – I had to comment out the $PYTHONPATH lines in my .bash_profile to get things basically working.
But that doesn’t seem to be the whole enchilada. I was able to pip install 3rd party apps with pip or through the requirements file, but I also have a lot of custom reusable apps with no setup.py. I can’t install them with pip, and if I put them manually in /src, /lib, or /include, my Django project fails to see that they’re present. They worked previously, when I was basing everything off the $PYTHONPATH, but now there is no pythonpath.
In general, what directories within the virtualenv are recommended for what?
Side question: How would you switch to using a different version of an installed package (such as Django?)
Thanks.
Hi shacker – I’m not sure what’s happening to your PYTHONPATH. The author’s code preserves the existing PYTHONPATH (see line 306 athttp://bitbucket.org/ianb/virt…. I personally only use virtual python environments and virtualenv’s –no-site-packages option so I don’t explicity set PYTHONPATH in my profile and haven’t run into this.
To answer your second question, if you need a package that doesn’t have a setup.py, you can take two approaches. Put the package directly in lib/python2.x/site-packages/ or use the add2virtualenv command, which creates a .pth file in lib/python2.x/site-packages/.
A virtual environment creates an isolated folder with bin, lib, include, etc. It’s similar to your system folders – executable scripts are in bin, lib contains python2.x, etc.
If you wanted to change the version of an installed package, you can run pip install -U Django==X, where X is the version of Django (or any other package) you want to modify.
Hope that helps!
Thanks for the response Rich. I think I confused things a bit. Yes, if I keep a $PYTHONPATH set in bash, it’s preserved in the virtualenv. I have commented out the path in .bash_profile. What I *thought* would happen was that when I entered the virtualenv I’d get a new, automatically created $PYTHONPATH that reflected the new environment. Apparently not.
Interestingly, even though echo pythonpath shows nothing, Django debug pages still show a full pythonpath reflecting the virtualenv. So it’s there, but the shell doesn’t know about it.
The import errors turned out to be my own mistake (tangled package paths only revealed when I started isolating things with virtualenv).
Thanks again.
Good intro, Rich. Btw, is there a way to change the virtualenv prompt?
Just in case anyone else ran into the same problem, I couldn’t find virtualenvwrapper_bashrc in /usr/local/bin/ on my OS X 10.5 machine, and turns out it was here, so I symlinked it from here and it worked:
ln -s /Library/Frameworks/Python.framework/Versions/2.6/bin/virtualenvwrapper_bashrc /usr/local/bin
If yours isn’t there, (from bash shell) just do a:
find / -name virtualenvwrapper_bashrc
(for the record I installed virtualenv and virtualenvwrapper with pip)
You can change the virtual environment’s prompt using the –prompt option to virtualenv:
–prompt==PROMPT Provides an alternative prompt prefix for this environment
So, where do you keep your real apps?
Can you put a bit of code on how you develop your app in this framework?
Let’s say you are making a site called mySite that has app1 app2 & app3.
I primarily use Django and all of my projects live outside the virtualenv completely. It’s the packages that my projects use that live in the virtualenv.
For example, let’s say I have a site called mycoolsite.com. I’ll create a virtualenv called mycoolsite. The actual project files and web server configurations are in a repo and are checked out somewhere (e.g. /var/www/mysite/code/. The web server configs pull everything together. In the instance of a Django project, I’d have a .wsgi file that sets up my paths and points to my project files.
A lot of updates have happened to virtualenvwrapper since I wrote this post. It’s definitely time for me to update and show off the new functionality.
Hi Rich,
So, if I used the add2virtualenv, then I decide that I want to remove it, how do I go about it?
I am using buildout right now, it is powerful, but it has its own problems. I am trying to move over to virenv. Any suggestions for the new converts?
Hi there – honestly I don’t use add2virtualenv at all. I just highlighted it because it’s available. But what it does is just put a .pth file inside site-packages. So you could just remove that file.
My preferred way of working is to use the “–no-site-packages” flag to mkvirtualenv. This creates a clean site-packages folder and doesn’t pull in system-wide site-packages. Then you have a nice, clean env to start with, installing all the pieces you need.
Also – it’s interesting to me that lots of people are coming from buildout to ve. I’m doing the exact opposite at my new job, although I could see that changing at some point.
I still like buildout for a “finished” product. I wish it had the freeze option so I didn’t have to use the version tag and manually lock up all working versions. But lets say that I want to quickly test a new django CMS that someone wrote. With buidout, I have to go about and creating a buildout env to get it tested. With virtualenv, in few short minutes, I have the new CMS/example app up and running. I’d think that if buildout and virtualevn marry someday, they would produce a beautiful offspring.
I’m confused. If virtualenvwrapper maintains your virtual environments, then it can’t reside inside any of those virtual environments where *does* it live?
Hi Ryan – you run “sudo easy_install virtualenv virtualenvwrapper”. It lives in your system Python’s site-packages.
You can also add a hook to your workon home directory, just like you do with each individual virtualenv in the bin directory.
Just create the hook file in your workon home (such as “postactivate”) and it will be applied when ANY virtualenv is called.
For example, you could create a hook in your workon home called “postactivate” and then add in:
cdvirtualenv
…to the file and any time a virtualenv is activated, it will automatically cd to the directory of your virtualenv. This makes it so you only need to declare your hooks once, not one time for each virtualenv.
Good write up!
Cheers,
Dana W.
This is a great write up, thanks.
I always keep my projects outside of the virtual environments and they live in a separate directory.
To keep things simple, I create a symlink to the virtual environments python inside my project.
ln -s /srv/environments/djnonrel/bin/python /srv/projects/myproject/python
Then you can just do something like this in myproject:
./python manage.py runserver
etc
And just forget about where you put the environment while you work on your project
Thanks for writing this up and the screencast. Coming from an acolyte’s level of experience with ruby, I eagerly anticipated something like rvm for python. It looks like virtualenvwrapper will serve that role well. What’s really fantastic about this environment tools is how easy it makes setting up new apps or systems.
Thanks. BTW your link to “Doug’s blog post” is missing.
Hope you found it helpful! Fixed that link – thanks for pointing it out.
I use closed environments for each projects with freeze versions, and I wonder how you supervise your packages deployed, to check outdated packages for example.
Is there any supervision tool, app to do this?
Hi Oliver – a colleague of mine, Corey Oordt, wrote some Fabric tools to check for outdated/out of sync dependencies: https://gist.github.com/106622…. That may be what you’re looking for.
Thanks, I have still have a look on, It’s pretty my needs, but I am not responsible of deployment, so I don’t have any ssh access… anyway, should be easy to port for webby approach!
Nice writeup, I’ve been linking to this in all my github project installation instructions.
The filename virtualenvwrapper_bashrc has been updated since prior posts, for version 3.0 of virtualenvwrapper the filename is virtualenvwrapper.sh