Note: since this was written I have renamed pyinstall to pip.
Have you ever been frustrated by easy_install? Yeah, me neither. But I have heard whispers of discontent.
So I’m introducing a new tool for the installation of Python packages: pyinstall.
pyinstall is mostly easy_install compatible. That means it finds distributions in the same way as easy_install and it installs packages via setuptools. If you are familiar with easy_install you’ll know how to use pyinstall right away. This is not a repudiation of the mechanisms of easy_install, but a refinement. What does it refine?
First, it is a more user friendly easy_install. It collects everything it needs before it installs anything. The console output is intended to be concise and helpful. It says *why* it is doing things, tracking the set of dependencies that led to the installation of each package, and some of how it found the packages. It knows a little more about Subversion than easy_install, and I plan to add native support for other version control systems directly to it (this is easier now than it would have been a few years ago, since it seems there’s a clear and finite set of viable version control systems).
It also has some added features that I think are important for version management. Specifically the features of its predecessor PoachEggs, which you may not have heard of - but pyinstall at this point is at least a third-generation implementation of these ideas, if not more. You can install a set of packages from a “requirements file”, which really is little more than a list of packages to install. This is a seemingly small improvement, but the idea is to move requirements out of setup.py and into something that is managed separately from any library or package. By separating out this management you can control the application environments without having to touch the applications or libraries themselves. In addition you can generate new requirements files from a working installation: just run pyinstall --freeze=requirements.txt and it will write out a file that can be used to install the exact version of everything that is installed. One of the most common complaints about easy_install once you get second-order dependencies is that you can’t easily reproduce working environments, and this fixes that.
Another feature is a “pybundle”, which is just an archive of a set of libraries. Here’s something you can try:
$ virtualenv my-app $ cd my-app $ source bin/activate ... set up my-app environment just how you want it ... $ pyinstall.py --freeze=my-app-req.txt $ pyinstall.py --bundle=my-app.pybundle -r my-app-req.txt ... then on another machine ... $ pyinstall.py my-app.pybundle
The pybundle format is pretty simple: it’s a lot of source files in one zip file. It isn’t a binary package format at all, it’s just source, and all it saves you is the finding and downloading of packages. But I think that simple process is a big part of why no-dependency libraries and frameworks are sought after (not to say that’s the only reason).
Despite the many rewrites preceeding this, pyinstall is still very young — but for the most part if you get something installed, then it worked, and if it didn’t get something installed then you can always fall back on easy_install (and submit a bug report). If you have suggestions, also submit a ticket or ask about it on the virtualenv list.
To get started: easy_install pyinstall or just grab the single-file executable.
I jsut installed and got a traceback, (same issue as ticket #2 but I couldn’t seem to add a comment to the ticket). Here’s the traceback:
Traceback (most recent call last):
File “/usr/bin/pyinstall.py”, line 5, in
pkg_resources.run_script(’pyinstall==0.1′, ‘pyinstall.py’)
File “/usr/lib/python2.5/site-packages/pkg_resources.py”, line 448, in run_script
self.require(requires)[0].run_script(script_name, ns)
File “/usr/lib/python2.5/site-packages/pkg_resources.py”, line 1166, in run_script
execfile(script_filename, namespace, namespace)
File “/usr/lib/python2.5/site-packages/pyinstall-0.1-py2.5.egg/EGG-INFO/scripts/pyinstall.py”, line 102, in
help=’Check out –editable packages into DIR (default %s)’ % base_src_prefix)
NameError: name ‘base_src_prefix’ is not defined
Comment by Ben on September 24, 2008 at 4:03 am
Do you plan on implementing uninstallation?
Comment by Ycros on September 24, 2008 at 4:19 am
Tried installing it; had a bit of a problem: http://gist.github.com/12511
Comment by Jon on September 24, 2008 at 4:43 am
Hi Ian,
This is great news! I’m sure a lot of people will want to use this.
A useful thing, by the way, would be some way to have to opt IN or at least verify with a yes/no prompt whether you want to install things into global site-packages. People do this far too often (e.g. they’re on Windows or are too loose with sudo, or just think that’s the way things have to be done).
Also (I just can’t resist): “This is a seemingly small improvement, but the idea is to move requirements out of setup.py and into something that is managed separately from any library or package”. Sounds a bit like buildout.
Cheers,
Martin
Comment by optilude on September 24, 2008 at 5:38 am
And most importantly, it doesn’t have “easy” in the name. As often is pointed out, Software that is called Easy or Simple rarely are.
Comment by Lennart Regebro on September 24, 2008 at 5:39 am
Hallelujah! Thanx Ian, this looks like it will add a little throb of joy to every chance I get to use it
Comment by Jean Jordaan on September 24, 2008 at 7:07 am
Does it solve the old “eggs get added to my Python PATH when installing with easy_install”? I remember that was a *major* grudge people used to cite when bad-mouthing easy_install. Actually, that might have been the *only* grudge for all I know.
Anyway, I loved the bundle idea!
Comment by David on September 24, 2008 at 7:57 am
The fix for the traceback above is:
$ /Library/Python/2.5/site-packages/pyinstall-0.1-py2.5.egg/EGG-INFO/scripts$ ls -l
total 184
-rwxr-xr-x 1 feamcor admin 91001 Sep 24 16:18 pyinstall.py
Edit the file pyinstall.py and merge the fix below:
if getattr(sys, ‘real_prefix’, None):
## FIXME: is build/ a good name?
base_prefix = os.path.join(sys.prefix, ‘build’)
base_src_prefix = os.path.join(sys.prefix, ’src’)
else:
## FIXME: this isn’t a very good default
base_prefix = os.path.join(os.getcwd(), ‘build’)
# FEAMCOR - wrong variable is being used here. Script fails to init on Mac OS X
#base_prefix = os.path.join(os.getcwd(), ’src’)
base_src_prefix = os.path.join(os.getcwd(), ’src’)
Comment by Fábio Corrêa on September 24, 2008 at 10:20 am
I agree, having all of the source code already packed into an install is HUGE! This is my one complaint about dependency based installations is that it is almost impossible to guarantee what is happening upstream is what you intended. Things change, there are network problems, etc, but if the source is located in one directory with the installer, then problem is solved at least for that specific version.
Comment by Noah Gift on September 24, 2008 at 10:27 am
How about checksums?
Comment by Rob on September 24, 2008 at 11:24 am
The bug noted is fixed in trunk: https://svn.openplans.org/svn/pyinstall/trunk/pyinstall.py — I had simply never tried it from outside a virtualenv environment, so I wasn’t seeing that problem.
Re: uninstallation, sure, at some point. Each package has a record of installed files, so it’s fairly easy.
I haven’t figured out how to do interactive stuff. I guess I just need a –no-interactive option or something. People will often want to drive this from scripts, so interactive things (like warning about global installation) is likely to be problematic. (There already is at least one prompt, though.)
David: this installs packages as eggs, bug as flat (not multi-version) eggs, so sys.path doesn’t get larger. This was a feature already in setuptools, setup.py install –single-version-externally-managed.
Noah: requirement files, when written sufficiently strictly, guarantee you get what you wanted. You can stuff those files into version control too (which I wouldn’t advise for the big pybundle zip files). So there’s actually kind of two solutions to that problem in pyinstall.
Comment by ianb on September 24, 2008 at 12:40 pm
Choice of names can have a subtle yet significant and often underrated effect on tool adoption. Why not just call it egg, since that’s what it installs. It’s three letters, easy to type and remember, all lowercase, no underscores, and you can call the internal implementation package the same thing.
I’ve programmed Python for a long time, but one nice thing I noticed during a brief Ruby foray is that Ruby’s install tool is called gem, and it installs gems, which have the file extension “.gem”. I never had trouble remembering the name of the tool.
Then to install the installer, the package can be called python-egg, packaged as a regular tarball for normal setup.py install action, and provided as rpm, deb, etc. - instead of the unorthodox download of some ez_setup. Consistency. The fewer names the better!
Comment by Brian Hawthorne on September 26, 2008 at 4:54 pm
One problem is that pyinstall isn’t really about eggs. Technically it does install eggs, though not in the way people expect (it installs eggs as a pkg_resources concept, not as a file format). It installs Python things, so pyinstall seemed like a very intuitive name.
There is a name problem, though, because I’m pretty sure it has to be a two-level command, like “pyinstall install Foo”, “pyinstall bundle Bar”, etc, but “pyinstall install” reads horribly.
Comment by ianb on October 1, 2008 at 2:45 pm
pyinstall remove is worse
Comment by jared jennings on October 3, 2008 at 4:58 pm
Similarly pyinstall uninstall.
I was just thinking about this the other day. I wanted to file a bug report against easy_install for going half way through my install and then telling me that a package conflicts with another egg.
Thats something else I don’t undrestand, how can they conflict. They have already hacked it till no end with all these .pth files so there is no way I can import to different versions at the same time. Or maybe thats why they conflict.
Another thing I could not figure out, was how come the dependencies begin from the last item in the list working upwards, is it just to inconvenience us???
Neways back to PIP, thanks for the great package. I was not sure if it is 2.6 compatible, so I installed easy_install. Not to be a hypocrite, but as soon as I can see its py26 capable Im on.
–
Hatem Nassrat
Comment by Pykler on November 4, 2008 at 1:03 am
Ian
Just wanted to say, you have done it again my man. This is *exactly* what I was looking for: in combination with virtualenv, I can now bootstrap my entire build environment in one command line. AWESOME!!!! You saved me a ton of work.
Thanks again, I shall enjoy using this immensely…
Jamie
Comment by Jamie Kirkpatrick on April 23, 2009 at 5:36 pm