September 17, 2006

Mixing setuptools and distutils

I’m making a Python package I’m currently calling ApyConfig, and I’m using setuptools to distribute it. My package depends on pyparsing, which is distributed using distutils. This has caused a minor problem with dependencies.

My setup.py:

import ez_setup
ez_setup.use_setuptools()

from setuptools import setup, find_packages

setup(name = "ApyConfig",
      version = "0.1",
      packages = find_packages("src"),
      package_dir = {'': 'src'},
      install_requires = ["pyparsing"],
      test_suite = "tests",
      dependency_links = ["http://sourceforge.net/project/showfiles.php"
                          "?group_id=97203",  # pyparsing
                          ]
      )

(That URL was the only one I could use to get easy_install to download pyparsing for me, right now at least. My theory is that setuptools has some built-in intelligence about SourceForge, but some changes SourceForge made broke setuptools.)

If you know me, you know I’m the kind of guy that puts everything outside of my home directory (and maybe /srv and /opt) under control of the system package manager, most commonly RPM in my case. distutils has long been able to build an RPM from a Python package. So I extract a pyparsing distribution, run python setup.py bdist_rpm, and install the resulting RPM. Then I go to run ApyConfig’s tests with python setup.py test:

[darkness@morgase ApyConfig]$ python setup.py test
running test
running egg_info
writing requirements to src/ApyConfig.egg-info/requires.txt
writing src/ApyConfig.egg-info/PKG-INFO
writing top-level names to src/ApyConfig.egg-info/top_level.txt
writing dependency_links to src/ApyConfig.egg-info/dependency_links.txt
reading manifest file 'src/ApyConfig.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/ApyConfig.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
...
  File "/usr/lib/python2.4/site-packages/pkg_resources.py", line 483, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
pkg_resources.DistributionNotFound: pyparsing

“Oops.” The problem here seems to be that pyparsing is missing an .egg-info directory. A little searching around, some experimentation, and I thought the right thing to do when installing pyparsing would be more like:

python -c 'import sys; sys.argv[0] = "setup.py";
           import setuptools; exec open("setup.py")' bdist_rpm

Except, of course, this doesn’t work with bdist_rpm. It would if I were just doing an install, I think; but bdist_rpm packs the source, builds an RPM spec file, and that spec file then runs setup.py without my little setuptools patch above.

In short, the easiest thing to do in a situation like this seems to be something like:

perl -pi -e 's/distutils.core/setuptools/g' setup.py
python setup.py bdist_rpm

This works like a charm, at least for this package. The moral of the story: if you want to use the various bdist_* targets, it seems like you need to patch setup.py. If someone knows a better/proper way to get around this, please let me know.

Comments (2)

  1. September 17, 2006
    Paul McGuire said...

    Sorry to hear that pyparsing is giving you some difficulty. I’m about to do a new release in the next few weeks, and if there is anything that will reduce your problems, let me know. (I’m afraid you’ll need to fairly lead me by the hand, though. distutils is enough of a toolkit for me to deal with, eggs are completely confusing to me.)

    – Paul

  2. September 18, 2006
    darkness said...

    For the record, pyparsing has been a big help to me in easily making a parser. I look forward to using it in the future and would recommend it to others. I have responded to Paul McGuire by e-mail regarding setuptools. (After all, my real problem here was basically with setuptools, not pyparsing.)

    – darkness