Subversion global-ignores and .egg-info in Python/Plone development

Subversion does a good job by ignoring most of build/temporary/unwanted files by default.

However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools.

If you are working with Python source code eggs, add the following line to your ~/.subversion/config

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject

Otherwise development tools like Mr. Developer might get confused.

Setup.py sdist not including all files

Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.

I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it.

By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file.

Example MANIFEST.in (plonetheme, built in PyDev):

recursive-include plonetheme *
recursive-include docs *
global-exclude *pyc
global-exclude .project
global-exclude .pydevproject

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

Mysterious buildout error – missing docs/HISTORY.txt file

I was getting the following error with Plone buildout

…

Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
  File "/tmp/tmp_G8621", line 11, in ?
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 9, in ?
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
    return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
    getattr(buildout, command)(args)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
    installed_develop_eggs = self._develop()
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
    zc.buildout.easy_install.develop(setup, dest)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
    assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError

My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while.

Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs.

There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install.

So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency.

It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t.

The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder.