Setup.py sdist not including all files
Posted on July 24, 2009 by Mikko Ohtamaa
Filed Under plone, python, technology
Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.
I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it.
By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file.
Example MANIFEST.in (plonetheme, built in PyDev):
recursive-include plonetheme * recursive-include docs * global-exclude *pyc global-exclude .project global-exclude .pydevproject
About the author Mikko Ohtamaa
Other posts by Mikko Ohtamaa
Hi Mikko,
An alternative to building a manifest is to build the sdist from within an svn checkout. Everything under version control will be included automatically.
Of course, then there are all the problems with setup.py and newer versions of svn….
Steve
You don’t need to declare a manifest file if you use a subversion working copy (<= svn 1.5 at the moment, another thing Distribute will help with). You could do the following:
$ svn co http://svn.plone.org/svn/collective/plonetheme.twinapex/tags/1.0 plonetheme.twinapex
$ cd plonetheme.twinapex
$ python setup.py sdist upload
This method uses the .svn/entries files to know what should be a part of the distribution.
That silent failure mode is actually inherited from the distutils, btw. Setuptools simply doesn’t fix it.
If you don’t use SVN, but something else (anything, actually), then you can maybe coerce setuptools to respect that other SCM, or be screwed. Found it out the hard way, too.
Thank you for everyone for very constructive comments.
I was indeed using SVN so I was bitten by the version bug.
For ordinal users, it is very hard to distinguish between setuptools and distutils. setup.py is the only interface they see and the only interface they should interact with (…because it’s the only one they see)
In the future versions I hope setuptools/distutils/distribute/whatever will simply have a setup.py statement
included_files=”x”
where x could be
1) version control system name which is used to get the included file list. Possible options svn1.4, svn1.5, hg, git, so on
2) regexp patterns which are used to include files (current MANIFEST.in content)
Because the option is in setup.py it would be clearly visible and self-explaining for everyone.