mFabrik - mobile sites, apps, HTML5 and CMS software development » setuptools

Run your Python application using PyPy – it’s fast!

Mikko Ohtamaa — Sat, 26 Mar 2011 18:14:18 +0000

Try your Python application with PyPy, the new hot Python JIT compiler

Your program will run faster…
… faster…
and ..faster!

Here are quick instructions for using PyPy on OSX. You should be able to apply these instructions to other UNIX systems as well.

Install PyPy by getting binary tarball.

wget http://pypy.org/download/pypy-1.4.1-osx64.tar.bz2
tar -xjf pypy-1.4.1-osx64.tar.bz

Install Distribute / easy_install command (setuptools fork) in order to install third party libraries.

cd pypy-1.4.1-osx64
wget http://python-distribute.org/distribute_setup.py
bin/pypy distribute_setup.py

Install libraries – easy_install seems to run just fine on PyPy as long as you don’t try to install Python native extensions

bin/easy_install plac # command line parser
bin/easy_install iso8601 # date format parser

Then run your application

~/code/pypy-1.4.1-osx64/bin/pypy exceptionanalysis.py     -s=2010-01-01T00:00:00 /Users/moo/xxx/log/client1.log

In my case the application was a simple log analyzer based on Python generators (code will be posted later). It reads a text file and tries to interpret it. The script utilizes only one CPU core and thus is very CPU bound. The speed comparison results are

With Python 2.4: Completed in 111.08 seconds
With PyPy: Completed in 30.39 seconds

That’s almost 4x speed-up!

Things did not work with PyPy

import multiprocessing – was not hard dependency, needed to comment out in plac_ext

Here is the traceback if anyone can help:

Traceback (most recent call last):
 File "app_main.py", line 53, in run_toplevel
 File "exceptionanalysis.py", line 20, in 
 import plac
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac.py", line 35, in 
 from plac_ext import Interpreter, import_main, ReadlineInput, stdout, runp, Monitor
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/plac-0.8.0-py2.5.egg/plac_ext.py", line 7, in 
 import itertools, traceback, multiprocessing, signal, threading
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/__init__.py", line 87, in 
 import _multiprocessing
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 7, in 
 __bootstrap__()
 File "/Users/moo/code/pypy-1.4.1-osx64/site-packages/multiprocessing-2.6.2.1-py2.5-macosx-10.6-i386.egg/multiprocessing/_multiprocessing.py", line 6, in __bootstrap__
 imp.load_dynamic(__name__,__file__)
AttributeError: 'module' object has no attribute 'load_dynamic'

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Installing MySQL-python connector on OSX

Mikko Ohtamaa — Wed, 02 Mar 2011 19:22:29 +0000

Preconditions

You are using MySQL installation from mysql.com (and not e.g. from Macports)
You want to install MySQL-python egg using easy_install or buildout

You’ll get an error

Getting distribution for 'MySQL-python'.
install_dir /Users/moo/project/eggs/tmp8-j13e
sh: mysql_config: command not found
Traceback (most recent call last):
 File "", line 1, in ?
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1855, in main
 with_ei_usage(lambda:
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage
 return f()  
...

 31, in 
 {'__file__':setup_script, '__name__':'__main__'}
 File "setup.py", line 15, in ?
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 43, in get_config
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 24, in mysql_config
EnvironmentError: mysql_config not found
An error occured when trying to install MySQL-python 1.2.3.Look above this message for any errors thatwere output by easy_install.
While:
 Installing instance.
 Getting distribution for 'MySQL-python'.
Error: Couldn't install: MySQL-python 1.2.3

The reason is that MySQL installer puts MySQL under /usr/local and does not expose MySQL utilites directly to path. MySQL-python expects to find utility mysql_config to configure compile time paths when building native bindings using GCC.

Workaround:

sudo -i
cd /usr/bin
ln -s /usr/local/mysql-5.1.46-osx10.5-x86_64/bin/mysql_config .

Then rerun installer.

Though it is a bit harsh solution to modify system folder, it gets the thing done in simple manner.

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Subversion global-ignores and .egg-info in Python/Plone development

Mikko Ohtamaa — Sat, 03 Oct 2009 12:47:04 +0000

Subversion does a good job by ignoring most of build/temporary/unwanted files by default.

However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools.

If you are working with Python source code eggs, add the following line to your ~/.subversion/config

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject

Otherwise development tools like Mr. Developer might get confused.

Setup.py sdist not including all files

Mikko Ohtamaa — Fri, 24 Jul 2009 09:56:44 +0000

Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.

I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it.

By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file.

Example MANIFEST.in (plonetheme, built in PyDev):

recursive-include plonetheme *
recursive-include docs *
global-exclude *pyc
global-exclude .project
global-exclude .pydevproject

About the author Mikko Ohtamaa

Mysterious buildout error – missing docs/HISTORY.txt file

Mikko Ohtamaa — Wed, 01 Oct 2008 15:55:03 +0000

I was getting the following error with Plone buildout

…

Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
  File "/tmp/tmp_G8621", line 11, in ?
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in 
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 9, in ?
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
    return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
    getattr(buildout, command)(args)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
    installed_develop_eggs = self._develop()
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
    zc.buildout.easy_install.develop(setup, dest)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
    assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError

My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while.

Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs.

There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install.

So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency.

It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t.

The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder.