Installing Python Imaging Library (PIL) under virtualenv or buildout

I have greatly struggled to have PIL library support in isolated Python environments like virtualenv –no-site-packages.

For example, when installing Satchmo shop under virtualenv:

../bin/clonesatchmo.pyhe Python Imaging Library is not installed. Install from your distribution binaries.

../bin/clonesatchmo.py
The Python Imaging Library is not installed. Install from your distribution binaries.

Though it clearly is there, installed by easy_install PIL command:

ls ../lib/python2.5/site-packages/PIL-1.1.7-py2.5-linux-x86_64.egg
ArgImagePlugin.py	 ExifTags.py		  GimpGradientFile.pyc...

Does anyone know if this problem is with PIL itself, eggified PIL or something else?

In any case, there is an easy workaround: use system-wide PIL (sudo apt-get install python-imaging) and symlink PIL from your site-wide installation under the isolated Python environment:

(satchmo-py25)mulli% pwd
/srv/plone/mmaspecial/satchmo-py25/lib/python2.5/site-packages
(satchmo-py25)mulli% ln -s /usr/lib/python2.4/PIL .

Subversion global-ignores and .egg-info in Python/Plone development

Subversion does a good job by ignoring most of build/temporary/unwanted files by default.

However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools.

If you are working with Python source code eggs, add the following line to your ~/.subversion/config

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject

Otherwise development tools like Mr. Developer might get confused.

Building a mobile site and applications with Django and Python

Recently we created a mobile site for an interactive bicycle tour. oulugo.mobi (you need to use mobile browser to access the site or you’ll get a redirect) is a multimedia enriched bicycle tour through the historic parts of the city of Oulu. All content is provided by OnGo.

The route, which you can bicycle through is drawn on Google Maps. There are nine  action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of  augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters.

Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts.

The tour is bilingual in Finnish and English.

There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service.

There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository.

The service is hosted on Python specific virtual server on Twinapex services server farm.

Features

• Premium content tailored for audio listening
• Dubbed in English and Finnish by a professional voice actor
• Bilingual: English/Finnish
• Adapts for smartphones (WebKit based browsers) and low end phones (XHTML mobile profile browsers)
• Streaming video and audio (RTSP / progressive HTTP download forv iPhone). Different audio quality is provided on depending on the handset features.
• Screen resolution detection based on user agent sniffing. Three different version of images are used.
• Custom Google Maps component for mobile is used. The component adapts for different mobile phones based on sniffing. Features include zoom, show action point, show the current location, search street address name. This component can be published on a request.
• Management interface features include video upload, video transcoding different mobile versions and editing bilingual content
• Apex Vertex handset database is used to detect the user’s mobile phone capabilities
• Apex Vertex logging and traffic analytics capabilities are used for the site statistics

Software stack

• Ubuntu 8.04 Hardy Heron virtual server
• Apache 2.2 / mod_python
• Python 2.5
• Django 1.0
• Django-CMS 1.0
• mobile.sniffer Python package to provide abstraction over different handset databases
• Apex Vertex streaming solution (RTSP based on Darwin streaming server by Apple)
• TinyMCE WYSIWYG editor
• Darwin streaming server
• ExtJS is extensively used in Apex Vertext management interface

Development effort

Development time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service.

The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

Plone Developer Manual, take #0.1

The first public version of  Plone developer manual is available here.

It is still very much draft, but I assure you will find it useful. You will find it even more useful after you put in the answers for your own problems.

In my previous Plone developer documentation rant my flow of though was little abstract and I couldn’t clearly explain how I want the community to maintain this crucial piece of documentation.  This time I made a comic.

* How to get support

** How to update Plone Developer Manual

Packing and copying Data.fs from production server for local development

These instructions help you to copy and transfer production server  ZODB database (Data.fs) to your local computer for development and testing. This allows you to do the testing against the copy of real data and the production server Plone instance set up.

See the original tip by cguardia.

Data.fs is ZODB file storage for transactional database. Journal history takes quite a lot of disk space there. Packing, i.e. removing the journal history,  usually reduces the size file considerably, making the file lighter for wire transfer. Depending on the database age the packed copy is less than 10% of the original size.

These instructions apply for Ubuntu/Debian based Linux systems. Apply to your own system using the operating system best practices.

We need ZODB Python package to work with the database. To use it, we’ll create virtualenv Python installation in /tmp. In virtualenv installation, installed Python packages do not pollute or break the system wide setup. Note that you might use easy-install-2.4 depending on the OS. The latest stable ZODB can be picked from PyPi listing. Plone 3.x default is ZODB 3.7.x, which is not available as Python egg, but you can use ZODB 3.8.x.

sudo easy-install virtualenv
cd /tmp
virtualenv packer
/tmp/packer/bin/easy_install ZODB=3.8.3

Data.fs cannot be modified in-place. You must create a copy of it to work with it. Data.fs copy can be created from a running system without the fear of corrupting the database, since ZODB is append only database.

cp /yoursite/var/filestorage/Data.fs /tmp/Data.fs.copy

Then create the following script snippet /tmp/pack.py using your favorite terminal editor.

import time
import ZODB.FileStorage
import ZODB.serialize

storage=ZODB.FileStorage.FileStorage('/tmp/Data.fs.copy')
storage.pack(time.time(),ZODB.serialize.referencesf)

And run it using virtualenv’ed Python setup with ZODB installed:

/tmp/packer/bin/python /tmp/pack.py

Lots of patience here… packing may take a while, but it’s still definitely faster than your Internet connection transfer rate.

Verify that the file is succesfully packed:

ls -lh Data.fs.copy
-rw-r--r-- 1 user user 30M 2009-09-01 13:24 Data.fs.copy

Woohoo 1 GB was shrunk to 30 MB. Then copy the file to your local computer using scp and place it to your development buildout.

scp user@server:/tmp/Data.fs.copy ~/mybuildout/var/filestorage/Data.fs

You just saved about 30-90 minutes of waiting of file transfer.

Setup.py sdist not including all files

Setuptools has many silent failure modes. One of them is failure to include all files in sdist release (well not exactly a failure, you could RTFM, but the default behavior is unexpected). This post will serve as a google-yourself-answer for this problem, until we get new, shinier, Distribute solving all of our problems.

I b0rked the release for plonetheme.twinapex. Version 1.0 package didn’t include media assets and ZCML configuration files. Luckily Python community reacted quickly and I got advised how to fix it.

By default, setuptools include only *.py files. You need to explicitly declare support for other filetypes in MANIFEST.in file.

Example MANIFEST.in (plonetheme, built in PyDev):

recursive-include plonetheme *
recursive-include docs *
global-exclude *pyc
global-exclude .project
global-exclude .pydevproject

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

XHTML mobile profile transformer and cleaner for Python

Mobile phones, and especially mobile site validators, are very picky about the validy of XHTML. It must not be any XHTML, but special mobile profile XHTML. Also, search engines like Google, will punish you in the mobile search results if your site fails to conform to mobile profile.

This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification.

To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which  uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools.

Highlights

• Turn any incoming HTML/XHTML to mobile profile compatible
• Enforce ALT text on images – especially useful for external tracking images (feedburner tracker). ALT texts are required by XHTML MP.
• Protect against Cross-Site Scripting Attacks (XSS) and other nastiness, as provided by lxml.xhtml.clean
• Unicode compliant – eats funky characters

As an example we integrated gomobile.xhtmlmp  to Feedfeeder Plone add-on product.

Enjoy.

Debugging Plone with Eclipse and PyDev remote debugger

Plone is a very heavy Python CMS containing thousands of modules. PyDev is Python extension for Eclipse IDE. Plone doesn’t run very well in PyDev debug mode:

• Debug mode is many times slower than normal Python execution
• At least on  some Linux system OS limitations prevent proper running [1]

But you can still use Eclipse and PyDev remote debugger to debug your software.

• All communications happen over sockets so you can debug software running on test server besides local machine
• Downsite is that you need to manually place pydevd.settrace() call to your code instead of just right clicking the line and choosing Set Breakpoint from menu

In both cases you need to use a custom non-forking Plone launch script from Plone and Eclipse tutorial which prevents Zope to escape itself into a child process and keeps the debug process information intact.

PyDev Extension remote debugger

Note that you need the commercial PyDev Extension license, this does not work with the basic plug-in. Free evaluation trial available. Get it here.

Benefits

• GUI stepping support
• Inspect Python run-time environment and variables using GUI
• Better integration with Eclipse console

Setting up local remote debugger

0. Install Eclipse. Do a local install, do not trust your Linux distribution package installation. http://johnpaulett.com/2009/06/26/install-eclipse-galileo-3-5-on-ubuntu-jaunty-9-04/

1. Set up idelauncher.py for your Zope application server

2. Right-click your project in Eclipse. Put pydevd in your PyDev – PYTHONPATH -> External libraries. Mine (local Eclipse 3.5 install) is /home/moo/eclipse3.5/plugins/org.pyhon.pydev.debug.XXX/pysrc

3. Switch to Eclipse Debug perspective. Start pydevd server by clicking the icon with a bug and P letter.

3. Start your Plone instance / unit tests idelauncher.py using Run (play icon – not debug). The debug server stays active even if you terminate the application launcher run.

4. When your code runs to *import pydevd ; pydevd.settrace()* (no underscore in settrace()) you should be able to start examining your site in the debugger

Now you can

• Examine and edit local variables in Variables view
• Execute Python console commands
• Examine different threads (kind of pointless with Plone…)

Note: Python console behaves funnily. You need to enter the commands to Console -> Display selected console (computer icon) -> Debug server view and output will appear  Console -> Display selected console (computer icon) -> your launcher name view.

Remote debugging not enabled

The following exception means that your pydevd server is not running. Click the P bug icon before running the launcher.

Traceback (most recent call last):
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 624, in trace_dispatch
    return dbFrame.trace_dispatch(frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 107, in trace_dispatch
    raise
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 102, in trace_dispatch
    self.doWaitSuspend(thread, frame, event, arg)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd_frame.py", line 25, in doWaitSuspend
    self._args[0].doWaitSuspend(*args, **kwargs)
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 532, in doWaitSuspend
    def doWaitSuspend(self, thread, frame, event, arg): #@UnusedVariable
  File "/home/moo/eclipse3.5/plugins/org.python.pydev.debug_1.4.7.2843/pysrc/pydevd.py", line 539, in doWaitSuspend
    self.writer.addCommand(cmd)

Pydevd and Linux Resource temporarily unavailable

[1] I get the following Resource temporarily unavailable traceback when I try to run PyDev debugger against Plone – I tried to pindown the reason using strace, but no luck. Applies both for unit testing and for the actual instance launch.

Installing PloneLanguageTool ... done (0.174s)
Running Products.SitsHospital.tests.rememberbase.RememberProfileLayer tests:
  Set up Products.PloneTestCase.layer.ZCML
Traceback (most recent call last):
...
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 688, in setup_layer
    setup_layer(base, setup_layers)
  File "/home/moo/sits/parts/zope2/lib/python/zope/testing/testrunner.py", line 692, in setup_layer
    layer.setUp()
  File "/home/moo/sits/parts/plone/PloneTestCase/layer.py", line 17, in setUp
...
  File "/usr/lib/python2.4/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.4/xml/sax/expatreader.py", line 348, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/xmlconfig.py", line 349, in endElementNS
    self.context.end()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 544, in end
    self.stack.pop().finish()
  File "/home/moo/sits/parts/zope2/lib/python/zope/configuration/config.py", line 692, in finish
    actions = self.handler(context, **args)
  File "/home/moo/sits/parts/plone/PlacelessTranslationService/patches.py", line 27, in compile_translations
    func(*args, **kwargs)
  File "/home/moo/sits/eggs/zope.i18n-3.7.0-py2.4.egg/zope/i18n/zcml.py", line 57, in registerTranslations
    for language in os.listdir(path):
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/moo/sits/parts/instance/etc/site.zcml", line 5.2-5.37
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/configure.zcml", line 6.2-6.30
    ZopeXMLConfigurationError: File "/home/moo/sits/parts/zope2/lib/python/Products/Five/i18n.zcml", line 24.4-24.52
    OSError: [Errno 11] Resource temporarily unavailable: '/home/moo/sits/parts/zope2/lib/python/zope/app/locales'

pygame goes mobile

Pygame, the easiest way to make computer games in the world, has just reached your pocket.

Check our announcement at

http://discussion.forum.nokia.com/forum/showthread.php?t=152969

Python-like urlparser module for Javascript

I had to deconstruct and reconstruct URLs from pieces when doing advanced Javascripting for Plone.

I found this nice library from Denis Laprise. However, it had a bug with fragment extractor and lacked reconstruction possibilies. So I decided to make a new version.

Download urlparse.js version 0.2. thank you

Couple of examples:

var p = new Poly9.URLParser('http://user:password@poly9.com/pathname?arguments=1#fragment');
p.getHost() == 'poly9.com';
p.getProtocol() == 'http';
p.getPathname() == '/pathname';
p.getQuerystring() == 'arguments=1';
p.getFragment() == 'fragment';
p.getUsername() == 'user';
p.getPassword() == 'password';

var p = new Poly9.URLParser("http://localhost:8080/path);

p.setQuerystring("foo=bar");
p.setFragment("anchor");
p.setPort(7070);

var url = p.getURL() // http://localhost:7070/path?foo=bar#anchor