.gitignore for Python developers

If you are using Git for version control for your Python egg and buildout development below are is a sample which you might want to put into your .gitignore file.

*.mo
*.egg-info
*.egg
*.EGG
*.EGG-INFO
bin
build
develop-eggs
downloads
eggs
fake-eggs
parts
dist
.installed.cfg
.mr.developer.cfg
.hg
.bzr
.svn
*.pyc
*.pyo
*.tmp*

Suggestions for new ignores are welcome.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Zoho integration for Python and Plone CMS

Zoho is a web application provider competing with Google Docs, Microsoft Office and Live.

Zoho provides a very wide set of browser based applications from text editing and spreadsheets to project management and customer relationship management (highlighted items should ring a bell for small software development companies). Especially the last one, CRM, is a very attractive deal as you get a hosted complex CRM application with API services for very affordable or free price. Small organizations are not necessarily rich enough to go for Salesforce API supported edition which would be 135 € / month / user.

mFabrik has been working on Zoho Python bindings as we use Zoho internally.

Zoho API is HTTP GET/POST based.

• Authentication, which is called a ticket in Zoho language, is HTTP POST with custom plain-text responses. The same authentication mechanism works in-browser (Javascript) and for machine-to-machine communication as far as I know
• Most functions can be performed as HTTP POST or GET. If you need to input compex data (like CRM leads), you’ll do it as HTTP POST of custom XML payload
• Some functions expose the output as JSON for HTTP GET, so that it can be directly consumed inside browser Javascript

mfabrik.zoho is a GPL’ed Python library which provides basic facilities for Zoho API calls. Currently the feature set is very CRM weighted, though it can be easily expanded for other Zoho applications.

mfabrik.plonezohointegration is a Plone CMS add-on product which marry Plone and Zoho together. The add-on provides a control panel where you can enter Zoho API key details for Plone. Forms for CRM lead generation are provided as standalone and as a portlet (you can see them in action on our web and blog site).

The source is hosted on Github, so you can easily start tailoring it for your own organization needs. I happily accept all merge requests, providing that unit tests for new features are included. If you do not feel comfortable with Python programming, but still want to integrate Zoho to your systems, please contact us for further help.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Easily install all Python versions under Linux and OSX using collective.buildout.python

Here are short instructions how to install all versions (2.4, 2.5, 2.6, 2.7 and 3.1) of Python interpreters on UNIX system. The instructions were tested on Ubuntu 10.04 Lucid Lynx Linux but should work on other systems as is. The installation is based of downloading, compiling and installing different Pythons and their libraries using buildout tool. A buildout configuration for doing this is maintained by a Plone community.

This buildout is especially useful to get Python 2.4 properly running under the latest Ubuntu 10.04 Lucid Lynx. This is because Ubuntu repositories won’t ship with Python 2.4 packages anymore.

The installation will also include static compilation of some very popular libraries. These are dependencies for other Python packages including, but not limited, to

• libjpeg
• Python imaging library
• readline

Prerequisites

• Some Python version is installed (OS default)
• GCC compiler is installed (sudo apt-get install build-essential)
• Subversion tool is installed (sudo apt-get install subversion)

Running it

svn co http://svn.plone.org/svn/collective/buildout/python/
cd python
python bootstrap.py
bin/buildout

Using it

All Pythons are under virtualenv installations. This means that you can activate one Python configuration for your shell once easily (python command will run under different Python versions).

Activating Python 2.4

source python/python-2.4/bin/activate
(python-2.4)moo@murskaamo:~/code$ python -V
Python 2.4.6

Check that Python Imaging Library works

python
Python 2.4.6 (#1, Jul 16 2010, 10:31:46)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL

(No exceptions raised, Python Imaging Library works well).

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Automatically generating description based on body text

Below is a sample script to automatically generate descriptions based on page body text. It is for Plone CMS, but should be applicable to any Python based CMS with some modifications.

The idea is that we take three first sentences and use them as a description.

Since Zope uses RestrictedPython for through-the-web created scripts, the user of this script cannot breach the server security (they cannot make Python calls they have no permission for). This sets some limitations for automating tasks like this, but we don’t hit those limitations in our use case.

def create_automatic_description(content, text_field_name="text"):
    """ Creates an automatic description from HTML body by taking three first sentences. 

    Takes the body text

    @param content: Any Plone contentish item (they all have description)

    @param text_field_name: Which schema field is used to supply the body text (may very depending on the content type)
    """

    # Body is Archetype "text" field in schema by default.
    # Accessor can take the desired format as a mimetype parameter.
    # The line below should trigger conversion from text/html -> text/plain automatically using portal_transforms
    field = content.Schema()[text_field_name]

    # Returns a Python method which you can call to get field's
    # for a certain content type. This is also security aware
    # and does not breach field-level security provded by Archetypes
    accessor = field.getAccessor(content)

    # body is UTF-8
    body = accessor(mimetype="text/plain")

    # Now let's take three first sentences or the whole content of body
    sentences = body.split(".")

    if len(sentences) > 3:
       intro = ".".join(sentences[0:3])
       intro += "." # Don't forget closing the last sentence
    else:
       # Body text is shorter than 3 sentences
       intro = body

    content.setDescription(intro)

# context is the reference of the folder where this script is run
for id, item in context.contentItems():
     # Iterate through all content items (this ignores Zope objects like this script itself)

     # Use RestrictedPython safe logging.
     # plone_log() method is permission aware and available on any contentish object
     # so we can safely use it from through-the-web scripts
     context.plone_log("Fixing:" + id)

     # Check that the description has never been saved (None)
     # or it is empty, so we do not override a description someone has
     # set before automatically or manually
     desc = context.Description() # All Archetypes accessor method, returns UTF-8 encoded string

     if desc is None or desc.strip() == "":
          # We use the HTML of field called "text" to generate the description
          create_automatic_description(item, "text")

# This will be printed in the browser when the script completes succesfully
return "OK"

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

How to split() strings of indefined item count to Python variables elegantly?

A common problem for  (space) separated string parsing is that there are a number of fixed items followed by some amount of optional items. You want to extract parsed values to Python variables and initialize optional values with None if they are missing.

E.g. we have option format description string with 2 or 3 items value1 value2 [optional value]

s = "foo bar" # This is a valid example option string

s2 = "foo bar optional" # This is also, with one optional item

Usually we’d like to parse this so that non-existing options are set to None. Traditonally one does Python code like below:

# Try get extration and deletion
parts = s.split(" ")
value1 = parts[0]
value2 = parts[1]
if len(parts) >= 3:
    optional_value = parts[2]
else:
    optionanl_value = None

However, this looks little bit ugly, considering  if there would always be just two options one could write:

value1, value2 = s.split(" ")

When the number of optional options grow, the boiler-plate code starts annoy greatly.

I have been thinking how to write a split() code to parse options with non-existing items to be set None. Below is my approach for a string with total of 4 options of which any number can be optional.

parts = s.split(" ", 4) # Will raise exception if too many options 
parts += [None] * (4 - len(parts)) # Assume we can have max. 4 items. Fill in missing entries with None.
value1, value2, optional_value, optional_value2 = parts

Any ideas for something more elegant?

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Using paster local commands with buildout and avoiding the infamous dependency issue

Introduction

Paste script is a part of Python Paste web development utilities.

Its paster create command is used by various Python frameworks to generate skeleton code for a development project.

Paster and local commands

Besides generic project templates, paster provides local commands which are project aware commands to add more modules into an existing project. Local commands are made available by paster extensions. For example, ZopeSkel product has various local commands to generate skeletons into Plone/Zope projects automatically

• Views
• Content types
• Forms
• Portlets

… and so on.

For example, you could generate a project template for Plone add-on module and then create content types there using a local paster command. The local commands become available when you execute paster command in the folder of your project.

Example:

paster create -t archetype myorg.customploneaddon
cd src/myorg.customploneaddon

# Now new paster commands are available
paster

Usage: ../../bin/paster COMMAND
usage: paster [paster_options] COMMAND [command_options]

...

Commands:
  ...

ZopeSkel local commands:
  addcontent   Adds plone content types to your project

Above, ZopeSkel paster template adds its addcontent templates. Now you can use addcontent local command to contribute to the existing project

paster addcontent -t contenttype MyShinyWebPage

ZopeSkel

For more information how to use paster to create add-ons and add-on submodules for Plone, see here.

To see list of available paster local commands, run paster command

../../bin/paster addcontent --list

… in your development project. For ZopeSkel specific projects the output should be something like this:

Available templates:
    atschema:     A handy AT schema builder
    contenttype:  A content type skeleton
    form:         A form skeleton

How paster local commands work

paster reads (evaluates) setup.py file which declares a Python egg. If it founds paster_plugins section there, it will look for local commands there. For example, Plone project templates declare the following paste_plugins in setup.py:

paster_plugins = ["ZopeSkel"]

setup.py install_requires

Python modules can dependencies to other modules using setup.py and install_requires section. For example, a Plone add-on might read:

install_requires=['setuptools',
                  # -*- Extra requirements: -*-
                  "five.grok",
                  "plone.directives.form"
                  ],

This means that when you use setuptools/buildout/pip/whatever Python package installation tool to install a package from Python Package Index (PyPi) it will also automatically install dependency packages declared in install_requires.

paster and install_requires

This is where things usually go haywire.

Let’s assume you are using paster in a project which contains N python packages. You probably use an external configuration system to manage your installed applications and their versions to make repeatable deployments possible (hint: buildout is gaining traction in Python community lately).

Paster is not aware of this external Python package configuration set (paster cannot see them in its PYTHONPATH). So what happens when you try to execute paster create which reads setup.py containing install_requires and encounters dependencies?

Paster will try automatically download and install them locally in that folder.

Plone and Zope ecosystem contains over hundreds of reusable components, in nice dependency hierarchy. paster create would try to pull all them in to your source tree as *.egg folders. See discussion here.

Do not ever run sudo easy_install ZopeSkel. Do not ever run paster local commands using a paster command from your system-wide Python installation.

Paste and buildout

If you are using buildout to manage your Python application deployment, you can integrate paster nicely with it.

Add to your buildout.cfg:

parts =
    ...
    paster

[paster]
recipe = zc.recipe.egg
eggs =
        PasteScript
        ZopeSkel
        ${instance:eggs}

After rerunning buildout, buildout adds paster command to bin folder.

Then you can run paster from buildout folder:

bin/paster

… or in a buildout managed project under src folder…

../../bin/paster

This way paster is aware of your deployment configuration and local commands won’t explode on your face anymore.

Thanks Martin Aspeli to helping with how buildout + paster should be done.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Eclipse plug-in for Plone/Zope/buildout based development

Few days ago Fabio announced Django plug-in for Eclipse. There also exists an effort to make Eclipse integrate smoother with Plone / Zope / Buildout world.

Please see PyPi page here.

It is not nearly as polished as Fabio’s work, as Fabio being main author of PyDev and started as a collection of company internal tools. However if you mare making many buildouts, write code > 5 Python modules at once and don’t want to start terminal everytime you need to run buildout, this plug-in is for you.

Also, it is not yet a real plug-in. It is a collection of PythonMonkey scripts. This means that to deploy the scripts you simply copy .py files them to scripts folder of any of your Eclipse projects. If you need to work on the codebase, you don’t need to start a separate Eclipse instance, but you can do it interactively through Eclipse console while you do other development. The script source code is well commented, so if you want to tune them for your own habits it should be easy.

Integrating and theming Wordpress with your CMS site using XDV

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="here/main_template/macros/master"
      i18n:domain="plone">

<body>

    <div metal:fill-slot="content">

        <div id="wordpress-content">
                <!-- Your Wordpress "left column" will go there -->
        </div>

    </div>

</body>
</html>

<?xml version="1.0" encoding="UTF-8"?>
<rules xmlns="http://namespaces.plone.org/xdv"
       xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
       xmlns:css="http://namespaces.plone.org/xdv+css">

    <!-- Remove Wordpress CSS by filtering out <style> tags-->
    <drop css:content="style" />

    <!-- Make sure that Wordpress metadata is present in <head> section -->
    <append css:content="head link" css:theme="head" />

    <!-- note: replace does not seem to handle multiple meta tags very well -->
    <drop css:theme="meta" />
    <append css:content="head meta" css:theme="head" />

    <!-- Use blog title instead of Plone page title -->
    <replace css:content="title" css:theme="title" />

    <!-- Put Wordpress sidebar to Plone's portlets section -->
    <append css:content="#r_sidebar" css:theme="#portal-column-one .visualPadding" />

    <!-- Place wordpress content into our theme content area -->
    <copy css:content="#contentleft" css:theme="#wordpress-content" />

    <!-- This mixes in Wordpress specific CSS sheet which is applied for pages
         served from Wordpress only and does not concern Plone CMS.
         This stylesheet will theme Wordpress specific tags,
         like blog posts and comment fields.
         We keep this file in Plone, but this could be served from elsewhere. -->
    <append css:theme="head">
        <style type="text/css">
           @import url(http://mfabrik.com/++resource++plonetheme.mfabrik/wordpress.css);
        </style>
    </append>

    <!-- This stylesheet is used by special spam protection plug-in NoSpamNX -->
    <append css:theme="head">
        <link rel="stylesheet" href="http://blog.mfabrik.com/wp-content/plugins/nospamnx/nospamnx.css" type="text/css" />
    </append>

    <!-- Remove Google Analytics script used for CMS site -->
    <drop css:theme="#page-bottom script" />

    <!-- Rebuild our Google Analytics code, using a different tracker id this time
         which is a specific to our blog.
      -->
    <append css:theme="#page-bottom">

        <script type="text/javascript">
                var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
                document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
        </script>

        <script type="text/javascript">
                try {
                       var pageTracker = _gat._getTracker("UA-8819100-2");
                       pageTracker._trackPageview();
                } catch(err) {
                }
        </script>
    </append>

</rules>

/* Font and block style fixes */

#wordpress-content h1 {
        border: 0;
}

#wordpress-content .post-end {
        margin-bottom: 60px;
}

#wordpress-content pre {
        width: 600px;
        overflow: auto;
        background: white;
        border: 1px solid #888;
}

#wordpress-content ul {
        margin-left: 20px;
}

#wordpress-content .post-info-date,
#wordpress-content .post-info-categories,
#wordpress-content .post-info-tags {
        font-size: 80%;
        color: #888;
}

/* Make sure that posts and comments look sane in our theme */

#wordpress-content .post {
        margin-top: 15px;
}

#wordpress-content .commentlist li {
        margin: 20px;
        background: white;
        padding: 10px;
}

#wordpress-content .commentlist li img {
        float: left;
        margin-right: 20px;
        margin-bottom: 20px;
}

#wordpress-content #commentform {
        margin: 20px;
}

#wordpress-content {
        margin-left: 20px;
        margin-right: 20px;
}

/* Make Wordpress "sidebaar" look like Plone "portlets */

.template-wordpress_listing #portal-column-one ul {
        list-style: none;
        margin-bottom: 40px;
}

.template-wordpress_listing #portal-column-one ul#Recent li {
        margin-bottom: 8px;
}

.template-wordpress_listing #portal-column-one ul#Categories a {
        line-height: 120%;
}

.template-wordpress_listing #portal-column-one h2 {
        background: transparent;
        border: 0;
        font-weight:normal;
        line-height:1.6em;
        padding:0;
        text-transform:none;
        font-size: 16px;
        color: #9b9b9b;
        border-bottom:4px solid #CDCDCD;
}

"""

 This command line Python script compiles your rules.xml to XDV XSL

 Modify it for your own needs.

 It assumes your buildout.cfg has xdv section and generated XDV
 commands under bin/

 To compile, execute in the buildout folder::

     python src/plonetheme.mfabrik/xdv.py

 To build test HTML::

     python src/plonetheme.mfabrik/xdv.py --test

 To build test HTML and preview it in browser, execute in buildout folder::

     python src/plonetheme.mfabrik/xdv.py --preview

"""

import getopt, sys
import os
import webbrowser

# rules XML for theming
RULES_XML = "src/plonetheme.mfabrik/deliverance/etc/rules.xml"

# Which XSL file to generate for compiled XDV
OUTPUT_FILE = "theme.xsl"

# Which file to generate applied theme test runs
TEST_HTML_FILE = "test.html"

# Our "theme.html" is a remote template served for each request.
# Because we are doing live integrattion, this is a HTTP resource,
# not a local file.
THEME="http://mfabrik.com/news/wordpress_listing/"

#
# External site you are theming.
# Note: must have ending slash (lxm cannot handle redirects)
#
SITE="http://blog.twinapex.fi/"

try:
    opts, args = getopt.getopt(sys.argv[1:], "pt", ["preview", "test"])
except getopt.GetoptError, err:
    # print help information and exit:
    print str(err) # will print something like "option -a not recognized"

# Convert options to simple list
opts = [ opt for opt, value in opts ]

print "Compiling transformation"
value = os.system("bin/xdvcompiler -o " + OUTPUT_FILE + " " + RULES_XML +" " + THEME)
if value != 0:
    print "Compilation failed"
    sys.exit(1)

if "-p" in opts or "--preview" in opts or "-t" in opts or "--test" in opts:
      print "Generating test HTML page"
      value = os.system("bin/xdvrun -o " + TEST_HTML_FILE + " " + OUTPUT_FILE + " " + SITE)
      if value != 0:
          print "Page transformation failed"
          sys.exit(1)

if "-p" in opts or "--preview" in opts:
    # Preview the result in a browser
    # NOTE: OSX needs Python >= 2.5 to make this work

    # Make sure test run succeeded
    url = "file://" + os.path.abspath(TEST_HTML_FILE)
    print "Opening:" + url

    # We prefer Firefox for preview for its superious
    # Firebug HTML debugger and XPath rule generator
    try:
        browser = webbrowser.get("firefox")
    except webbrowser.Error:
        # No FF on the system, or OSX which can't find its browsers
        browser = webbrowser.get()

    browser.open_new_tab(url)

python src/plonetheme.mfabrik/xdv.py

bin/xdvrun -o theme.html theme.xsl http://blog.twinapex.fi
firefox theme.xhtml

python src/plonetheme.mfabrik/xdv.py --preview

sudo -i
apt-get install libxslt1-dev libapache2-mod-apreq2 libapreq2-dev apache2-threaded-dev
wget http://html-xslt.googlecode.com/files/mod-transform-html-xslt.tgz
wget http://html-xslt.googlecode.com/files/mod-depends-html-xslt.tgz
tar -xzf mod-transform-html-xslt.tgz
tar -xzf mod-depends-html-xslt.tgz
cd mod-depends-html-xslt ; ./configure ; make ; make install ; cd ..
cd mod-transform-html-xslt ; ./configure ; make ; make install ; cd ..

a2enmod filter
a2enmod ext_filter

LoadModule depends_module /usr/lib/apache2/modules/mod_depends.so
LoadModule transform_module /usr/lib/apache2/modules/mod_transform.so

/etc/init.d/apache2 force-reload

<VirtualHost *>

    ServerName blog.mfabrik.com
    ServerAdmin info@mfabrik.com

    LogFormat       combined
    TransferLog     /var/log/apache2/blog.mfabrik.com.log

    # Basic Wordpress setup

    Options +Indexes FollowSymLinks +ExecCGI

    DocumentRoot /srv/www/wordpress

    <Directory /srv/www/wordpress>
        Options FollowSymlinks
        AllowOverride All
    </Directory>

    AddType application/x-httpd-php .php .php3 .php4 .php5
    AddType application/x-httpd-php-source .phps

    # Theming set-up

    # This chain is used for public web pages
    FilterDeclare THEME
    FilterProvider THEME XSLT resp=Content-Type $text/html

    TransformOptions +ApacheFS +HTML
    # This is the location of compiled XSL theme transform
    TransformSet /theme.xsl

    # This will make Apache not to reload transformation every time
    # it is performed. Instead, a compiled version is hold in the
    # virtual URL declared above.
    TransformCache /theme.xsl /srv/plone/twinapex.fi/theme.xsl

    # We want to apply theme only for
    # 1. public pages (otherwise Wordpress administrative interface stops working)
    <Location "/">
        FilterChain THEME
    </Location>

    # 2. Admin interface and feeds should not receive any kind of theming
    <LocationMatch "(wp-login|wp-admin|wp-includes)">
        # The following resets the filter chain
        # http://httpd.apache.org/docs/2.2/mod/mod_filter.html#filterchain
        FilterChain !
    </LocationMatch>

</VirtualHost>

#!/bin/sh
#
# Periodically update Wordpress theme to reflect changes on CMS site
#

# Recompile theme
sudo -H -u twinapex /bin/sh -c cd /srv/plone/twinapex.fi ; python src/plonetheme.mfabrik/xdv.py

# Make Apache aware of theme changes
sudo apache2ctl graceful

# Make Wordpress XDV theme to reflect changes on CMS
0,15,30,45 * * * * /srv/plone/twinapex.fi/update-wordpress-theme.sh

// http://codex.wordpress.org/Editing_wp-config.php#WordPress_address_.28URL.29
define('WP_HOME','http://blog.mfabrik.com');
define('WP_SITEURL','http://blog.mfabrik.com');

Installing Python Imaging Library (PIL) under virtualenv or buildout

I have greatly struggled to have PIL library support in isolated Python environments like virtualenv –no-site-packages.

For example, when installing Satchmo shop under virtualenv:

../bin/clonesatchmo.pyhe Python Imaging Library is not installed. Install from your distribution binaries.

../bin/clonesatchmo.py
The Python Imaging Library is not installed. Install from your distribution binaries.

Though it clearly is there, installed by easy_install PIL command:

ls ../lib/python2.5/site-packages/PIL-1.1.7-py2.5-linux-x86_64.egg
ArgImagePlugin.py	 ExifTags.py		  GimpGradientFile.pyc...

Does anyone know if this problem is with PIL itself, eggified PIL or something else?

In any case, there is an easy workaround: use system-wide PIL (sudo apt-get install python-imaging) and symlink PIL from your site-wide installation under the isolated Python environment:

(satchmo-py25)mulli% pwd
/srv/plone/mmaspecial/satchmo-py25/lib/python2.5/site-packages
(satchmo-py25)mulli% ln -s /usr/lib/python2.4/PIL .

Subversion global-ignores and .egg-info in Python/Plone development

Subversion does a good job by ignoring most of build/temporary/unwanted files by default.

However, there is one exception still existing at least in Subversion 1.6: Python egg folders. All folders whose name ends up with .egg-info should not committed or considered in version controlling actions. your.package.name.egg-info folder is generated inside your Python egg source folder when you run setup.py / setuptools.

If you are working with Python source code eggs, add the following line to your ~/.subversion/config

global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store *.egg-info *.pyc *.pyo .project .pydevproject

Otherwise development tools like Mr. Developer might get confused.