The first day of Sauna Sprint was spent introducing new EESTEC people to Plone. Almost everyone got Ubuntu installed, there was a short crash course for IRC and then we started installing Plone.

We had a bunch of EESTEC students as a test subject and our goal was to map obstacles in the process of installing Plone for local computer for development. We got some interesting results.

The data was collected on whiteboard where everyone was free to express any problem he or she had during the installation. The advisors also monitored the user group for possible problems and made notes.

Aperture Science test subjects installing Plone in hope they’ll get a cake if they success

Good job, but the cake was a lie.

Plone is #%!!”€&! difficult, but we can fix it

The comments below are collected from the whiteboard.

Don’t make assumptions in the documentation – people are tabula rasa (i.e. how to use cd command and so on)

Documentation needed to install gcc, PIL dependencies etc. on OS level

No good instructions how to set-up Python with virtualenv (was being fixed in the sprint). Buildout actually does not work with the system Python out of the box: you need to break your OS http://stackoverflow.com/questions/5818100/buildout-tries-to-update-system-wide-distribute-installation-and-refuses-to-run

People try to run buildout by “cd bin ; ./buildout” – it does not work and the error message is confusing

Unified Installer – ZEO installation is broken (need clarification what was broken)

Detect Python version in buildout and don’t try to run on incompatible version

When creating first Plone site input field has label “Path identifier” – people try to enter Plone FS path there

README.txt in Unified Installer needs updates

ZopeSkel template plone4_buildout should autodetect the latest Plone version (now was 4.0.1)

Update the out of the box Plone installation front page to contain more developer friendly instructions too

plone.org/downloads: Show add-on version compatibility as the first item on the page – if now compatibility set warn that the compatibility is unknown (same kind of warning if no updates for one year)

Have an easy way to report broken packages

Force a process to improve add-on documentation in p.org, especially regarding installation. Only 1/5 themes was successfully installed by the test personel without guidance, mostly due to lack of instructions and version incompatibilities.

Some work in progress on PiratePad regarding the installation of Plone.

Evening program

We had Sauna time in Hervanta, student style. EESTEC lived up to their reputation as party animals.

Delicious pizza was baked, though the audience loudly disagree when the pizza was burnt enough to be eaten

Magic Toni Sause

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Download speed

We see around 33% – 40% increase

Response time

Though this is little harder to read, due to the sampler having issues, the response time has dropped from 250 ms to near 100 ms.

The update process

Here is the process what we did.

Create a duplicate of buildout.cfg with Plone 4 based recipes and run bootstrap.py with Python 2.6.

Uninstall a lot of unneeded add-ons which had collected cruft over the years: plone4artists audio, video, LinguePlone.

Uninstall Products.CacheSetup. This is replaced with plone.app.caching in Plone 4.

Use wildcard.fixpersistentutilities to clean up leftovers of broken add-on uninstallers (Singing & Dancing, p4a).

Remove broken persistent tools (Notifica, Search & Replace, etc.).

Make sure that the site custom theme extends Plone 4 Classic Theme.

Replace old collective.slideshow with Products.Carousel. The migration script is available in collective.developermanual.

Replace old collective.fancyzoomview with Products.PipBox. The migration script and PipBox code are available in collective.developermanual.

Copy Data.fs from Plone 3.3 to new Plone 4 buildout.

… in this point we move from Plone 3 buildout to Plone 4 buildout …

Run portal_migration.

Converted Language portal_catalog index from LanguageIndex to FieldIndex (instructions available in c.developermanual).

Rebuild the catalog.

Reset theme on portal_skins properties tab. Shuffle portal_skins layers until they work.

Migrate custom Archetypes image scale sizes from monkey-patch to plone.app.imaging and imaging_properties.

Remove kukit-devel in portal_javascripts.

In portal_transforms -> safe_html, disable_transform=1.

Change custom folder listing from atct_album_view32 to atct_album_view.

Futher info

Some of these upgrade tips are collected to collective.developermanual.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Instructions assume you are installing Nginx via buildout.

Nginx sticky session can be loaded from here

• Nginx sticky sessions module

Manually extract nginx-sticky-module under src folder:

cd src
wget http://nginx-sticky-module.googlecode.com/files/nginx-sticky-module-1.0-rc2.tar.gz

Then add it do Nginx configure options in buildout part:

[nginx-build]
recipe = zc.recipe.cmmi
url = http://sysoev.ru/nginx/nginx-0.7.65.tar.gz
extra_options = --add-module=${buildout:directory}/src/nginx-sticky-module-1.0-rc2

Now test recompling Nginx via buildout:

mv parts/nginx-build/ parts/nginx-build-old # Make sure full rebuild is done
bin/buildout install nginx-build

See that it compiles without erros. Here is the line of compiling sticky module:

gcc -c -O -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Wunused-function -Wunused-variable -Wunused-value -Werror -g   -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs -I src/http -I src/http/modules -I src/mail \
        -o objs/addon/nginx-sticky-module-1.0-rc2/ngx_http_sticky_module.o \

Now add sticky to the load balancer section of nginx config:

[balancer]
recipe = gocept.nginx
nginx = nginx-build
...
http {
        client_max_body_size 64M;
        upstream zope {
                sticky;
                server ${hosts:client1}:${ports:client1} max_fails=3 fail_timeout=30s;
                server ${hosts:client2}:${ports:client2} max_fails=3 fail_timeout=30s;
                server ${hosts:client3}:${ports:client3} max_fails=3 fail_timeout=30s;
        }

Reinstall nginx balaner configs and start-up scripts:

bin/buildout install balancer

See that generated configuration is ok:

bin/nginx-balancer configtest

Restart Nginx:

bin/nginx-balancer stop ;bin/nginx-balancer start

Check that some (non-anonymous) page has route cookie set by sticky module:

Huiske-iMac:tmp moo$ wget -S http://yoursite.com/sisalto/secret-travel-destination-infoa
--2011-03-21 21:31:40--  http://yoursite.com/sisalto/secret-travel-destination-infoa
Resolving yoursite.com (yoursite.com)... 12.12.12.12
Connecting to yoursite.com (yoursite.com)|12.12.12.12|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/0.7.65
  Content-Type: text/html;charset=utf-8
  Set-Cookie: route=7136de9c531fcda112f24c3f32c3f52f
  Content-Language: fi
  Expires: Sat, 1 Jan 2000 00:00:00 GMT
  Set-Cookie: I18N_LANGUAGE="fi"; Path=/
  Content-Length: 41471
  Date: Mon, 21 Mar 2011 19:31:40 GMT
  X-Varnish: 1979481774
  Age: 0
  Via: 1.1 varnish
  Connection: keep-alive

Now test it by doing manual session related activity and see that your shopping cart is not “lost”.

More info

• http://code.google.com/p/nginx-sticky-module/source/browse/trunk/README
• http://nathanvangheem.com/news/nginx-with-built-in-load-balancing-and-caching

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• You are using MySQL installation from mysql.com (and not e.g. from Macports)
• You want to install MySQL-python egg using easy_install or buildout

You’ll get an error

Getting distribution for 'MySQL-python'.
install_dir /Users/moo/project/eggs/tmp8-j13e
sh: mysql_config: command not found
Traceback (most recent call last):
 File "<string>", line 1, in ?
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1855, in main
 with_ei_usage(lambda:
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage
 return f()  
...

 31, in <lambda>
 {'__file__':setup_script, '__name__':'__main__'}
 File "setup.py", line 15, in ?
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 43, in get_config
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 24, in mysql_config
EnvironmentError: mysql_config not found
An error occured when trying to install MySQL-python 1.2.3.Look above this message for any errors thatwere output by easy_install.
While:
 Installing instance.
 Getting distribution for 'MySQL-python'.
Error: Couldn't install: MySQL-python 1.2.3

The reason is that MySQL installer puts MySQL under /usr/local and does not expose MySQL utilites directly to path. MySQL-python expects to find utility mysql_config to configure compile time paths when building native bindings using GCC.

Workaround:

sudo -i
cd /usr/bin
ln -s /usr/local/mysql-5.1.46-osx10.5-x86_64/bin/mysql_config .

Then rerun installer.

Though it is a bit harsh solution to modify system folder, it gets the thing done in simple manner.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

If you install multiple Pythons, the latter installations might not become active in the registry automatically, and your Python wrapper still rely on the old version. This leads to version incompatibilities and you are unable to start the Python applications.

Since only one Python interpreter can be activate at a time, it is little bit difficult to develop multi-version Python code Windows.
This kind of situation could be that you develop Plone 3 sites (Python 2.4) and Plone 4 sites (Python 2.6) simultaneously.

Below is regpy.py code which changes the active Python interpreter. The orignal author is unknown, I picked up this code from some paste board long time ago. Just run this code with your Python and the running interpreter becomes active.

Example how to activate alternative Python interpreter:

C:\Plone\python\python.exe regpy.py

regpy.py:

import sys

from _winreg import *

# tweak as necessary
version = sys.version[:3]
installpath = sys.prefix

regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
installkey = "InstallPath"
pythonkey = "PythonPath"
pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
    installpath, installpath, installpath
)

def RegisterPy():
    try:
        reg = OpenKey(HKEY_LOCAL_MACHINE, regpath)
    except EnvironmentError:
        try:
            reg = CreateKey(HKEY_LOCAL_MACHINE, regpath)
            SetValue(reg, installkey, REG_SZ, installpath)
            SetValue(reg, pythonkey, REG_SZ, pythonpath)
            CloseKey(reg)
        except:
            print "*** Unable to register!"
            return
        print "--- Python", version, "is now registered!"
        return
    if (QueryValue(reg, installkey) == installpath and
        QueryValue(reg, pythonkey) == pythonpath):
        CloseKey(reg)
        print "=== Python", version, "is already registered!"
        return
    CloseKey(reg)
    print "*** Unable to register!"
    print "*** You probably have another Python installation!"

if __name__ == "__main__":
    RegisterPy()

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

This buildout is especially useful to get Python 2.4 properly running under the latest Ubuntu 10.04 Lucid Lynx. This is because Ubuntu repositories won’t ship with Python 2.4 packages anymore.

The installation will also include static compilation of some very popular libraries. These are dependencies for other Python packages including, but not limited, to

• libjpeg
• Python imaging library
• readline

Prerequisites

• Some Python version is installed (OS default)
• GCC compiler is installed (sudo apt-get install build-essential)
• Subversion tool is installed (sudo apt-get install subversion)

Running it

svn co http://svn.plone.org/svn/collective/buildout/python/
cd python
python bootstrap.py
bin/buildout

Using it

All Pythons are under virtualenv installations. This means that you can activate one Python configuration for your shell once easily (python command will run under different Python versions).

Activating Python 2.4

source python/python-2.4/bin/activate
(python-2.4)moo@murskaamo:~/code$ python -V
Python 2.4.6

Check that Python Imaging Library works

python
Python 2.4.6 (#1, Jul 16 2010, 10:31:46)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL

(No exceptions raised, Python Imaging Library works well).

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Paste script is a part of Python Paste web development utilities.

Its paster create command is used by various Python frameworks to generate skeleton code for a development project.

Paster and local commands

Besides generic project templates, paster provides local commands which are project aware commands to add more modules into an existing project. Local commands are made available by paster extensions. For example, ZopeSkel product has various local commands to generate skeletons into Plone/Zope projects automatically

• Views
• Content types
• Forms
• Portlets

… and so on.

For example, you could generate a project template for Plone add-on module and then create content types there using a local paster command. The local commands become available when you execute paster command in the folder of your project.

Example:

paster create -t archetype myorg.customploneaddon
cd src/myorg.customploneaddon

# Now new paster commands are available
paster

Usage: ../../bin/paster COMMAND
usage: paster [paster_options] COMMAND [command_options]

...

Commands:
  ...

ZopeSkel local commands:
  addcontent   Adds plone content types to your project

Above, ZopeSkel paster template adds its addcontent templates. Now you can use addcontent local command to contribute to the existing project

paster addcontent -t contenttype MyShinyWebPage

ZopeSkel

For more information how to use paster to create add-ons and add-on submodules for Plone, see here.

To see list of available paster local commands, run paster command

../../bin/paster addcontent --list

… in your development project. For ZopeSkel specific projects the output should be something like this:

Available templates:
    atschema:     A handy AT schema builder
    contenttype:  A content type skeleton
    form:         A form skeleton

How paster local commands work

paster reads (evaluates) setup.py file which declares a Python egg. If it founds paster_plugins section there, it will look for local commands there. For example, Plone project templates declare the following paste_plugins in setup.py:

paster_plugins = ["ZopeSkel"]

setup.py install_requires

Python modules can dependencies to other modules using setup.py and install_requires section. For example, a Plone add-on might read:

install_requires=['setuptools',
                  # -*- Extra requirements: -*-
                  "five.grok",
                  "plone.directives.form"
                  ],

This means that when you use setuptools/buildout/pip/whatever Python package installation tool to install a package from Python Package Index (PyPi) it will also automatically install dependency packages declared in install_requires.

paster and install_requires

This is where things usually go haywire.

Let’s assume you are using paster in a project which contains N python packages. You probably use an external configuration system to manage your installed applications and their versions to make repeatable deployments possible (hint: buildout is gaining traction in Python community lately).

Paster is not aware of this external Python package configuration set (paster cannot see them in its PYTHONPATH). So what happens when you try to execute paster create which reads setup.py containing install_requires and encounters dependencies?

Paster will try automatically download and install them locally in that folder.

Plone and Zope ecosystem contains over hundreds of reusable components, in nice dependency hierarchy. paster create would try to pull all them in to your source tree as *.egg folders. See discussion here.

Do not ever run sudo easy_install ZopeSkel. Do not ever run paster local commands using a paster command from your system-wide Python installation.

Paste and buildout

If you are using buildout to manage your Python application deployment, you can integrate paster nicely with it.

Add to your buildout.cfg:

parts =
    ...
    paster

[paster]
recipe = zc.recipe.egg
eggs =
        PasteScript
        ZopeSkel
        ${instance:eggs}

After rerunning buildout, buildout adds paster command to bin folder.

Then you can run paster from buildout folder:

bin/paster

… or in a buildout managed project under src folder…

../../bin/paster

This way paster is aware of your deployment configuration and local commands won’t explode on your face anymore.

Thanks Martin Aspeli to helping with how buildout + paster should be done.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

For example, when installing Satchmo shop under virtualenv:

../bin/clonesatchmo.pyhe Python Imaging Library is not installed. Install from your distribution binaries.

../bin/clonesatchmo.py
The Python Imaging Library is not installed. Install from your distribution binaries.

Though it clearly is there, installed by easy_install PIL command:

ls ../lib/python2.5/site-packages/PIL-1.1.7-py2.5-linux-x86_64.egg
ArgImagePlugin.py	 ExifTags.py		  GimpGradientFile.pyc...

Does anyone know if this problem is with PIL itself, eggified PIL or something else?

In any case, there is an easy workaround: use system-wide PIL (sudo apt-get install python-imaging) and symlink PIL from your site-wide installation under the isolated Python environment:

(satchmo-py25)mulli% pwd
/srv/plone/mmaspecial/satchmo-py25/lib/python2.5/site-packages
(satchmo-py25)mulli% ln -s /usr/lib/python2.4/PIL .

…

Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
  File "/tmp/tmp_G8621", line 11, in ?
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 9, in ?
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
    return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
    getattr(buildout, command)(args)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
    installed_develop_eggs = self._develop()
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
    zc.buildout.easy_install.develop(setup, dest)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
    assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError

My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while.

Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs.

There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install.

So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency.

It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t.

The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder.

Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure but there’s of course some learning curve, so the real question is should you spend time to learn it or not?

I found out that zc.buildout has some nice features like:

• Automatic requirements processing through setuptools
• Automatic (yet simple) removal of directories during recipe uninstall
• Clear structure (install(), update() & uninstall() methods)
• INI-syntax, python does have a clear syntax but INI is always clearer for a newbie
• Easy script creation (adjust python paths somewhat automatically)
• Easily repeatable
• Passing of arguments from one recipe to another
• etc.

The problems?

• It takes a while to learn zc.buildout
• It takes ‘another while’ to learn to write recipes
• Too much hassle for little things
• INI-syntax is very limited in features
• Buildout easily updates all your packages (that means also the ones you didn’t want to!)
• Lack of documentation (it has good docs to get you going.. but after a while it leaves you with open questions)
• Unnecessary overhead (for each script you launch, you’ll need a launcher script created via buildout)

There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however.

I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up.