How do you prefer your documentation?

Lately there have been three long email chains related to Plone development documentation here, here and here (total ~100 messages). This little post tries to summarize the current discussion.

I think the dicussion is mostly fueled by third party developers’ frustration with the current development documentation situation. Developing for Plone is difficult, since finding references, how tos and examples for those little things you need is very hard. This is a turn off for many developers who would otherwise use this great system – high developer learning curve and gaps in the documentation makes the system useless.

Points everyone agree are

• Plone development documentation is not good
• Sphinx is good for API documentation – already happening
• Contributors are needed

Points discussed are

• Should developer documentation be more open ended (Wiki-like). This covers
  • Developer manual
  • How tos and other misc. references
  • API documentation
• Mostly, it is not about generic end user, admin or system integrator or getting started documentation (here, here)

Pro wiki-like documentation stances

• It should be more open ended (here, here, here)
• It’s better to have something messy instead of nothing at all – the current approval process discourages contribution (here)

Con wiki stances

• It should not be open ended, since this results to messy documentation (here, here)
• There should be less plone.org documents (here)
• Wikis are bad (here)
• Incomplete wikis are discouraging (here)

Let’s wait and see where this goes.

Userland templates for Plone – template engine abstraction layer for Python

I have been working with collective.easytemplate product which allows users to use template tags on various places on Plone site. Currently supporting

• Kupu
• Outgoing email actions (Content rules ones)

The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content.

I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together.

Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it.

Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like

• Applying a template
• Adding a template context variables
• Registering custom template tags

Currently collective.templateengines supports Cheetah and Django templates.

So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer?

How to unit test security declarations in Plone and Zope

Security is hard. Unit testing security in Plone seems to be even harder. Here is a fool proof example how to do it. After comments I plan to release this as plone.org How to. I hope some of these ideas could get into PloneTestCase itself, so there wouldn’t be need to reinvent the wheel on every product.

Since 2004, when I was first introduced to Plone, it has been great mystery to me how to properly unit test your content type and workflow security declarations. Archetypes itself uses ugly hack where it creates secure Python Scripts from strings in Zope and then executes them. There had to be something better, but after asking questions no one seem to know what.

Function security declarations (security.declareProtected & co.) are only effective when Python is run in restricted mode. Entering to “restricted Zope Python” has not been very well documented anywhere, until RestrictedPython package Read me got revamped. This finally gave a clue how to one could hit Unauthorized exceptions in unit testing.

To enter the promised world of sandboxed Python you need to do following

• Create a globals dictionary containing secured version of all __builtin__ functions and accessable objects
• Compile your Python code through RestrictedPython compiler
• Evaluate the result

Zope get_safe_globals() will overwrite __getattr__ with guarded_getattr, etc. providing automatic code execution level security. This information is not usable only for unit testing, but for scripting purposes also – it is a developer heaven to be able to give a sandboxed template environment to the users to play around withoutworry that they can escalate privileges.

But getting into restricted mode was not enough…  after that all kind of kinks started to hit me. Namely, in some places of Plone items are cached over the request lifecycle. Since unit tests do not create new requests, the cache will contain invalid values. Here borg.localroles bit me badly – I had to dig through the security management layers manually to see why the unit test code was giving bad results. Maybe it would be wise to have a flag for caches and disable them when running on a test layer?

Below is the my example code for normal Document content type and simple_publication_workflow. All sandboxed code are declared in independend functions, but it is easy to pass arguments for them. If there is no need to reuse the sandboxed functions, I recommend use Python lambda: function declaration.

Functions which should succesfully pass sandbox testing are evaluated using self.execUntrusted(). Functions which are expect to fail are evaluated using self.assertUnauthorized().

import unittest

# Zope security imports
from AccessControl import getSecurityManager
from AccessControl.SecurityManagement import newSecurityManager
from AccessControl.SecurityManagement import noSecurityManager
from AccessControl.SecurityManager import setSecurityPolicy
from AccessControl import ZopeGuards
from AccessControl.ZopeGuards import guarded_getattr, get_safe_globals
from AccessControl.ImplPython import ZopeSecurityPolicy
from AccessControl import Unauthorized

# Restricted Python imports
from RestrictedPython import compile_restricted
from RestrictedPython.Guards import safe_builtins
from RestrictedPython.SafeMapping import SafeMapping

from zope.component import getUtility, getMultiAdapter, getSiteManager
from Products.CMFCore.tests.base.security import UserWithRoles
from Products.CMFCore.WorkflowCore import WorkflowException
from Products.CMFCore.utils import getToolByName

__docformat__ = "epytext"
__author__ = "Mikko Ohtamaa <mikko@redinnovation.com>"
__license__ = "BSD"

class WorkflowTestCase(PloneTestCase):
    """ Test workflow access rights. """

    def afterSetUp(self):
        self.workflow = getToolByName(self.portal, 'portal_workflow')
        self.acl_users = getToolByName(self.portal, 'acl_users')
        self.types = getToolByName(self.portal, 'portal_types')
        self.registration =  getToolByName(self.portal, 'portal_registration')
        self.membership =  getToolByName(self.portal, 'portal_membership') 

        # Create a normal registered portal member
        # to be used in tests
        self.registration.addMember("testmember", "secret", ["Member",], properties={ 'username': "testmember", 'email' : "foobar@foobar.com" })

        # Set verbose security policy, making debugging Unauthorized
        # exceptions great deal easier in unit tests
        setSecurityPolicy(ZopeSecurityPolicy(verbose=True))

    def clearLocalRolesCache(self):
        """ Clear borg.localroles cache.

        borg.localroles check role implementation caches user/request combinations.
        If we edit the roles for a user we need to clear this cache,
        """
        from zope.annotation.interfaces import IAnnotations
        ann = IAnnotations(self.app.REQUEST)
        for key in ann.keys():
            del ann[key]

    def loginAsPortalMember(self, id):
        ''' Login as a normal portal member.

        @param id. username
        '''
        self.login(id)

    def _execUntrusted(self, debug, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place.

        Calls func() in an sandboxed environment. The security mechanism
        should catch all unauthorized function calls (declared
        with a class SecurityManager).

        Security is effective only inside the function itself -
        The function security declarations themselves are ignored.

        @param func: Function object
        @param args: Parameters delivered to func
        @param kwargs: Parameters delivered to func
        @param debug: If True, break into pdb debugger just before evaluation
        @return: Function return value
        """

        # Create global variable environment for the sandbox
        globals = get_safe_globals()
        globals['__builtins__'] = safe_builtins
        globals['_getattr_'] = guarded_getattr

        # Create variable context available in the restricted Python
        data = { "func" : func,
                "args" : ZopeGuards.SafeIter(args),
                "kwargs" : kwargs } # TODO: Do we need to map this to SafeMappings?

        globals.update(data)

        # Our magic code
        body = """func(*args, **kwargs)"""

        # The following will replace all function calls
        # in the code with Zope call guard proxies
        code = compile_restricted(body, "<string>", "eval")

        # Here is a good place to break in
        # if you need to do some ugly permission debugging
        if debug:
            import pdb
            pdb.set_trace()

        return eval(code, globals)

    def execUntrusted(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place. """
        return self._execUntrusted(False, func, *args, **kwargs)

    def execUntrustedDebug(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python debug environment with Zope security in place. """
        return self._execUntrusted(True, func, *args, **kwargs)

    def assertUnauthorized(self, func, *args, **kwargs):
        """ Check that calling func with currently effective roles will raise Unauthroized error. """
        try:
            self.execUntrusted(func, *args, **kwargs)
        except Unauthorized, e:
            return

        raise AssertionError, 'Unauthorized exception was expected'

    def test_document_workflow_access(self):
        """ Check that anonymous users cannot access diagnosis in unwanted state. """

        def check_set_access(doc, text="foobar"):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            doc.setText(text)

        def check_read_access(doc):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            return doc.getText()

        def check_workflow_action(portal, action):
            """ Publish the document.

            Stresses secure workflow execution
            """
            portal.portal_workflow.doActionFor(portal.doc, action)

        # Login as a manager and create
        # an item which is initially private page to play around with
        self.loginAsPortalOwner()
        self.portal.invokeFactory("Document", "doc")
        doc = self.portal.doc
        # Item is private by default and editably by creator
        self.execUntrusted(check_set_access, doc)
        self.logout()

        # Anonymous cannot access the document when it's private
        self.assertUnauthorized(check_read_access, doc)
        self.assertUnauthorized(check_set_access, doc)       

        # Relogin as a normal member and see we cannot access the item
        self.loginAsPortalMember("testmember")
        self.assertUnauthorized(check_set_access, doc)
        self.logout()

        # Now relogin as the manager and share manager role with a member
        self.loginAsPortalOwner()
        self.membership.setLocalRoles(obj=doc,
                      member_ids=["testmember"],
                      member_role="Owner",
                      reindex=True)
        # IMPORTANT: This is a very invisible feature of Plone 3.1 -
        # setLocalRoles is ineffective in unit tests unless the cache is cleared
        self.clearLocalRolesCache()
        self.logout()

        # Relogin as a normal member and now we should be able to edit the document
        self.loginAsPortalMember("testmember")
        doc = self.portal.doc
        # Rich text is automatically paragraphed unless it
        # begins with HTML element
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.execUntrusted(check_set_access, doc)
        self.execUntrusted(check_workflow_action, self.portal, "submit")
        # Only site manager can publish items
        try:
            self.execUntrusted(check_workflow_action, self.portal, "publish")
            raise AssertionError("Publishing as normal member should not be possible")
        except WorkflowException:
            # WorkflowException: No workflow provides the '${action_id}' action.
            pass
        self.logout()           

        # Now the portal owner publishes the document
        self.loginAsPortalOwner()
        self.execUntrusted(check_workflow_action, self.portal, "publish")
        self.logout()

        # Anonymous should now have read access/no edit
        self.execUntrusted(check_set_access, doc)
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")      

        # Member should be still able to read and edit the document
        self.loginAsPortalMember("testmember")
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.logout()

def test_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(WorkflowTestCase))
    return suite

Mysterious buildout error – missing docs/HISTORY.txt file

I was getting the following error with Plone buildout

…

Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
  File "/tmp/tmp_G8621", line 11, in ?
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 9, in ?
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
    return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
    getattr(buildout, command)(args)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
    installed_develop_eggs = self._develop()
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
    zc.buildout.easy_install.develop(setup, dest)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
    assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError

My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while.

Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs.

There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install.

So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency.

It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t.

The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder.

Designing a high usability Plone theme

This is my brain dump of instructions for artists how to design good Plone themes. I hope I can receive some comments, some feedback from the artists itself and then publish this as a plone.org tutorial.

Often external artist is used to design a site theme. Artists might or might not have seen Plone, artists might or might not have any basic usability know how. This article should explain the elements which “must be there” to make a match between the theme and Plone easily.

The basic layout

This document describes the elements of multilingual high usability Plone theme. It is based on fluid div layout, meaning that the content stays very readable on small screens or when CSS is not loaded (screen readers). See the example layout.

The layout must not break down when user is using non-default font size. E.g. all element accept two rows text, even if the default case is usually one row.

Plone layout

Here we are designing a “normal” site theme where Plone is used to publish textual content for external readers. This might not always be the case – for example if Plone is used as a professional tool one might want to use all available screen space to display as much as possible action shortcuts to make the tool to quick to use. The latter is actual case I have seen in medical applications.

Plone layout is formed by seven main parts.

The layout must be designed so that

• The maximum width of content area newer exceeds a certain amount (1200px) so that characters-per-line count stays readable on wide screen displays. Left body padding and right body padding fill the wide screen gap to keep the content in sane max-width.
• Left portlets and right portlets parts can be missing, giving space to content part
• Header and footer are resizable. Usually logo is at the left site of the header and actions at the right side of the header. These sections should not overlap with the width of 780 px.
• Left body padding and right body padding can have shadow effect on the central column a la blogger.

Alternative layout cases

The layout must be formed from such a blocks that left or right portlets can be easily dropped without breaking the layout.

Right portlets missing:

Both portlets missing (front page view):

Header elements

The header should have the following elements

• Logo
• Language selector – either a drop down list, flag icons or language names
• Site action links with icons: site map, accessibility help, toggle easy read (make fonts larger), contact
• Search box
• Section (main level) navigation tabs
• Personal bar. Contains the username and log out buttons. Only available when logged in. To see an example, create plone.org account and login to ploen.org.

The header must scale between 780 – 1280 px. Section navigation tabs may trigger drop down menus, see http://www.jyu.fi/

Content area elements

The content area contains

• Content title
• Content description: one paragraph, no HTML formatting
• Breadcrumbs path bar displaying the position in the navigation hierarchy. Example.
• Content actions: send to, print, RSS feed. Icons with or without text labels. New Plone versions place these at the bottom, but they used to be at the top.
• Document byline: creator name w/link, creation date or edit date. This is usually only available for logged in members.
• Notification messages: When user performs actions (e.g. save) this feedback box is rendered with white i(nfo) letter in blue circle having a message like “Save succeeded.”
• The main body text and images

The whole portlet section can be dropped, making space for the content.

Portlets

Portlets are boxes on the left and right side of the content containing section specific actions.

Example portlets:

• Navigation

• Search (not the header search)

• News

• Recently changed

The portlet consists of

• Title (usually link)
• Content area
• Optional More link, located at the right bottom corner.

See http://www.siggraph.org/ for creative use of portlets.

Footer elements

The footer has

• Copyright clause
• Link to about page
• Contact information

The footer must scale between 780 – 1280 px.

Complete picture

Special cases

These are often required and might shoot you into a foot

• The current section (main level folder) where the user is located is identified somehow: a changing big topic picture, text. For example, see http://www.archred.com/updates/news and you see that there is big Updates title.
• To make more space available for front page or top level content, breadcrumbs and portlets are missing. A special front page animation may be used.See http://www.archred.com
• Breadcrumbs is also often disabled for top level elements
• There is a special print.css which is loaded when the document is printed. It sets display:none to porlets, header and footer so that only the main content area is printed.
• Visual editor (Kupu) text might require special CSS rules to have the same formatting rules as in non-edit mode to get real WYSIWYG.
• A special attention must be paid for the style of generic HTML elements like a, h1, h2 and p. They are used on the content area and outside it, often with different colors.

Colors

Plone uses mechanism to have color variables in CSS. See base_properties.prop to get an idea what colors there are and try to guess how they are used.

Icons

Plone uses generic icon mechanism to apply icons to any action available on site. Icons are 16×16, transparent with one pixel transparent border leaving 14×14 pixels for the content.

The icons should preferably be suitable for dark and light backgrounds. This might be hard to achieve, though, so it is suitable to use ligh background icons, since this is the Plone default.

Actions

• Search
• Smaller font
• Larger font
• Print
• Send to (email)
• Contact
• Sitemap
• Member icon
• Settings
• Log out
• Login
• RSS feed

Language flags

Plone default flags can be used

• Finnish
• English
• Swedish
• etc.

Content icons

These reflect different Plone content types

• Page
• Folder
• News
• Event
• Collection
• File

Link icons

Plone uses Javascript to apply special icons for external links

• HTTP link (globe)
• HTTPS link (lock)
• mailto link (letter)

Favorite icon

• The site bookmark icon, the one seen in the address bar, uses normal favicon.ico practices

Speeding up Plone loading with PTS_LANGUAGES

If you are not a Finnish speaker (like 99,9% of you) you might not want to (re)load Finnish and other unwanted language catalogs during the Plone start up. This is possible for Plone 3.1, as Reinout van Rees explains (found out afterwards).

For your Plone launcher, set environment variables (space separated list)

PTS_LANGUAGES=en mylanguagecodehere

If your Data.fs is not fresh (i.e. you have an existing Plone instance) there is still one task to do. Go to Placess Translation Service in Zope. Delete all translation catalogs. If there exists a translation catalog entry in ZODB a reload event seem to be triggered even though PTS_LANGUAGES settings is effective. Restart Zope. Maybe this is a bug? Do this on a development box only – this code seems to be quite new.
The magic code is in PlacelessTranslationService/load.py.

Updated: Eclipse web developer plug-in memo

Below are my personal notes what plug-ins are needed to get “perfect” Eclipse web development set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.

This post is update to previous Eclipse web developer plug-in memo post. New versions are available and some plug-ins have become deprecated. This blog post reflects those changes.

These instructions are good for:

• Python developer
• PHP developer
• Java developer

Choosing Eclipse distribution

• On Window, use EasyEclipse
• On Linux, use Eclipse provided by the distribution – Eclipse links against the embedded Mozilla browser and this is distribution specific – EasyEclipse has some issues here. For Ubuntu users:

sudo apt-get install sun-java6 eclipse

• On OSX, use EasyEclipse

EasyEclipse bundles some of the stuff listed here with it – when using EasyEclipse you don’t need to have separate PyDev and Subclipse downloads.

Eclipse for 64-bit Linux has various problems. You might want to run 32-bit Eclipse (another relevant blog post). When you use Linux distribution specific Eclipse install, all your personal Eclipse files go to .eclipse folder under your home folder.

Installing plug-ins

Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location.

Python

PyDev is a plug-in for Python and Jython development. It has enhanced commercial extensions for professional developers with more intelligent autocomplete and debugger.

Site URL: http://pydev.sourceforge.net

PyDev Eclipse update URL: http://pydev.sourceforge.net/updates/

PyDev extensions Eclipse update URL (this commercial, but worth of every penny): http://www.fabioz.com/pydev/updates

PDT

PDT download provides Eclipse, HTML editor, PHP editor and CSS editor.

Site URL: http://www.eclipse.org

Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/

Java

If you need to do J2EE development use IBM’s Web Tools Platform. If you don’t need Java capabilities don’t install these, since they just bloat Eclipse and make the start up time worse.

Subclipse

Subclipse provides Subversion version control integration to Eclipse.

Eclipse update site URL: http://subclipse.tigris.org/update_1.4.x/

In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules.

Aptana Studio

Aptana Studio is state-of-the-art Web 2.0 development suite for Eclipse. It has Javascript, CSS and HTML editors. It supports various Javascript libraries out of the box and has support for Firefox and IE in-browser Javascript debugging.

Eclipse update site URL: http://update.aptana.com/update/studio/3.2/site.xml

ShellEd

Syntax coloring for Unix shell scripts

Project site: http://sourceforge.net/projects/shelled

SQL Explorer

SQL terminal and SQL editor with some GUI capabilities.

Eclipse update site URL http://eclipsesql.sourceforge.net/

SQL Explorer needs MySQL JDBC driver. Download from here. Install MySQL connector by extracting the file and adding it from SQL Explorer preferences.

Zope Zeo vs. standalone setups

We do some Plone development here at Redi. As known, Plone is a powerful, but unfortunately quite a heavy CMS which is best suited for Intranets. Thus, we are always looking for speed increase.

Enter Zeo cluster – a feature that nowadays comes bundled with Zope and allows one database (practically Data.fs) to be used by multiple Zope instances, or more accurately Zeo clients. In standalone installation only one CPU / CPU core can be used for processing requests (as Zope / Python implementation is single-threaded AFAIK). So if there are any concurrent requests the database (ZODB, the Zope Object Database) usually has to wait for the request processing before it is asked for the data and only part of the processing power is used as requests are queued. Using Zeo server-client architecture however, each Zeo client can do the processing on their own CPU/core (thus efficiently using the whole CPU prosessing power available) and also minimize the hard disk idle time by asking for data in an ~asynchronous manner (in separate queues). Actually ZODB even serves the same object simultaneously to different client processes for performance reasons. This might raise database ConflictErrors, which are nothing to fear of, however, as noted some paragraphs below.

Similarly, you could also deploy Zeo clients on different computers in local network (or wherever you want), but that’s not the scope of this article. Having clients running on different machines is a similar case with the same performance basis, but there are connection lags, bandwith limits and such that decrease performance.

Theory vs. practice

Deploying a Zeo cluster instead of standalone Zope instance should theoretically increase the performance by factor of extra available CPUs / CPU cores. There might be some overheads from this setup though, so we tested it out using ApacheBenchmark – the benchmarking module that comes bundled with Apache nowadays. But first something about…

Setting up Zeo & converting from standalone mode

In the easiest scenario, setting Zeo up is rather easy: the unified installer supports Zeo-server setup out of the box (=there is a recipe for it). Just run the unified installer like:

$ ./install.sh zeo

Luckily, the unified installer uses buildout from Plone 3.1 onwards. Thus, converting your current buildout instances to Zeo cluster is nothing but change of buildout configuration. Where you would normally need ‘instance’ section in your buildout.cfg you will now need the following:

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:12000
#effective-user = __EFFECTIVE_USER__

[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
# The line below sets only the initial password. It will not change an
# existing password.
user = admin:mysecretpassword
http-address = 12001
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}
    ${plone:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products
    ${productdistros:location}
    ${plone:products}

To add more clients (which is quite the point here), append as many times the extra client sections like this:

[client2]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
user = ${client1:user}
http-address = 12002
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on
eggs = ${client1:eggs}
zcml = ${client1:zcml}
products = ${client1:products}

That minimizes the need for retyping user names, password etc. These examples were taken from Plone unified installer buildout.cfg with ports changed.

Starting, stopping & restarting

Now, to start your Zeo-powered Plon clients you could type:

bin/zeoserver start
bin/client1 start
bin/client2 start
...same for all the clients...

However, the unified installer has a recipe which automatically generates nice and simple shell scripts to control your cluster. In the end of your buildout.cfg, add:

[unifiedinstaller]
recipe = plone.recipe.unifiedinstaller
user = ${client1:user}
primary-port = ${client1:http-address}

That should generate the scripts. In fact, it propably does also something else, something which I’m not aware of. However, I didn’t bump into any problems, yet Anyway, to start the whole cluster (server & clients), type:

bin/startcluster.sh

And that does it (it start server and the clients). Shut it down via:

bin/shutdowncluster.sh

And restart:

bin/restartcluster.sh

ConflictErrors – not that errerous

As noted before, in Zeo mode the ZODB might serve the same objects to two more clients at the same time. If one client manipulates the object before others (ie. edits values and saves changes) the other requests will propably fail. This raises ConflicError which looks like this:

ConflictError: database conflict error (oid 0x0f39, class HelpSys.HelpSys.ProductHelp)

In this case ZODB tries to reprocess the failed requests. This should be common database approach and thus a feature, not a bug (although Zope might want to tell that in error message!). For more accurate explanation see Plone discussion.

Parsing it together with web server

The Zeo components (server and clients) talk to each other via standard Internet protocols (TCP or UDP, not sure). In the default setup, the Zeo server listens to port 8100 and Zeo clients to 8080, 8081, etc. Thus, to access the separate clients as ‘one site’ we need to serve the requests to multiple clients. This can be achieved with load balancers. Apache has at least one: mod_proxy_balancer which should do exactly what we need. Apache isn’t the best choice for achieving high requests per second values, but it will do for our tests (compare to more lightweight but also more limited lighttpd). Just remember that there are other alternatives/methods available, like using squid as load balancer.

Our configuration is as follows (inside VirtualHost-directive):

  <Proxy balancer://lb>
    BalancerMember http://127.0.0.1:12001/
    BalancerMember http://127.0.0.1:12002/
    BalancerMember http://127.0.0.1:12003/
    BalancerMember http://127.0.0.1:12004/
  </Proxy>

  <Location /balancer-manager>
    SetHandler balancer-manager
    Order Deny,Allow
    Allow from all
  </Location>

  ProxyPass /balancer-manager !
  ProxyPass             / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/
  ProxyPassReverse      / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/

This setup also allows us to use the balancer-manager (accessible at /balancer-manager) that comes with mod_proxy_balancer. It’s useful for checking if the configuration is working and balancer is dividing the requests equally. In my setup the balancer is using the default Request Counting -algorithm which divides the requests numerically equally between the instances, but you might want to also try Weighted Traffic Counting, which should be for actual use. In our test only the frontpage is accessed however, so each request’s data transfer is equal and the weighted traffic counting isn’t of use.

The test

The server machine

• Ubuntu 8.04 virtual server
• Intel Xeon 2.0Ghz (4 cores)
• 2 GB of RAM
• Hard disk drive (7200rpm?)

The setup

• Standalone Plone instance
• Plone via Zeo server with 4 clients (as many clients as cores in processor)
• Plone via Zeo server with 6 clients (for curiosity)

The tests where run locally in development environment to minimize the network lag (was 0-1ms).

The test commands

ApacheBenchmark commands:

$ ab -n N -c C myurl

where N was either 1000 or 9000 (requests) and C 1, 10, 100 or 1000 (concurrent requests).

The results

You can download the more in-depth test sheet Plone Standalone vs. Zeo installation (PDF).

To put it simple: theory and practise meet well – Zeo server is a lot more powerful with concurrent requests. On non-concurrent requests the results are about the same.

Having as many Zeo clients as CPUs / CPU cores can boost the performance up to number of extra CPUs/cores. For example, in our quad-core server with Zeo setup we gained nearly 4 times the requests per second of standalone installation (~370% to be accurate). Increasing Zeo clients to 6 didn’t help any as there’s no processing power left from 4 heavily stressed client processes. Also to be noted is that the waiting times for clients nearly tripled (median jumped from 126 to 305 ms) when raising concurrency from 1 to 10. This isn’t bad though – those are still low figures compared to standalone’s median of 1215 ms! Only when raising concurrency to 100 we began to see some 3,6 seconds waiting times (6 seconds for standalone). Increasing concurrency didn’t bring down the requests/second rates much (less than 5%) as expected.

Overall, the results were expected, but now we have evidence of it: under concurrent request load Zeo server is a good option to multiply the performance of your site. With very low traffic sites which rarely get more than 1 request at time this doesn’t matter.

One bad word about the resource requirements though: The used RAM increase for 6 client Zeo setup (standard Plone 3.1.2 + 12 additional Products) was whopping 621 MB (1132 MB -> 1753 MB). That means about 100 MB per Zeo client as the Zeo server memory intake was only about 12-15 MB. Thus, only use as many Zeo clients as absolutely necessary or you might find your beloved server machine under very serious Zope flu!

Plone, KSS, Javascript, field validation and the cup of WTF

Knowledge does hurt. Today I had my sweet cup of WTF. We are developing a medical database based on Plone 3.1. It uses very advanced AJAX framework called KSS – basically you can avoid the pain of writing pure Javascript by crafting special CSS like stylesheets which bind server-side Python code to any Javascript event. This makes AJAX programming a joy. You can easily combine server-side logic with user interface events, like field validation.

Well… then there was an error. KSS validation was not working for the text fields on a certain pages…. or it did sometimes. We were not sure. This is so called Heisenbug. I armed myself with sleepy eyes, Firebug and a lot of energy drinks.

I saw a KSS error in the Firebug log window and failed HTTP POST in the server logs.

Invalid request.

The parameter, value, was omitted from the request.

Looks like the field value was not properly posted for the field validation.

The first thing was locate the error and get function traceback for the messy situation. Unfortunately Firefox Javascript engine or Firebug cannot show tracebacks properly… the grass is so much greener on the other side of the fence. So I had to manually search through the codebase by manually plotting console.log() calls here and there.

Finally I thought I pinpointed the cause of the failure. By shaking finger (excitement, tireness and all that extra caffeinen from energy drinks), I opened the Javascript file just to realize why Javascript is utterly utterly shitty and why no sane person wants to do low level Javascript development. If ECMA standard committee had been clever and had been able to enforce anything long time ago, the following piece could be replaced with one function call.

fo.getValueOfFormElement = function(element) {
    // Returns the value of the form element / or null
    // First: update the field in case an editor is lurking
    // in the background
    this.fieldUpdateRegistry.doUpdate(element);
    if (element.disabled) {
        return null;
    }
    // Collect the data
    if (element.selectedIndex != undefined) {
        // handle single selects first
        if(!element.multiple) {
                if (element.selectedIndex < 0) {
                    value="";
                } else {
                    var option = element.options[element.selectedIndex];
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value = option.value || option.text;
                }
        // Now process selects with the multiple option set
        } else {
            var value = [];
            for(i=0; i<element.options.length; i++) {
                var option = element.options[i];
                if(option.selected) {
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value.push(option.value || option.text);
                }
            }
        }
    // Safari 3.0.3 no longer has "item", instead it works
    // with direct array access []. Although other browsers
    // seem to support this as well, we provide checking
    // in both ways. (No idea if item is still needed.)
    } else if (typeof element.length != 'undefined' &&
        ((typeof element[0] != 'undefined' &&
        element[0].type == "radio") ||
        (typeof element.item(0) != 'undefined' &&
        element.item(0).type == "radio"))) {
        // element really contains a list of input nodes,
        // in this case.
        var radioList = element;
        value = null;
        for (var i=0; i < radioList.length; i++) {
            var radio = radioList[i] || radioList.item(i);
            if (radio.checked) {
                value = radio.value;
            }
        }
    } else if (element.type == "radio" || element.type == "checkbox") {
        if (element.checked) {
           value = element.value;
        } else {
            value = null;
        }
    } else if ((element.tagName.toLowerCase() == 'textarea')
               || (element.tagName.toLowerCase() == 'input' &&
                    element.type != 'submit' && element.type != 'reset')
              ) {
        value = element.value;
    } else {
        value = null;
    }
    return value;
};

It turned out that the element in this case was an empty list of radio buttons. When you are tab keying through a radio button group without any value selected, like in the case a content object is just created, KSS validation is triggered even though there is no value in any of the radio buttons. This makes KSS think the value is null and it does not properly handle the situation. This does not cause any user visible effects unless you have Javascript debugging on (Firebug + debugging mode in Plone’s Javascript registry).

But this was not the bug I was looking for. It was just masking the original bug, because I had an empty radio button group next to the text field whose validation was not correctly done. More server side debugging…

I inserted some funky debug prints to Archetypes.Field.validate_validators():

validate_validators()

Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))

We can see that not triggered validator, validDecRange03, is still with us. Then I add more debug prints to see where things go wrong, this time to to Products.validation.chain.__call__.

Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))
Name:isEmptyNoError value:234234234234234234234234234232342342342344534232342344234534554 result:True

Ok – we have a case here. isEmptyNoError validator is executed before our custom validator. Since this validator is flagged as “sufficient” other validators are not evaluated. I think this has not been the case before and our validator have worked properly… maybe there was API change in Plone 3.1 which broke the things?

After digging and digging and digging I found this 4 years old bug. Let’s open the famous isEmptyNoError source code in Products.validation.validators.EmptyValidator:

class EmptyValidator:
    __implements__ = IValidator

    def __init__(self, name, title='', description='', showError=True):
        self.name = name
        self.title = title or name
        self.description = description
        self.showError = showError

    def __call__(self, value, *args, **kwargs):
        isEmpty  = kwargs.get('isEmpty', False)
        instance = kwargs.get('instance', None)
        field    = kwargs.get('field', None)

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)
            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

        if isEmpty:
            return True
        elif value == '' or value is None:
            return True
        else:
            if getattr(self, 'showError', False):
                return ("Validation failed(%(name)s): '%(value)s' is not empty." %
                       { 'name' : self.name, 'value': value})
            else:
                return False

There is my WTF. Or XXX – thanks for the kisses. My guess is that because KSS validation is executed in special context, the magical REQUEST might not be there. The “is sufficient” validator fails because of the some sort of god forgotten magic and thus all custom validators fail in KSS when the field is not required.

The workaround: I add my own greetings to the code:

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)

            # XXX: Whatever this all does, it does not work for KSS validation
            # requests and we should ignore this
            if "fieldname" in request:
              return False

            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

If Zope 3 drives you smoking Plone 3 drives me drinking. No wonder newbies steer away from Plone – if you hadn’t been armed with 8 years of web development experience you would never have figured out what’s going on with such a simple thing as adding a custom validator. A comment added to the bug tracker.

Eclipse web developer plug-in memo

Currently I work in quite wide field of software development: Python (standalone, Plone, Zope, Django), PHP, Java, Symbian and embedded Linux. I am using Eclipse for development, since it’s pretty much the only consistent platform filling my needs. The nature of work also forces me to use different computers (Mac/Windows/Linux) with different clients. This drives me to reinstall Eclipse now and then.

Below are my personal notes what plug-ins are needed to get “perfect” Eclipse set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.

Eclipse setup

Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. You can use dummy text as the name of update site.

Eclipse WTP (Web Tools Platform)

Eclipse Web Tools Platform bundles Eclipse, Java development tools, HTML editor, CSS editor and some other generic useful stuff.

• No separate Eclipse download needed. Download the bundle from http://download.eclipse.org/webtools/downloads/

Python

PyDev is a plug-in for Python and Jython development.

Site URL: http://pydev.sourceforge.net

Eclipse update site URL: http://pydev.sourceforge.net/updates/

PDT

PDT download provides Eclipse, HTML editor, PHP editor and CSS editor.

Site URL: http://www.eclipse.org

Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/

Subclipse

Subclipse provides Subversion version control integration to Eclipse.

Eclipse update site URL: http://subclipse.tigris.org/update_1.2.x

In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules.

JSEclipse

JSEclipse provides a better editor (over WTP) for Javascript files, with impressive outlining and autofill capabilities.

Download requires Adobe developer account or similar fill-in-the-fields crap.

Site URL: http://labs.adobe.com/technologies/jseclipse/

ShellEd

Syntax coloring for Unix shell scripts

Project site: http://sourceforge.net/projects/shelled

SQL Explorer

SQL editor with limited GUI capabilities. Based on Eclipse platform. Comes standalone and as Eclipse plug-in.

• Download ZIP from http://sourceforge.net/project/showfiles.php?group_id=132863

needs MySQL JDBC driver

• http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.0.8.tar.gz/from/pick#mirrors

Technorati tags: Python Plone Django PHP Eclipse Web development Subclipse Javascript SQL