What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much  as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption)

32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB).

There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article.

Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support.

Installing

Basic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch.

apt-get install debootstrap
apt-get install schroot

# Old schroot uses global schroot.conf, new versions have
# chroot.d directory

# This is a heredoc, but use what ever editor you like
# to create the configuration
cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf
[hardy_i386]
description=Ubuntu 8.04 Hardy for i386
location=/srv/chroot/hardy_i386
personality=linux32
root-users=bob
run-setup-scripts=true
run-exec-scripts=true
type=directory
users=bob,john,alice,ploneuser
EOF

mkdir -p /srv/chroot/hardy_i386
debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \

http://archive.ubuntu.com/ubuntu/

# Check that the chroot is created and working
schroot -l

# Enter the chroot (logged in as bob)
schroot -c hardy_i386 -u root

# Once inside, install python2.4-dev and other needed tools
# Installing PIL with easy_install didn't work for some
# reason, so we use python-imaging package.
apt-get install python2.4-dev python-setuptools python-imaging
easy_install-2.4 ZopeSkel

And that’s about it.

Chroot’ed environment will have

• It’s own application binaries (and userland bitness)

Chroot’ed environment will share with the host system

• Ports
• Processes
• User accounts

Chroot’ed users

• Can’t list filesystem outside chroot

…giving additional safety for shared hosting in the case of chroot’ed environment is compromised.

Entering chroot’ed environment as a specific user

Try

schroot -c hardy_i386 -u root

…or…

schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment

All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below.

Creating chroot sessions

Sessions enable running commands in chroot without the need to have it constantly open.

# Create a new schroot session
schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession \
--begin-session

# Run commands in the created session
schroot --chroot=plonesession --user=ploneuser --run-session \
/srv/plone/yourplonesite/bin/instance start

# Ending session
schroot --chroot=plonesession --user=ploneuser --end-session

Note that --chroot parameter takes in both actual chroot installations and session ids.

Doing resets for chrooted environment

The session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent.
chroot’ed environment is temporary unless you explicitly specify it not be
Thus if you want to run daemonized services in chrooted environment you need to take care of session handling manually.
Here is an example how do you construct a session (as a real root user) and then launch a shell script which will take care of launching applications inside the chroot.

# Terminate previous session if any
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session

#Start the session (again)
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session

# Run a start script inside the chroot'ed environment which will start Plone
# NGINX and other necessary 32-bit services
schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh

Running sessions at startup

You can add schroot bootstrap in real /etc/rc.local:

schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession --begin-session \
&& schroot --chroot=plonesession --run-session \
/srv/plone/inst/bin/instance start

Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this

adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser

Then to create the directory inside the chroot and set its ownership to the newly created user and group.

32-bit vs 64-bit memory consumption

Reason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010)

Memory figures from a development Zope/Plone 3.3.5 server

This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase.

More information

• http://manpages.ubuntu.com/manpages/hardy/man1/schroot.1.html
• http://en.wikipedia.org/wiki/Chroot

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

*.mo
*.egg-info
*.egg
*.EGG
*.EGG-INFO
bin
build
develop-eggs
downloads
eggs
fake-eggs
parts
dist
.installed.cfg
.mr.developer.cfg
.hg
.bzr
.svn
*.pyc
*.pyo
*.tmp*

Suggestions for new ignores are welcome.

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

For Easter holidays we bring you a trunk release instructions. Note: you might want to change checkout protocol from “https” to “http” in the buildout cfg. The most able and most inpatient might want to test this now when it is hot.

There exists three production site where you can see the product in action

• http://mfabrik.com (try with your web browser and phone and experience the joy of automatic mobile browser detection)
• http://saariselka.com and http://northvillage.mobi
• http://plonecommunity.mobi

See the feature set which will beat crap out of other mobile CMSs.

What’s missing for the real release is

• packaging it as a eggs
• making microsite
• clean up documentation (YES. It has documentation since day 0)

It has not been tested on Plone 4.

Thanks Karl Horak for bringing up the issue.

Please see PyPi page here.

It is not nearly as polished as Fabio’s work, as Fabio being main author of PyDev and started as a collection of company internal tools. However if you mare making many buildouts, write code > 5 Python modules at once and don’t want to start terminal everytime you need to run buildout, this plug-in is for you.

Also, it is not yet a real plug-in. It is a collection of PythonMonkey scripts. This means that to deploy the scripts you simply copy .py files them to scripts folder of any of your Eclipse projects. If you need to work on the codebase, you don’t need to start a separate Eclipse instance, but you can do it interactively through Eclipse console while you do other development. The script source code is well commented, so if you want to tune them for your own habits it should be easy.

• Kupu
• Outgoing email actions (Content rules ones)

The users can place ${title}, ${object_url} and other template in the edit mode. These variables which are directly mapped from Archetypes fields when the content is viewed/sent. Also, one can register custom snippet generators like $list_folder_content.

I hope Easy Template to cover some more actions in the future. I have noted PloneFormGen and Singing & Dancing product authors that we could add some mixed in functionality together.

Currently Easy Template uses Cheetah template backend. Cheetah is not Zope security friendly and exposing templated actions should be allowed only to trusted members. I am not huge fan of Plone’s TAL template language which is based on XML attributes and thus suitable only be used in XML context – this language is aimed only for hardcore hackers and software designers and ordinary folk really cannot wrap their minds around it.

Because I am not sure which will be the chosen template backend in the future I chose to abstract the template engine layer away. I created collective.templateengines product. It is a bunch of Zope interfaces and utility functions to abstract away common template actions like

• Applying a template
• Adding a template context variables
• Registering custom template tags

Currently collective.templateengines supports Cheetah and Django templates.

So, dear audience, what do you think of all this? What template engine would you suggest which would be Kupu friendly – you can edit the template language in WYSIWYG editor? Do you see any other usages for collective.templateengines? Which other projects could adopt template engine abstraction layer?

Buildout is from the creators of Zope (I suppose) so you can expect a powerful project code management tool. The question is, however, whether or not it suits your needs. In my case I found out it too heavy. I mean, to add even a simple task you have to create a new “recipe” (a package) that does the tricks. Of course some recipes are generic (found from PyPi) and you can just run them with your own INI options, but in my case I had to do some custom implementation. Creating a new python package isn’t that hard for sure but there’s of course some learning curve, so the real question is should you spend time to learn it or not?

I found out that zc.buildout has some nice features like:

• Automatic requirements processing through setuptools
• Automatic (yet simple) removal of directories during recipe uninstall
• Clear structure (install(), update() & uninstall() methods)
• INI-syntax, python does have a clear syntax but INI is always clearer for a newbie
• Easy script creation (adjust python paths somewhat automatically)
• Easily repeatable
• Passing of arguments from one recipe to another
• etc.

The problems?

• It takes a while to learn zc.buildout
• It takes ‘another while’ to learn to write recipes
• Too much hassle for little things
• INI-syntax is very limited in features
• Buildout easily updates all your packages (that means also the ones you didn’t want to!)
• Lack of documentation (it has good docs to get you going.. but after a while it leaves you with open questions)
• Unnecessary overhead (for each script you launch, you’ll need a launcher script created via buildout)

There’s no denying zc.buildout is powerful, but I wouldn’t use it for projects which need reasonable amount of customization. It’s just plain easier and quicker to write shell scripts and while those won’t provide you with any sort of ready tools you won’t propably need them. For bringing up somewhat static environment, where you don’t need to hack things (like that for Plone) it’s quite a decent option, however.

I also explored alternatives to zc.buildout. I’ve been reading about earlier virtualenv but haven’t really tried it out until now. It looks very promising and creates a more flexible environment compared to zc.buildout. Of course their goals are not exactly the same. Also, there are a few other alternatives out there, among them a new Python code management tool called Paver (just look at that cool logo.. it does remind you of Indiana Jones, does it not?). I glanced through the Paver docs and it looks like it might be the way to go (Paver also supports virtualenv), but didn’t quite get the grasp of the benefits just yet. Anyway, if you are still interested in code management and deployment, I’d recommend you to read the Paver release announcement and also Paver forewords. They should clear things up.

Enter Zeo cluster – a feature that nowadays comes bundled with Zope and allows one database (practically Data.fs) to be used by multiple Zope instances, or more accurately Zeo clients. In standalone installation only one CPU / CPU core can be used for processing requests (as Zope / Python implementation is single-threaded AFAIK). So if there are any concurrent requests the database (ZODB, the Zope Object Database) usually has to wait for the request processing before it is asked for the data and only part of the processing power is used as requests are queued. Using Zeo server-client architecture however, each Zeo client can do the processing on their own CPU/core (thus efficiently using the whole CPU prosessing power available) and also minimize the hard disk idle time by asking for data in an ~asynchronous manner (in separate queues). Actually ZODB even serves the same object simultaneously to different client processes for performance reasons. This might raise database ConflictErrors, which are nothing to fear of, however, as noted some paragraphs below.

Similarly, you could also deploy Zeo clients on different computers in local network (or wherever you want), but that’s not the scope of this article. Having clients running on different machines is a similar case with the same performance basis, but there are connection lags, bandwith limits and such that decrease performance.

Theory vs. practice

Deploying a Zeo cluster instead of standalone Zope instance should theoretically increase the performance by factor of extra available CPUs / CPU cores. There might be some overheads from this setup though, so we tested it out using ApacheBenchmark – the benchmarking module that comes bundled with Apache nowadays. But first something about…

Setting up Zeo & converting from standalone mode

In the easiest scenario, setting Zeo up is rather easy: the unified installer supports Zeo-server setup out of the box (=there is a recipe for it). Just run the unified installer like:

$ ./install.sh zeo

Luckily, the unified installer uses buildout from Plone 3.1 onwards. Thus, converting your current buildout instances to Zeo cluster is nothing but change of buildout configuration. Where you would normally need ‘instance’ section in your buildout.cfg you will now need the following:

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:12000
#effective-user = __EFFECTIVE_USER__

[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
# The line below sets only the initial password. It will not change an
# existing password.
user = admin:mysecretpassword
http-address = 12001
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}
    ${plone:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products
    ${productdistros:location}
    ${plone:products}

To add more clients (which is quite the point here), append as many times the extra client sections like this:

[client2]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
user = ${client1:user}
http-address = 12002
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on
eggs = ${client1:eggs}
zcml = ${client1:zcml}
products = ${client1:products}

That minimizes the need for retyping user names, password etc. These examples were taken from Plone unified installer buildout.cfg with ports changed.

Starting, stopping & restarting

Now, to start your Zeo-powered Plon clients you could type:

bin/zeoserver start
bin/client1 start
bin/client2 start
...same for all the clients...

However, the unified installer has a recipe which automatically generates nice and simple shell scripts to control your cluster. In the end of your buildout.cfg, add:

[unifiedinstaller]
recipe = plone.recipe.unifiedinstaller
user = ${client1:user}
primary-port = ${client1:http-address}

That should generate the scripts. In fact, it propably does also something else, something which I’m not aware of. However, I didn’t bump into any problems, yet Anyway, to start the whole cluster (server & clients), type:

bin/startcluster.sh

And that does it (it start server and the clients). Shut it down via:

bin/shutdowncluster.sh

And restart:

bin/restartcluster.sh

ConflictErrors – not that errerous

As noted before, in Zeo mode the ZODB might serve the same objects to two more clients at the same time. If one client manipulates the object before others (ie. edits values and saves changes) the other requests will propably fail. This raises ConflicError which looks like this:

ConflictError: database conflict error (oid 0x0f39, class HelpSys.HelpSys.ProductHelp)

In this case ZODB tries to reprocess the failed requests. This should be common database approach and thus a feature, not a bug (although Zope might want to tell that in error message!). For more accurate explanation see Plone discussion.

Parsing it together with web server

The Zeo components (server and clients) talk to each other via standard Internet protocols (TCP or UDP, not sure). In the default setup, the Zeo server listens to port 8100 and Zeo clients to 8080, 8081, etc. Thus, to access the separate clients as ‘one site’ we need to serve the requests to multiple clients. This can be achieved with load balancers. Apache has at least one: mod_proxy_balancer which should do exactly what we need. Apache isn’t the best choice for achieving high requests per second values, but it will do for our tests (compare to more lightweight but also more limited lighttpd). Just remember that there are other alternatives/methods available, like using squid as load balancer.

Our configuration is as follows (inside VirtualHost-directive):

  <Proxy balancer://lb>
    BalancerMember http://127.0.0.1:12001/
    BalancerMember http://127.0.0.1:12002/
    BalancerMember http://127.0.0.1:12003/
    BalancerMember http://127.0.0.1:12004/
  </Proxy>

  <Location /balancer-manager>
    SetHandler balancer-manager
    Order Deny,Allow
    Allow from all
  </Location>

  ProxyPass /balancer-manager !
  ProxyPass             / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/
  ProxyPassReverse      / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/

This setup also allows us to use the balancer-manager (accessible at /balancer-manager) that comes with mod_proxy_balancer. It’s useful for checking if the configuration is working and balancer is dividing the requests equally. In my setup the balancer is using the default Request Counting -algorithm which divides the requests numerically equally between the instances, but you might want to also try Weighted Traffic Counting, which should be for actual use. In our test only the frontpage is accessed however, so each request’s data transfer is equal and the weighted traffic counting isn’t of use.

The test

The server machine

• Ubuntu 8.04 virtual server
• Intel Xeon 2.0Ghz (4 cores)
• 2 GB of RAM
• Hard disk drive (7200rpm?)

The setup

• Standalone Plone instance
• Plone via Zeo server with 4 clients (as many clients as cores in processor)
• Plone via Zeo server with 6 clients (for curiosity)

The tests where run locally in development environment to minimize the network lag (was 0-1ms).

The test commands

ApacheBenchmark commands:

$ ab -n N -c C myurl

where N was either 1000 or 9000 (requests) and C 1, 10, 100 or 1000 (concurrent requests).

The results

You can download the more in-depth test sheet Plone Standalone vs. Zeo installation (PDF).

To put it simple: theory and practise meet well – Zeo server is a lot more powerful with concurrent requests. On non-concurrent requests the results are about the same.

Having as many Zeo clients as CPUs / CPU cores can boost the performance up to number of extra CPUs/cores. For example, in our quad-core server with Zeo setup we gained nearly 4 times the requests per second of standalone installation (~370% to be accurate). Increasing Zeo clients to 6 didn’t help any as there’s no processing power left from 4 heavily stressed client processes. Also to be noted is that the waiting times for clients nearly tripled (median jumped from 126 to 305 ms) when raising concurrency from 1 to 10. This isn’t bad though – those are still low figures compared to standalone’s median of 1215 ms! Only when raising concurrency to 100 we began to see some 3,6 seconds waiting times (6 seconds for standalone). Increasing concurrency didn’t bring down the requests/second rates much (less than 5%) as expected.

Overall, the results were expected, but now we have evidence of it: under concurrent request load Zeo server is a good option to multiply the performance of your site. With very low traffic sites which rarely get more than 1 request at time this doesn’t matter.

One bad word about the resource requirements though: The used RAM increase for 6 client Zeo setup (standard Plone 3.1.2 + 12 additional Products) was whopping 621 MB (1132 MB -> 1753 MB). That means about 100 MB per Zeo client as the Zeo server memory intake was only about 12-15 MB. Thus, only use as many Zeo clients as absolutely necessary or you might find your beloved server machine under very serious Zope flu!

Well then… I haven’t used Zope 3 interfaces standalone before, so the first thing what I do is writing “zope 3 interfaces” into my Google search this.

This page comes up.

It’s horrible – the very reason I write this quick blog entry. Some notes below (I have written things from the point of external visitor – I have hands deep in Zope myself, so you don’t need to clarify these things for me or teach anything)

• The information is tangled mess: please use subtitles and images
• You could mention that the package can be used outside Zope
• You could mention tat the package is a generic Python package
• The list of CamelCaseWordsWhichGoesOnAndOnUnexplained made me puke. Please use proper English and explain the meaning of the links.
• It has a comment box: I tried to comment the page but I will get permission error. Please do not show the comment box if the commenting is not actually possbile.
• Then I tried to register to Zope.org to comment or fix the basic things on the page. You have a join link in zope.org, but there is no registration form. Manual email conversation seem to be prerequisite for the registration. Is Zope a secret society or something…? The contribution barrier just rise too high.
• In this point, I give up. Even if you had the best interface package out there, I don’t care anymore. Looks like getting involved to fixing thing takes too much of my time. You had your 10 minutes of time to impress me and you failed. Not only that, but I got so frustrated that I want to learn smoking, fast.

In the post “No, you are not smart enough for Zope” Martjin Faassen highlights some problems of Zope community. “It’s hard to get good content written” Martjin claims. I disagree. Whoever created the page originally could have thought what people coming to the page want. They don’t want to decrypt the brain core dump of hardcore Zope developer. They want to know what is this thing, how this thing is beneficial for them, how do I get started with it and how do I use it.

You all know how Internet works. You all have visit on web pages. You all are customers for the same thing you also produce. So writing a basic web page is not something you couldn’t do.

Hints:

• apt-get install python-zopeinterface
• README.TXT

Pardon me the tone of this post. Zope is the 23th best thing out there, but the Zope community has stagnated badly in some aspects. Some things were acceptable ten years ago when web was still young and Python developers hardcore, but if you don’t keep with the pace you lose all the mindshare.