scp -C -o CompressionLevel=9 user@yoursite.com:~/plonefolder/zinstance/var/filestorage/Data.fs .

This command line enables real-time compression with the highest compression level. This speeds up copying somewhat, as ZODB database compresses pretty well.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Below is an example Python script snippet which reads this custom Apache log file and creates Excel export out of it. After this, you can easily open this file in OpenOffice Calc or Microsoft Excel and draw nice diagrames out of it to see where the performance bottlenecks are.

The script splits duration column for the output file. You can easily add your own splits or use Calc’s Text to Columns editor.

The script:

#!/bin/python

import shlex
from pyExcelerator.Workbook import Workbook

FILE = "/var/log/apache2/yourlogfile.log"

def get_parts(line):
	""" Extract the parts of a line"""

	parts = shlex.split(line) # Split reserving parts inside quotation marks
	# 'duration:0/42327', 'io:643/4507', 'balancer:http://127.0.0.1:13003/']

	return parts

def get_duration(duration):
	""" Extract duration in ms from log line """

	keyword, args = duration.split(":")
	seconds, microseconds = args.split("/")

	mseconds = int(microseconds) 

	return mseconds

wb = Workbook()
sheet = wb.add_sheet('0')

print "Reading log"
i = 0

for line in open(FILE, "rt"):
	parts = get_parts(line)
	j = 0

	for part in parts:
		sheet.write(i,j,part)
		j += 1
         	sheet.write(i,len(parts)+1,get_duration(parts[-3]))
	i += 1

wb.save('output.xls')

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much  as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption)

32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB).

There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article.

Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support.

Installing

Basic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch.

apt-get install debootstrap
apt-get install schroot

# Old schroot uses global schroot.conf, new versions have
# chroot.d directory

# This is a heredoc, but use what ever editor you like
# to create the configuration
cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf
[hardy_i386]
description=Ubuntu 8.04 Hardy for i386
location=/srv/chroot/hardy_i386
personality=linux32
root-users=bob
run-setup-scripts=true
run-exec-scripts=true
type=directory
users=bob,john,alice,ploneuser
EOF

mkdir -p /srv/chroot/hardy_i386
debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \

http://archive.ubuntu.com/ubuntu/

# Check that the chroot is created and working
schroot -l

# Enter the chroot (logged in as bob)
schroot -c hardy_i386 -u root

# Once inside, install python2.4-dev and other needed tools
# Installing PIL with easy_install didn't work for some
# reason, so we use python-imaging package.
apt-get install python2.4-dev python-setuptools python-imaging
easy_install-2.4 ZopeSkel

And that’s about it.

Chroot’ed environment will have

• It’s own application binaries (and userland bitness)

Chroot’ed environment will share with the host system

• Ports
• Processes
• User accounts

Chroot’ed users

• Can’t list filesystem outside chroot

…giving additional safety for shared hosting in the case of chroot’ed environment is compromised.

Entering chroot’ed environment as a specific user

Try

schroot -c hardy_i386 -u root

…or…

schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment

All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below.

Creating chroot sessions

Sessions enable running commands in chroot without the need to have it constantly open.

# Create a new schroot session
schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession \
--begin-session

# Run commands in the created session
schroot --chroot=plonesession --user=ploneuser --run-session \
/srv/plone/yourplonesite/bin/instance start

# Ending session
schroot --chroot=plonesession --user=ploneuser --end-session

Note that --chroot parameter takes in both actual chroot installations and session ids.

Doing resets for chrooted environment

The session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent.
chroot’ed environment is temporary unless you explicitly specify it not be
Thus if you want to run daemonized services in chrooted environment you need to take care of session handling manually.
Here is an example how do you construct a session (as a real root user) and then launch a shell script which will take care of launching applications inside the chroot.

# Terminate previous session if any
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session

#Start the session (again)
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session

# Run a start script inside the chroot'ed environment which will start Plone
# NGINX and other necessary 32-bit services
schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh

Running sessions at startup

You can add schroot bootstrap in real /etc/rc.local:

schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession --begin-session \
&& schroot --chroot=plonesession --run-session \
/srv/plone/inst/bin/instance start

Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this

adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser

Then to create the directory inside the chroot and set its ownership to the newly created user and group.

32-bit vs 64-bit memory consumption

Reason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010)

Memory figures from a development Zope/Plone 3.3.5 server

This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase.

More information

• http://manpages.ubuntu.com/manpages/hardy/man1/schroot.1.html
• http://en.wikipedia.org/wiki/Chroot

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

See the original tip by cguardia.

Data.fs is ZODB file storage for transactional database. Journal history takes quite a lot of disk space there. Packing, i.e. removing the journal history,  usually reduces the size file considerably, making the file lighter for wire transfer. Depending on the database age the packed copy is less than 10% of the original size.

These instructions apply for Ubuntu/Debian based Linux systems. Apply to your own system using the operating system best practices.

We need ZODB Python package to work with the database. To use it, we’ll create virtualenv Python installation in /tmp. In virtualenv installation, installed Python packages do not pollute or break the system wide setup. Note that you might use easy-install-2.4 depending on the OS. The latest stable ZODB can be picked from PyPi listing. Plone 3.x default is ZODB 3.7.x, which is not available as Python egg, but you can use ZODB 3.8.x.

sudo easy-install virtualenv
cd /tmp
virtualenv packer
/tmp/packer/bin/easy_install ZODB=3.8.3

Data.fs cannot be modified in-place. You must create a copy of it to work with it. Data.fs copy can be created from a running system without the fear of corrupting the database, since ZODB is append only database.

cp /yoursite/var/filestorage/Data.fs /tmp/Data.fs.copy

Then create the following script snippet /tmp/pack.py using your favorite terminal editor.

import time
import ZODB.FileStorage
import ZODB.serialize

storage=ZODB.FileStorage.FileStorage('/tmp/Data.fs.copy')
storage.pack(time.time(),ZODB.serialize.referencesf)

And run it using virtualenv’ed Python setup with ZODB installed:

/tmp/packer/bin/python /tmp/pack.py

Lots of patience here… packing may take a while, but it’s still definitely faster than your Internet connection transfer rate.

Verify that the file is succesfully packed:

ls -lh Data.fs.copy
-rw-r--r-- 1 user user 30M 2009-09-01 13:24 Data.fs.copy

Woohoo 1 GB was shrunk to 30 MB. Then copy the file to your local computer using scp and place it to your development buildout.

scp user@server:/tmp/Data.fs.copy ~/mybuildout/var/filestorage/Data.fs

You just saved about 30-90 minutes of waiting of file transfer.

For your Plone launcher, set environment variables (space separated list)

PTS_LANGUAGES=en mylanguagecodehere

If your Data.fs is not fresh (i.e. you have an existing Plone instance) there is still one task to do. Go to Placess Translation Service in Zope. Delete all translation catalogs. If there exists a translation catalog entry in ZODB a reload event seem to be triggered even though PTS_LANGUAGES settings is effective. Restart Zope. Maybe this is a bug? Do this on a development box only – this code seems to be quite new.
The magic code is in PlacelessTranslationService/load.py.