| Running 32-bit chroot on 64-bit Ubuntu server to reduce Python memory usagePosted on August 3, 2010 by Mikko OhtamaaFiled Under linux, plone, python, technology, ubuntu, zope Here are documented brief instructions how to run 32-bit chroot’ed environment on 64-bit Ubuntu server. chroot means that you run re-rooted and jailed system inside another system. What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption) 32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB). There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article. Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support. InstallingBasic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch. apt-get install debootstrap apt-get install schroot # Old schroot uses global schroot.conf, new versions have # chroot.d directory # This is a heredoc, but use what ever editor you like # to create the configuration cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf [hardy_i386] description=Ubuntu 8.04 Hardy for i386 location=/srv/chroot/hardy_i386 personality=linux32 root-users=bob run-setup-scripts=true run-exec-scripts=true type=directory users=bob,john,alice,ploneuser EOF mkdir -p /srv/chroot/hardy_i386 debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \ http://archive.ubuntu.com/ubuntu/ # Check that the chroot is created and working schroot -l # Enter the chroot (logged in as bob) schroot -c hardy_i386 -u root # Once inside, install python2.4-dev and other needed tools # Installing PIL with easy_install didn't work for some # reason, so we use python-imaging package. apt-get install python2.4-dev python-setuptools python-imaging easy_install-2.4 ZopeSkel And that’s about it. Chroot’ed environment will have
Chroot’ed environment will share with the host system
Chroot’ed users
…giving additional safety for shared hosting in the case of chroot’ed environment is compromised. Entering chroot’ed environment as a specific userTry schroot -c hardy_i386 -u root …or… schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below. Creating chroot sessionsSessions enable running commands in chroot without the need to have it constantly open. # Create a new schroot session schroot --chroot=hardy_i386 --user=ploneuser \ --session-name=plonesession \ --begin-session # Run commands in the created session schroot --chroot=plonesession --user=ploneuser --run-session \ /srv/plone/yourplonesite/bin/instance start # Ending session schroot --chroot=plonesession --user=ploneuser --end-session Note that --chroot parameter takes in both actual chroot installations and session ids. Doing resets for chrooted environmentThe session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent. # Terminate previous session if any schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session #Start the session (again) schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session # Run a start script inside the chroot'ed environment which will start Plone # NGINX and other necessary 32-bit services schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh Running sessions at startupYou can add schroot bootstrap in real /etc/rc.local: schroot --chroot=hardy_i386 --user=ploneuser \ --session-name=plonesession --begin-session \ && schroot --chroot=plonesession --run-session \ /srv/plone/inst/bin/instance start Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser Then to create the directory inside the chroot and set its ownership to the newly created user and group. 32-bit vs 64-bit memory consumptionReason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010) Memory figures from a development Zope/Plone 3.3.5 server
This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase. More information
Packing and copying Data.fs from production server for local developmentPosted on September 1, 2009 by Mikko OhtamaaFiled Under plone, technology These instructions help you to copy and transfer production server ZODB database (Data.fs) to your local computer for development and testing. This allows you to do the testing against the copy of real data and the production server Plone instance set up. See the original tip by cguardia. Data.fs is ZODB file storage for transactional database. Journal history takes quite a lot of disk space there. Packing, i.e. removing the journal history, usually reduces the size file considerably, making the file lighter for wire transfer. Depending on the database age the packed copy is less than 10% of the original size. These instructions apply for Ubuntu/Debian based Linux systems. Apply to your own system using the operating system best practices. We need ZODB Python package to work with the database. To use it, we’ll create virtualenv Python installation in /tmp. In virtualenv installation, installed Python packages do not pollute or break the system wide setup. Note that you might use easy-install-2.4 depending on the OS. The latest stable ZODB can be picked from PyPi listing. Plone 3.x default is ZODB 3.7.x, which is not available as Python egg, but you can use ZODB 3.8.x. sudo easy-install virtualenv Data.fs cannot be modified in-place. You must create a copy of it to work with it. Data.fs copy can be created from a running system without the fear of corrupting the database, since ZODB is append only database. cp /yoursite/var/filestorage/Data.fs /tmp/Data.fs.copy Then create the following script snippet /tmp/pack.py using your favorite terminal editor. import time
import ZODB.FileStorage
import ZODB.serialize
storage=ZODB.FileStorage.FileStorage('/tmp/Data.fs.copy')
storage.pack(time.time(),ZODB.serialize.referencesf)
And run it using virtualenv’ed Python setup with ZODB installed: /tmp/packer/bin/python /tmp/pack.py Lots of patience here… packing may take a while, but it’s still definitely faster than your Internet connection transfer rate. Verify that the file is succesfully packed: ls -lh Data.fs.copy -rw-r--r-- 1 user user 30M 2009-09-01 13:24 Data.fs.copy Woohoo 1 GB was shrunk to 30 MB. Then copy the file to your local computer using scp and place it to your development buildout. scp user@server:/tmp/Data.fs.copy ~/mybuildout/var/filestorage/Data.fs You just saved about 30-90 minutes of waiting of file transfer. Speeding up Plone loading with PTS_LANGUAGESPosted on August 19, 2008 by Mikko OhtamaaFiled Under Uncategorized If you are not a Finnish speaker (like 99,9% of you) you might not want to (re)load Finnish and other unwanted language catalogs during the Plone start up. This is possible for Plone 3.1, as Reinout van Rees explains (found out afterwards). For your Plone launcher, set environment variables (space separated list) PTS_LANGUAGES=en mylanguagecodehere If your Data.fs is not fresh (i.e. you have an existing Plone instance) there is still one task to do. Go to Placess Translation Service in Zope. Delete all translation catalogs. If there exists a translation catalog entry in ZODB a reload event seem to be triggered even though PTS_LANGUAGES settings is effective. Restart Zope. Maybe this is a bug? Do this on a development box only – this code seems to be quite new. |
Latest Top Shouts | Want to get mobile? |
