Running 32-bit chroot on 64-bit Ubuntu server to reduce Python memory usage

Here are documented brief instructions how to run 32-bit chroot’ed environment on 64-bit Ubuntu server. chroot means that you run re-rooted and jailed system inside another system.

What we do here is enabling 32-bit chroot’ed userland on 64-bit server. 32-bit userland and 32-bit Python environment reduces the memory usage of heavy website applications we are running (read: Plone/Zope), since 32-bit has only half of the pointer size and object-oriented programming is all about pointers. Zope is especially memory hungry, because it uses ZODB object database. The developer does not need to worry about when doing queries, updates or caches that much  as the persistent site state is transparent to Python (objects are automatically loaded from database or cache when they are referred). Easy persistency means that almost everything is in the database and you need to have big object cache. Plone has huge client-side, in-process, cache for persistent objects. The default setting is for the cache size 5000 objects. (sidenote: since ZODB cache is in-process and Python does not do threading too well, running big Plone sites means that you need run several processes to handle parallel requests – having multiple processes with big in-process caches means loads of memory consumption)

32-bit userland is especially useful if you need to run Plone on 64-bit VPS (virtual private server) with low amount of available memory (512 MB or 1 GB).

There are some brief measurements about 32-bit vs. 6-bit Python memory usage at the end of this article.

Unless otherwise specified, all command here should be run as the root user of the host system. Commands here are for example only and you need to know what you are doing. If you lack advanced UNIX administration skills we gladly arrange you some commercial training or hosting support.

Installing

Basic schroot installation instructions for Ubuntu can be found here https://help.ubuntu.com/community/DebootstrapChroot . We also install ZopeSkel in the chroo’ed environment for starting creating Plone sites. Note that we are using Ubuntu 8.04 which still ships with Python 2.4 – for later Ubuntus you need to compile Python 2.4 from the scratch.

apt-get install debootstrap
apt-get install schroot

# Old schroot uses global schroot.conf, new versions have
# chroot.d directory

# This is a heredoc, but use what ever editor you like
# to create the configuration
cat <<EOF > /etc/schroot/chroot.d/hardy_i386.conf
[hardy_i386]
description=Ubuntu 8.04 Hardy for i386
location=/srv/chroot/hardy_i386
personality=linux32
root-users=bob
run-setup-scripts=true
run-exec-scripts=true
type=directory
users=bob,john,alice,ploneuser
EOF

mkdir -p /srv/chroot/hardy_i386
debootstrap --variant=buildd --arch i386 hardy /srv/chroot/hardy_i386 \

http://archive.ubuntu.com/ubuntu/

# Check that the chroot is created and working
schroot -l

# Enter the chroot (logged in as bob)
schroot -c hardy_i386 -u root

# Once inside, install python2.4-dev and other needed tools
# Installing PIL with easy_install didn't work for some
# reason, so we use python-imaging package.
apt-get install python2.4-dev python-setuptools python-imaging
easy_install-2.4 ZopeSkel

And that’s about it.

Chroot’ed environment will have

• It’s own application binaries (and userland bitness)

Chroot’ed environment will share with the host system

• Ports
• Processes
• User accounts

Chroot’ed users

• Can’t list filesystem outside chroot

…giving additional safety for shared hosting in the case of chroot’ed environment is compromised.

Entering chroot’ed environment as a specific user

Try

schroot -c hardy_i386 -u root

…or…

schroot -c hardy_i386 -u plone_user # After you have set-uped normal user for chroot'ed environment

All background processes you leave running in chroot’ed environment are terminated when you exit this environment, unless you create sessions as described below.

Creating chroot sessions

Sessions enable running commands in chroot without the need to have it constantly open.

# Create a new schroot session
schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession \
--begin-session

# Run commands in the created session
schroot --chroot=plonesession --user=ploneuser --run-session \
/srv/plone/yourplonesite/bin/instance start

# Ending session
schroot --chroot=plonesession --user=ploneuser --end-session

Note that --chroot parameter takes in both actual chroot installations and session ids.

Doing resets for chrooted environment

The session processes exist as long as the session exist. Unless you explicitly start a new session with –begin-session the processes are terminated as soon as you log out from the chroot’ed environent.
chroot’ed environment is temporary unless you explicitly specify it not be
Thus if you want to run daemonized services in chrooted environment you need to take care of session handling manually.
Here is an example how do you construct a session (as a real root user) and then launch a shell script which will take care of launching applications inside the chroot.

# Terminate previous session if any
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --end-session

#Start the session (again)
schroot --chroot=hardy_i386 --user=plone_user --session-name=plone_session --begin-session

# Run a start script inside the chroot'ed environment which will start Plone
# NGINX and other necessary 32-bit services
schroot --chroot=plone_session --user=plone_user --run-session /srv/plone/myplonesite/restart-all.sh

Running sessions at startup

You can add schroot bootstrap in real /etc/rc.local:

schroot --chroot=hardy_i386 --user=ploneuser \
--session-name=plonesession --begin-session \
&& schroot --chroot=plonesession --run-session \
/srv/plone/inst/bin/instance start

Remember that the users have to be created outside the chrooted environment. If you set the home directory to something that exists only in the chrooted environment, use something like this

adduser --no-create-home --home HOMEDIR_IN_CHROOT ploneuser

Then to create the directory inside the chroot and set its ownership to the newly created user and group.

32-bit vs 64-bit memory consumption

Reason why we even tried this is that some python applications, like Zope, use references heavily and moving from 32bit to 64bit references increases memory usage. (J Stahl 2010)

Memory figures from a development Zope/Plone 3.3.5 server

This is far from a complete study, but it would seem that the chroot does pay off even though it has to load 32bit versions of basic libraries along. If running more than one instance on same server memory savings should increase.

More information

• http://manpages.ubuntu.com/manpages/hardy/man1/schroot.1.html
• http://en.wikipedia.org/wiki/Chroot

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Easily install all Python versions under Linux and OSX using collective.buildout.python

Here are short instructions how to install all versions (2.4, 2.5, 2.6, 2.7 and 3.1) of Python interpreters on UNIX system. The instructions were tested on Ubuntu 10.04 Lucid Lynx Linux but should work on other systems as is. The installation is based of downloading, compiling and installing different Pythons and their libraries using buildout tool. A buildout configuration for doing this is maintained by a Plone community.

This buildout is especially useful to get Python 2.4 properly running under the latest Ubuntu 10.04 Lucid Lynx. This is because Ubuntu repositories won’t ship with Python 2.4 packages anymore.

The installation will also include static compilation of some very popular libraries. These are dependencies for other Python packages including, but not limited, to

• libjpeg
• Python imaging library
• readline

Prerequisites

• Some Python version is installed (OS default)
• GCC compiler is installed (sudo apt-get install build-essential)
• Subversion tool is installed (sudo apt-get install subversion)

Running it

svn co http://svn.plone.org/svn/collective/buildout/python/
cd python
python bootstrap.py
bin/buildout

Using it

All Pythons are under virtualenv installations. This means that you can activate one Python configuration for your shell once easily (python command will run under different Python versions).

Activating Python 2.4

source python/python-2.4/bin/activate
(python-2.4)moo@murskaamo:~/code$ python -V
Python 2.4.6

Check that Python Imaging Library works

python
Python 2.4.6 (#1, Jul 16 2010, 10:31:46)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL

(No exceptions raised, Python Imaging Library works well).

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

How to install Joomla! on your Ubuntu/Linux server with basic security

The instructions may apply for other versions too.

Prerequisitements

What you need to have in order to use this how to

• Basic UNIX file permissions knowledge
• Basic UNIX shell knowledge
• You have a Linux server (Ubuntu / Debian) for which you have root user access and you plan to use this server to host one or several Joomla! sites
• Apache and MySQL instaleld on your server

User setup

Set-up an UNIX user on a dedicated server for Joomla! hosting. The user can SSH in the box and write to his home folder, /tmp and /var/www site folder.

We create a user called “user” in this instructions. Replace it with the username you desire. We also use the example site name (www).yoursite.com.

sudo adduser user # Asks for the password and created /home/user

sudo mkdir /var/www/user
sudo chmod -R user:user /var/www/user # Only user has writing access to this folder

Setup MySQL user account

Install MySQL as per Debian/Ubuntu instructions.

Login as MySQL admin user (may vary depending how your MySQL is configured). Note that first you will be asked for sudo password, then for MySQL administrative user password.

sudo mysql -u admin -p

CREATE DATABASE user DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON user.* TO 'user'@'localhost' identified by 'zxc123zxc'; 

Extract Joomla! installation files

Enter the folder which will contain web site PHP files.

sudo -i -u user # pose yourself as UNIX user who runs the site

cd /var/www/user

wget http://joomlacode.org/gf/download/frsrelease/12350/51111/Joomla_1.5.18-Stable-Full_Package.zip

unzip Joomla_1.5.18-Stable-Full_Package.zip

exit

Set file permission

Note that during Joomla’s browser based installation Apache’s www-data must have write access to folder in order to create configuration.php file. We will later remove this access right.

We will set Joomla! files under UNIX group group www-data so that Apache can read them. Certain files are set to be writable. This must be done as root user.

sudo chown -R user:www-data /var/www/user # Make user group to www-data

sudo chmod g+wrx /var/www/user # Read only access to www-data user. Write access for installation, will be later removed.

Now ls -l command in /var/www/user should give you something like this for fil masks:

drwxr-xr-x 11 user www-data    4096 2010-05-28 10:22 plugins
-rwxr--r--  1 user www-data     304 2010-05-28 10:21 robots.txt
drwxr-xr-x  6 user www-data    4096 2010-05-28 10:22 templates

Creating Apache configuration

sudo nano /etc/apache2/sites-enabled/yoursite.conf

   ServerName yoursite.com
   ServerAlias www.yoursite.com
   ServerAdmin info@yourcompany.com

   LogFormat       combined
   TransferLog     /var/log/apache2/yoursite.log

   # Make sure this virtual host if capable of executing PHP5
   Options +ExecCGI
   AddType application/x-httpd-php .php .php5

   # Point to www folder where Joomla! is extracted
   DocumentRoot /var/www/yoursite

   # Do not give illusion of safety
   # as PHP safe_mode really is a crap
   # and only causes problems
   php_admin_flag safe_mode off

   #
   # This entry will redirect traffic www.yoursite.com -> yoursite.com
   # Assume mod_rewrite is installed and enabled on Apache
   # 301 is HTTP Permanent Redirect code
   RewriteEngine On
   RewriteCond %{HTTP_HOST} ^www\.yoursite\.com [NC]
   RewriteRule (.*) http://yoursite.com$1 [L,R=301]

</VirtualHost>

Faking the DNS entry

sudo gedit /etc/hosts

# Force this hostname to go to your server public IP address from your local computer

123.123.123 yoursite.com www.yoursite.com

Start Joomla! browser based installation

http://yoursite.com

Secure the configuration

sudo chmod -R g-w /var/www/user # Remote write permission
sudo rm -rf /var/www/user/installation # Remove installation directory
# Add write permission to folders which contain writable files
sudo chmod -R g+x /var/www/user/logs
sudo chmod -R g+x /var/www/user/images
sudo chmod -R g+x /var/www/user/tmp
sudo chmod -R g+x /var/www/user/images

Setting up htaccess files

Joomla! comes with a sample htaccess file which has some security measurements by having RewriteRules to prevent malformed URL access.

To install this file do the following

sudo -i
cd /var/www/user
cp htaccess.txt .htaccess
chmod user:www-data .htaccess # Set file permission to be readable by Apache and writable by the UNIX user

Then we create a .htaccess file which we will place in all folders with Joomla! write access to prevent execution of PHP files in these folders. First we create htaccess.limited file which we use as a template.

sudo -i
cd /var/www/user
nano htaccess.limited # Open text editor

Use the following htaccess.limited content

# secure directory by disabling script execution
AddHandler cgi-script .php .pl .py .jsp .asp .htm .shtml .sh .cgi
Options -ExecCGI -Indexes

And put the master template htaccess.limited  to proper places

cp htaccess.limited media/.htaccess
chown -R user:www-data media/.htaccess 

cp htaccess.limited tmp/.htaccess
chown -R user:www-data tmp/.htaccess 

cp htaccess.limited logs/.htaccess
chown -R user:www-data logs/.htaccess

cp htaccess.limited images/.htaccess
chown -R user:www-data images/.htaccess

Start using the site

Setting outgoing email

Test outgoing email

Maintaining file permission

sudo usermod -a -G www-data user # Add user to www-data group so that it can set group permissions

sudo -i -u user # Login as your UNIX user
chgrp -R www-data templates libraries # Fix group ownership
chmod -R g+rx libraries templates # Set read access for the group

Read our blog  Subscribe mFabrik blog in a reader Follow us on Twitter Mikko Ohtamaa on LinkedIn

Setting up multi-touch scrolling for Ubuntu 9.10 Karmic Koala Linux on Asus Eee 1005HA netbook

This post is specific to Asus Eee 1005HA netbook, but the technique explained here can be used on any computer having Synaptics touchpad.

Multi-touch gestures allow you to perform user interface actions by doing two finger gestures on touchpad. Apple introduced this feature on Macbooks and after you get used to it, it greatly enhances your web browsing on mouseless netbook. The most important gesture is scroll text by swiping the touchpad with two fingers.

Apple has also many patents related to the gestures so they are not enabled by default.

The real multi-finger touch support needs multi-finger aware (capacitive) touchpad. Most PC laptops are not equipped with one. Luckily some of the simple gestures, like two finger scrolling, can be emulated on normal pressure point sensitive touchpad via clever calculations and other tricks.

Note: Ubuntu HAL support for Synaptics seem to be broken. Only shell script at the end of the post will work. HAL options in FDI file are being ignored.

Setting up Synaptics driver

Type in terminal

gksudo gedit /etc/hal/fdi/policy/11-x11-synaptics.fdi

Create and save file with this content:

<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
 <device>
   <match key="info.capabilities" contains="input.touchpad">
       <merge key="input.x11_driver" type="string">synaptics</merge>
       <merge key="input.x11_options.SHMConfig" type="string">On</merge>

       <merge key="input.x11_options.EmulateTwoFingerMinZ" type="string">40</merge>
       <merge key="input.x11_options.VertTwoFingerScroll" type="string">1</merge>
       <merge key="input.x11_options.HorizTwoFingerScroll" type="string">1</merge>
       <merge key="input.x11_options.TapButton1" type="string">1</merge>
       <merge key="input.x11_options.TapButton2" type="string">3</merge>  <!--two finger tap -> middle clieck(3) -->
       <merge key="input.x11_options.TapButton3" type="string">2</merge>  <!--three finger tap -> right click(2). almost impossible to click -->
   </match>
 </device>
</deviceinfo>

This allows us to use synclient utility to watch touchpad real-time data in console window.

Now restart X

sudo /etc/init.d/gdm restart

And open terminal again.

Type in command

synclient -m 100

And you should see data like this scrolling in the terminal:

129.355  2912 3469  59 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 129.455  2952 3529  59 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 time     x    y   z f  w  l r u d m     multi  gl gm gr gdx gdy
 129.555  3283 3516  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.656  3928 3517  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.756  4364 3637  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.856  4020 3329  49 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 129.956  3634 3122  58 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.057  3320 2957  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.157  2779 3312  61 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.257  2557 3739  61 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.358  2636 3485  39 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.458  2659 3104  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.558  2671 2988  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0

f column tells the number of fingers. w is the touched area width. z is the pressure.

If you put two fingers on touchpad and you see value f=2 then your hardware has multi-touch aware touchpad. Unfortunately Asus Eee 1005HA doesn’t seem to have one

Emulation approach

Synaptics driver can emulate two-finger touch with the following conditions

• Touched area width exceeds certain threshold (min width)
• Touch pressure exceeds certain thresholds

When the conditions are met the driver thinks “Wow looks this guy is pressing us really hard. maybe he is using two fingers?” Note that touchpad values are touchpad specific and values applying for one model don’t work on another computer.

Synaptics driver settings are described here. Synaptic driver settings can be modified run-time using xinput command. Run synclient -m 100 in one terminal window and change threshold values in other until you find correct emulation parameters for your laptop. Below is my xinput tests. Test scrolling on Firefox and any long web page.

moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 7
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 280
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 11
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 50
moo@huiskuttaja:~$

Looks like the following parameters are good for two finger emulation for Asus Eee 1005HA:

• Width: 8
• Pressure (Z): 10

You can also use command synclient -l to dump the current settings.

Below is the final script you need to run during log-in (see note about broken HAL at the beginning of the post):

#!/bin/sh

#!/bin/sh
#
# Use xinput --list-props "SynPS/2 Synaptics TouchPad" to extract data
#

# Set multi-touch emulation parameters
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 10
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 8
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Two-Finger Scrolling" 8 1
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Scrolling" 8 1 1

# Disable edge scrolling
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Edge Scrolling" 8 0 0 0 

# This will make cursor not to jump if you have two fingers on the touchpad and you list one
# (which you usually do after two-finger scrolling)
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Jumpy Cursor Threshold" 32 110

Jumpy cursor after two finger scroll

When you do a two-finger scroll and lift your one finger before the other the mouse cursor/scrolling may jump. Synaptics driver does not seem to have an option to filter out this bad event. If anyone knows solution for this please comment.

Other resources

• Linux Multi-Touch is a project to collect set of Perl scripts related to multi-touch events. If you want to customize your multi-touch experience and you have some development insight, this is the place to begin.
• Notes about multi-touch emulation (see the last comment).
• Diagnosing multitouch support and editing HAL files.
• Related idea on Ubuntu Brainstorm.
• Synaptics X11 driver source. Synaptics driver project page. Looks like there is no one central figure behind it, except for maintaining, and big bunch of patches from distributors.

Building a mobile site and applications with Django and Python

Recently we created a mobile site for an interactive bicycle tour. oulugo.mobi (you need to use mobile browser to access the site or you’ll get a redirect) is a multimedia enriched bicycle tour through the historic parts of the city of Oulu. All content is provided by OnGo.

The route, which you can bicycle through is drawn on Google Maps. There are nine  action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of  augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters.

Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts.

The tour is bilingual in Finnish and English.

There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service.

There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository.

The service is hosted on Python specific virtual server on Twinapex services server farm.

Features

• Premium content tailored for audio listening
• Dubbed in English and Finnish by a professional voice actor
• Bilingual: English/Finnish
• Adapts for smartphones (WebKit based browsers) and low end phones (XHTML mobile profile browsers)
• Streaming video and audio (RTSP / progressive HTTP download forv iPhone). Different audio quality is provided on depending on the handset features.
• Screen resolution detection based on user agent sniffing. Three different version of images are used.
• Custom Google Maps component for mobile is used. The component adapts for different mobile phones based on sniffing. Features include zoom, show action point, show the current location, search street address name. This component can be published on a request.
• Management interface features include video upload, video transcoding different mobile versions and editing bilingual content
• Apex Vertex handset database is used to detect the user’s mobile phone capabilities
• Apex Vertex logging and traffic analytics capabilities are used for the site statistics

Software stack

• Ubuntu 8.04 Hardy Heron virtual server
• Apache 2.2 / mod_python
• Python 2.5
• Django 1.0
• Django-CMS 1.0
• mobile.sniffer Python package to provide abstraction over different handset databases
• Apex Vertex streaming solution (RTSP based on Darwin streaming server by Apple)
• TinyMCE WYSIWYG editor
• Darwin streaming server
• ExtJS is extensively used in Apex Vertext management interface

Development effort

Development time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service.

The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter

Packing and copying Data.fs from production server for local development

These instructions help you to copy and transfer production server  ZODB database (Data.fs) to your local computer for development and testing. This allows you to do the testing against the copy of real data and the production server Plone instance set up.

See the original tip by cguardia.

Data.fs is ZODB file storage for transactional database. Journal history takes quite a lot of disk space there. Packing, i.e. removing the journal history,  usually reduces the size file considerably, making the file lighter for wire transfer. Depending on the database age the packed copy is less than 10% of the original size.

These instructions apply for Ubuntu/Debian based Linux systems. Apply to your own system using the operating system best practices.

We need ZODB Python package to work with the database. To use it, we’ll create virtualenv Python installation in /tmp. In virtualenv installation, installed Python packages do not pollute or break the system wide setup. Note that you might use easy-install-2.4 depending on the OS. The latest stable ZODB can be picked from PyPi listing. Plone 3.x default is ZODB 3.7.x, which is not available as Python egg, but you can use ZODB 3.8.x.

sudo easy-install virtualenv
cd /tmp
virtualenv packer
/tmp/packer/bin/easy_install ZODB=3.8.3

Data.fs cannot be modified in-place. You must create a copy of it to work with it. Data.fs copy can be created from a running system without the fear of corrupting the database, since ZODB is append only database.

cp /yoursite/var/filestorage/Data.fs /tmp/Data.fs.copy

Then create the following script snippet /tmp/pack.py using your favorite terminal editor.

import time
import ZODB.FileStorage
import ZODB.serialize

storage=ZODB.FileStorage.FileStorage('/tmp/Data.fs.copy')
storage.pack(time.time(),ZODB.serialize.referencesf)

And run it using virtualenv’ed Python setup with ZODB installed:

/tmp/packer/bin/python /tmp/pack.py

Lots of patience here… packing may take a while, but it’s still definitely faster than your Internet connection transfer rate.

Verify that the file is succesfully packed:

ls -lh Data.fs.copy
-rw-r--r-- 1 user user 30M 2009-09-01 13:24 Data.fs.copy

Woohoo 1 GB was shrunk to 30 MB. Then copy the file to your local computer using scp and place it to your development buildout.

scp user@server:/tmp/Data.fs.copy ~/mybuildout/var/filestorage/Data.fs

You just saved about 30-90 minutes of waiting of file transfer.

Autoconfiguring dual monitors on Ubuntu

The following shell script is a helper script for laptop users who connect an external monitor now and then to their laptop. Since Ubuntu does not provide clever ways to arrange desktop or detect connected displays, you need to run the script from terminal when you change your monitor configuration.

• detect the numbers of monitors you have attached
• Activate the second (external) monitor if there are two monitors
• Use the external monitory as the primary display, otherwise use internal display as a primary display
• Move your taskbars to the primary display

It is based on disper tool by Willem van Engen. For now (2009), it’s nVidia only. ATI support could be possible.

#!/bin/sh
#
# Detect displays and move panels to the primary display
#
# Copyright 2009 Twinapex Research
#
# Author <mikko.ohtamaa@twinapex.com>
#

# disper command will detect and configure monitors
disper --displays=auto -e

# parse output from disper tool how many displays we have attached
# disper prints 2 lines per displer
lines=`disper -l|wc -l`

display_count=$((lines / 2))

echo $display_count

echo "Detected display count:" $display_count

# Make sure that we move panels to the correct display based
# on the display count
if [ $display_count = 1 ] ; then
   echo "Moving panels to the internal LCD display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "0"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "0"
else
   echo "Moving panels to the external display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "1"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "1"
fi

How to encode h264 video files for Nokia Series 60 standalone playback

Bored with Spiderman 3 which came with your Nokia N95 8 GB? This guide shortly tells how to get movies into your N95 on Ubuntu Linux using ffmpeg video encoder. The aim is to encode video suitable for playback from Nokia N-series (N95, N78, others) mobile phone memory card. We use h264 + AAC codecs which provides the best quality/compression rate for Nokia phones currently.

Ubuntu does not distribute proprietary codes. First thing you need to do is to rebuild ffmpeg.  Since Ubuntu 8.04 Hardy Heron ships with ffmpeg from 2007, which is aeons old in video codec years, you need to build libx264 and ffmpeg from SVN sources. Here are detailed, valid, instructions. Note that FFMPEG trunk is not currently stable (September 2008), so you need to use revision 15261 which needs this little patch. Indeed, this is a very difficult month to start your career in the dark world of video encoders.

To make it legal and support open source codec development,  please pay for your codecs.

Then we use this guide by Robert Swain. We have a tiny sub 2,4″ screen, we do not care about the quality and do one pass encoding. By empirical research, I have found that the following MPEG-4 profile parameters are compatible with N95 8 GB and provide the optimal result. You can vary video and audio bitrate depending on your taste.

Here is a script which recursivelu encodes all detected video files suitable for mobile format:

#!/bin/sh
#
# Optimal movie encoding for Nokia N-series mobile phones
#
# Copyright 2008 Red Innovation Ltd.
#
# Say hi if you find this useful.
# We do some professional mobile video publishing, so if you
# need a helping hand please call us.
#
# Usage: Run encode.sh in any folder and all video files are recursively converted to mobile phone suitable format
#
# Note: We expect all the source material be in 16:9 aspect ration
#
# Also see http://www.nseries.com/index.html#l=support,search,faq,general,video%20encoding,53848
#

VIDEO_BITRATE=300k

AUDIO_BITRATE=72k

# Assume locally build ffmpeg + x264 in /usr/local/bin
# http://ubuntuforums.org/showthread.php?t=786095
export LD_LIBRARY_PATH=/usr/local/lib

# Search all source AVI, MPG and WMV video files
# Place all encoded files to the same folder with the source, with added .mp4 extension
find . -iname "*.avi" -or -iname "*.wmv" -or -iname "*.mpg" | while read src ; do
        srcfile=`basename "$src"`
	srcfolder=`dirname "$src"`
	dstfile="$srcfolder"/"$srcfile".mp4

	# The magical string!
	# Size and cropping is for 16:9 source material, so that 320:240 display will have black bars.
	# Fex pixels off... note that h264 sizes must be multiplies of 16, use 256x144 for streaming
	# N95 RealMedia player does not seem to respect MPEG-4 embedded aspect ration info.
	/usr/local/bin/ffmpeg -y -i "$srcfile" -acodec libfaac -ab $AUDIO_BITRATE -s 320x176 -aspect 16:9 -vcodec libx264 -b $VIDEO_BITRATE -qcomp 0.6 -qmin 16 -qmax 51 -qdiff 4 -flags +loop -cmp +chroma -subq 7 -refs 6 -g 250 -keyint_min 25 -rc_eq 'blurCplx^(1-qComp)' -sc_threshold 40 -me_range 12 -i_qfactor 0.71 -directpred 3 "$dstfile"

done

Perfect dual boot crypted hard disk setup with Truecrypt and LUKS

I have a work laptop used in Symbian and web development. I need to be able to boot both Vista and Linux. Due to client privacy, both operating systems must be crypted for the case of lost laptop. Even if I do not use Windows actively, its web browser data may contain stored password for client systems and it would be catastrophic to leak them accidentally.

Here are instructions how to encrypt your hard disk in safely but performance effective manner with Ubuntu 8.04 Hardy Heron and Windows Vista. These instructions can be applied for any version of Vista, since we use third party open source Truecrypt suite to encrypt the Windows partition. The instructions also give priority for Grub boot loader, so that the computer will boot to Linux if there is no user interaction during the boot.

1. Install Windows Vista from the factory first boot installer
2. Download Ubuntu 8.04 alternative install CD. The alternative install CD contains installer menus to encrypt your HD using LVM and LUKS.
3. For the sake of performance, we only crypt /home directory on Linux partition which contains all user editable files. All other files in Linux, maybe excluding configuration files in /etc, are open source and encrypting them only slows your application start-up times. It is possible to encrypt /home after install, but it is much easier during the install time. Here are instructions how to set up encrypted home partition with alternative install CD.
4. After this comes the exciting part. You must encrypt the Windows system partition using Truecrypt. Since Truecrypt is going to overwrite Ubuntu’s Grub bootloader on Master Boot Record (MBR), some magic is needed (detailed instructions).
   1. Install Truecrypt and overwrite MBR.
   2. Boot Ubuntu from live CD. Alternative install CD doesn’t work as it does not have grub binary. You could also try to boot from your Linux partition by giving out manual kernel root file system parameters for the CD boot loader.
   3. Back-up Truecrypt’s MBR to a file on /boot partition using dd
   4. Add Truecrypt’s MBR as a chain boot loader in Grub
   5. Rewrite MBR using Grub

For foreigners: You might want to keep the US keymap in hand, since the installer environment has not necessarily keymap set up correctly.

Note: Since my HP Pavilion dv9000 laptop has two 250 GB hds, the actual setup is following: windows system partition, windows data partition, rest is set up for Linux using LVM in stripe RAID containing the root partition and the crypted home. This effectively gives near 100 MB/s read speed from two 5400 RPM hds.

Updated: Eclipse web developer plug-in memo

Below are my personal notes what plug-ins are needed to get “perfect” Eclipse web development set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.

This post is update to previous Eclipse web developer plug-in memo post. New versions are available and some plug-ins have become deprecated. This blog post reflects those changes.

These instructions are good for:

• Python developer
• PHP developer
• Java developer

Choosing Eclipse distribution

• On Window, use EasyEclipse
• On Linux, use Eclipse provided by the distribution – Eclipse links against the embedded Mozilla browser and this is distribution specific – EasyEclipse has some issues here. For Ubuntu users:

sudo apt-get install sun-java6 eclipse

• On OSX, use EasyEclipse

EasyEclipse bundles some of the stuff listed here with it – when using EasyEclipse you don’t need to have separate PyDev and Subclipse downloads.

Eclipse for 64-bit Linux has various problems. You might want to run 32-bit Eclipse (another relevant blog post). When you use Linux distribution specific Eclipse install, all your personal Eclipse files go to .eclipse folder under your home folder.

Installing plug-ins

Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location.

Python

PyDev is a plug-in for Python and Jython development. It has enhanced commercial extensions for professional developers with more intelligent autocomplete and debugger.

Site URL: http://pydev.sourceforge.net

PyDev Eclipse update URL: http://pydev.sourceforge.net/updates/

PyDev extensions Eclipse update URL (this commercial, but worth of every penny): http://www.fabioz.com/pydev/updates

PDT

PDT download provides Eclipse, HTML editor, PHP editor and CSS editor.

Site URL: http://www.eclipse.org

Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/

Java

If you need to do J2EE development use IBM’s Web Tools Platform. If you don’t need Java capabilities don’t install these, since they just bloat Eclipse and make the start up time worse.

Subclipse

Subclipse provides Subversion version control integration to Eclipse.

Eclipse update site URL: http://subclipse.tigris.org/update_1.4.x/

In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules.

Aptana Studio

Aptana Studio is state-of-the-art Web 2.0 development suite for Eclipse. It has Javascript, CSS and HTML editors. It supports various Javascript libraries out of the box and has support for Firefox and IE in-browser Javascript debugging.

Eclipse update site URL: http://update.aptana.com/update/studio/3.2/site.xml

ShellEd

Syntax coloring for Unix shell scripts

Project site: http://sourceforge.net/projects/shelled

SQL Explorer

SQL terminal and SQL editor with some GUI capabilities.

Eclipse update site URL http://eclipsesql.sourceforge.net/

SQL Explorer needs MySQL JDBC driver. Download from here. Install MySQL connector by extracting the file and adding it from SQL Explorer preferences.