Use cases

• You are running a backend servers behind Apache
• You need to make sure that the users are served from the same backend every time: the backend itself caches or stores user specific data and this data is not shared across the backends. For example, Plone 3 may have issues with content item creation if sticky sessions are not used.

The method described here works in every situation and does not rely on client IP address, etc. The only downside is that if one balancer member goes down all subsequent requests for it will die. So this method can be only used for load balancing, not for high availability (I am not sure if BALANCER_ROUTE_CHANGED environment variable is set when a balancer member is lost and would redirect the clients to a new balancer member).

This requests were tested on Ubuntu Linux, but may as well work in other environments.

Setting route configution in virtual host

Create a balancer

<Proxy balancer://yourlb>
 BalancerMember http://127.0.0.1:13001/ route=1
 BalancerMember http://127.0.0.1:13002/ route=2
 BalancerMember http://127.0.0.1:13003/ route=3
 BalancerMember http://127.0.0.1:13004/ route=4
</Proxy>

Set the cookie using mod_headers. Note that the cookie must be in format [session name].[route id] (the dot is required). It seems to be possible to leave session name empty.

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

Make ProxyPass  follow the cookie (Zope virtual host monster style with HTTPS)

ProxyPass / balancer://lbsits/http://localhost/VirtualHostBase/https/yoursite.org:443/yourplonesiteid/VirtualHostRoot/ stickysession=ROUTEID

Note: Hard restart is required. apache2ctl graceful is not enough to make new balancer rules effective.

Testing

Use wget

wget -S https://yoursite.org/

See that Set-Cookie: ROUTE_ID is present and it contains a valid value (is not empty)

HTTP/1.1 200 OK
 Date: Wed, 13 Apr 2011 15:21:52 GMT
 Server: Zope/(Zope 2.10.9-final, python 2.4.5, linux2) ZServer/1.1 Plone/3.3.3
 Content-Length: 23197
 Expires: Sat, 01 Jan 2000 00:00:00 GMT
 Content-Type: text/html;charset=utf-8
 Content-Language: en
 Set-Cookie: I18N_LANGUAGE="en"; Path=/
 Set-Cookie: ROUTEID=.1; path=/

More info

• Sticky sessions with Nginx
• mod_proxy_balancer example
• Misc. Apache docs for Plone

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Prerequisites

What you need in order to benefit from these instructions

• Basic UNIX command-line knowledge
• SSH access to the server
• MySQL access to the database
• LAMP stack ready on the new server

These instructions have been tested on Ubuntu/Debian/Linux but they should work in other environments too.

Write down database access information

Get password from config.php file on the old server:

cd /var/www/phpBB3
cat config.php

Write down database name, username and password.

Copy files

Use rsync to remotely copy forum files to a new computer. On new computer, in /var/www folder

rsync -av --compress-level=9 user@oldserver.com:/var/www/phpBB3 .

Dump and copy database

Execute the following command on the new server. It takes SSH connection to the old server and dumps phpBB3 database to the new server over the SSH connection.

ssh user@oldserver.com -C -o CompressionLevel=9 mysqldump -u databaseuser --password=databasepassword --skip-lock-tables --add-drop-table databasename > phpbb3.sql

Create a new database

Use the old access information from config.php to create a database with identical access information on the new server. You need a MySQL root access to create new databases.

mysql -uroot -p

Create database and grant access to phpBB3 user for it.

mysql> create database databasename;

mysql> GRANT ALL ON databasename.* TO 'databaseuser'@'localhost' identified by 'databasepassword';

Load the database on the new server from the dump file:

mysql> connect databasename;

mysql> source phpbb3.sql

Configure Apache virtualhost for the new server

The last step is to set-up Apache virtual host on the new server, so you can access the phpBB3 using a domain name. Note that this doesn’t need to be a real domain name, but you can spoof the domain name using /etc/hosts file on your local workstation.

Add file /etc/apache2/sites-enabled/phpbb3.conf (or pick a filename based on forum name if you host multiple forums)

<VirtualHost *>
 ServerName yourdomainname.com

 DocumentRoot /var/www/phpBB3
 <Directory />
   Options FollowSymLinks
   AllowOverride None
 </Directory>

</VirtualHost>

Note that <virtualhost *> may change depending on how Apache has been set up to listen IP addresses and ports. Also if you are using a shared hosting package or VPS you might need to use the server control panel (cPanel) to do this step.

Then check if your new config file is ok and restart Apache:

apache2ctl configtest
apache2ctl graceful

Hosts spoofing trick

If you are not having a DNS server of your own which you can use for the copy you can always use /etc/hosts file trick to spoof domain names. This way you can make Apache to serve the forum from the server even if the forum is not connected to any real domain name yet.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Use your favorite editor to edit php.ini file in /Applications/XAMPP/etc/php.ini – sudo priviledges needed, Smultron does it out of the box.

Change lines:

log_errors = Off

;error_log = filename

To:

log_errors = on

error_log = /tmp/php.log

Restart Apache using XAMPP controller in Finder -> Applications.

Now use the following UNIX command to see continuous log flow in your terminal:

tail -f /tmp/php.log

See also the earlier article about XAMPP and file permissions.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

However if you wish to use Apache virtual hosts with Varnish there is a trick in it.

We use the following setup

• Varnish listens to port 80, HTTP
• Apache listens to port 81
• Varnish uses Apache as a backend

The related varnish.vcl is

backend backend_apache {
.host = "127.0.0.1";
.port = "81";
}

sub vcl_recv {
 ...
 elsif (req.http.host ~ "^blog.mfabrik.com(:[0-9]+)?$") {
    set req.backend = backend_apache;
 }
 ...
}

Note that the backend IP is 127.0.0.1 (localhost). By default, with Debian or Ubuntu Linux, Apache configuration does not do virtual hosting for this.

So if /etc/apache2/sites-enabled/blog.mfabrik.com looks like:

<VirtualHost *:81>

 ServerName blog.mfabrik.com
 ...
 LogFormat       combined
 TransferLog     /var/log/apache2/blog.mfabrik.com.log

 ...

 ExpiresActive On
 ExpiresByType image/gif A3600
 ExpiresByType image/png A3600
 ExpiresByType image/image/vnd.microsoft.icon A3600
 ExpiresByType image/jpeg A3600
 ExpiresByType text/css A3600
 ExpiresByType text/javascript A3600
 ExpiresByType application/x-javascript A3600
</VirtualHost>

And now the trick – you need to add the following to /etc/apache2/httpd.conf

NameVirtualHost *:81

Unless you do all this, Apache will just pick the first virtualhost file in /etc/apache2/sites-enabled and use it for all requests.

Also you need to edit ports.conf and change Apache to listen to port 81:

Listen 81

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Below is an example Python script snippet which reads this custom Apache log file and creates Excel export out of it. After this, you can easily open this file in OpenOffice Calc or Microsoft Excel and draw nice diagrames out of it to see where the performance bottlenecks are.

The script splits duration column for the output file. You can easily add your own splits or use Calc’s Text to Columns editor.

The script:

#!/bin/python

import shlex
from pyExcelerator.Workbook import Workbook

FILE = "/var/log/apache2/yourlogfile.log"

def get_parts(line):
	""" Extract the parts of a line"""

	parts = shlex.split(line) # Split reserving parts inside quotation marks
	# 'duration:0/42327', 'io:643/4507', 'balancer:http://127.0.0.1:13003/']

	return parts

def get_duration(duration):
	""" Extract duration in ms from log line """

	keyword, args = duration.split(":")
	seconds, microseconds = args.split("/")

	mseconds = int(microseconds) 

	return mseconds

wb = Workbook()
sheet = wb.add_sheet('0')

print "Reading log"
i = 0

for line in open(FILE, "rt"):
	parts = get_parts(line)
	j = 0

	for part in parts:
		sheet.write(i,j,part)
		j += 1
         	sheet.write(i,len(parts)+1,get_duration(parts[-3]))
	i += 1

wb.save('output.xls')

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

By default XAMPP users nobody user for running Apache. If you try to load any file created by your normal system user Apache will give access denied error. This means that either you must make all PHP files readable for everyone, or change the file owner. This is painful if you need to repeat the process for every file.

Instead, it is easier to make XAMPP run under the permissions of your local user.

Edit /Applications/XAMPP/etc/httpd.conf:

<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
User yourusername
Group staff
</IfModule>
</IfModule>

<IfModule !mpm_winnt_module><IfModule !mpm_netware_module>

DocumentRoot "/Users/yourusername/yoursiteroot"

You also need to change the location of setting default folder execution rights htdocs location to your development location

#
# This has been changed from /Applications/XAMPP/htdocs
#
<Directory "/Users/moo/code/joomla/mobilejoomla_dev">
    Options Indexes FollowSymLinks ExecCGI Includes
   AllowOverride All

</Directory>

And now Apache will happily eat all files you create or edit.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

# timestamp referer useragent, server time seconds/microseconds, transfered bytes input/output, which balancer was used
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" duration:%T/%D io:%I/%O balancer:%{BALANCER_WORKER_NAME}e"
TransferLog     /var/log/apache2/yoursite.log

More info

• http://www.ducea.com/2008/02/06/apache-logs-how-long-does-it-take-to-serve-a-request/

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The instructions may apply for other versions too.

Prerequisitements

What you need to have in order to use this how to

• Basic UNIX file permissions knowledge
• Basic UNIX shell knowledge
• You have a Linux server (Ubuntu / Debian) for which you have root user access and you plan to use this server to host one or several Joomla! sites
• Apache and MySQL instaleld on your server

User setup

Set-up an UNIX user on a dedicated server for Joomla! hosting. The user can SSH in the box and write to his home folder, /tmp and /var/www site folder.

We create a user called “user” in this instructions. Replace it with the username you desire. We also use the example site name (www).yoursite.com.

sudo adduser user # Asks for the password and created /home/user

sudo mkdir /var/www/user
sudo chmod -R user:user /var/www/user # Only user has writing access to this folder

Setup MySQL user account

Install MySQL as per Debian/Ubuntu instructions.

Login as MySQL admin user (may vary depending how your MySQL is configured). Note that first you will be asked for sudo password, then for MySQL administrative user password.

sudo mysql -u admin -p

CREATE DATABASE user DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON user.* TO 'user'@'localhost' identified by 'zxc123zxc'; 

Extract Joomla! installation files

Enter the folder which will contain web site PHP files.

sudo -i -u user # pose yourself as UNIX user who runs the site

cd /var/www/user

wget http://joomlacode.org/gf/download/frsrelease/12350/51111/Joomla_1.5.18-Stable-Full_Package.zip

unzip Joomla_1.5.18-Stable-Full_Package.zip

exit

Set file permission

Note that during Joomla’s browser based installation Apache’s www-data must have write access to folder in order to create configuration.php file. We will later remove this access right.

We will set Joomla! files under UNIX group group www-data so that Apache can read them. Certain files are set to be writable. This must be done as root user.

sudo chown -R user:www-data /var/www/user # Make user group to www-data

sudo chmod g+wrx /var/www/user # Read only access to www-data user. Write access for installation, will be later removed.

Now ls -l command in /var/www/user should give you something like this for fil masks:

drwxr-xr-x 11 user www-data    4096 2010-05-28 10:22 plugins
-rwxr--r--  1 user www-data     304 2010-05-28 10:21 robots.txt
drwxr-xr-x  6 user www-data    4096 2010-05-28 10:22 templates

Creating Apache configuration

sudo nano /etc/apache2/sites-enabled/yoursite.conf

   ServerName yoursite.com
   ServerAlias www.yoursite.com
   ServerAdmin info@yourcompany.com

   LogFormat       combined
   TransferLog     /var/log/apache2/yoursite.log

   # Make sure this virtual host if capable of executing PHP5
   Options +ExecCGI
   AddType application/x-httpd-php .php .php5

   # Point to www folder where Joomla! is extracted
   DocumentRoot /var/www/yoursite

   # Do not give illusion of safety
   # as PHP safe_mode really is a crap
   # and only causes problems
   php_admin_flag safe_mode off

   #
   # This entry will redirect traffic www.yoursite.com -> yoursite.com
   # Assume mod_rewrite is installed and enabled on Apache
   # 301 is HTTP Permanent Redirect code
   RewriteEngine On
   RewriteCond %{HTTP_HOST} ^www\.yoursite\.com [NC]
   RewriteRule (.*) http://yoursite.com$1 [L,R=301]

</VirtualHost>

Faking the DNS entry

sudo gedit /etc/hosts

# Force this hostname to go to your server public IP address from your local computer

123.123.123 yoursite.com www.yoursite.com

Start Joomla! browser based installation

http://yoursite.com

Secure the configuration

sudo chmod -R g-w /var/www/user # Remote write permission
sudo rm -rf /var/www/user/installation # Remove installation directory
# Add write permission to folders which contain writable files
sudo chmod -R g+x /var/www/user/logs
sudo chmod -R g+x /var/www/user/images
sudo chmod -R g+x /var/www/user/tmp
sudo chmod -R g+x /var/www/user/images

Setting up htaccess files

Joomla! comes with a sample htaccess file which has some security measurements by having RewriteRules to prevent malformed URL access.

To install this file do the following

sudo -i
cd /var/www/user
cp htaccess.txt .htaccess
chmod user:www-data .htaccess # Set file permission to be readable by Apache and writable by the UNIX user

Then we create a .htaccess file which we will place in all folders with Joomla! write access to prevent execution of PHP files in these folders. First we create htaccess.limited file which we use as a template.

sudo -i
cd /var/www/user
nano htaccess.limited # Open text editor

Use the following htaccess.limited content

# secure directory by disabling script execution
AddHandler cgi-script .php .pl .py .jsp .asp .htm .shtml .sh .cgi
Options -ExecCGI -Indexes

And put the master template htaccess.limited  to proper places

cp htaccess.limited media/.htaccess
chown -R user:www-data media/.htaccess 

cp htaccess.limited tmp/.htaccess
chown -R user:www-data tmp/.htaccess 

cp htaccess.limited logs/.htaccess
chown -R user:www-data logs/.htaccess

cp htaccess.limited images/.htaccess
chown -R user:www-data images/.htaccess

Start using the site

Setting outgoing email

Test outgoing email

Maintaining file permission

sudo usermod -a -G www-data user # Add user to www-data group so that it can set group permissions

sudo -i -u user # Login as your UNIX user
chgrp -R www-data templates libraries # Fix group ownership
chmod -R g+rx libraries templates # Set read access for the group

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The route, which you can bicycle through is drawn on Google Maps. There are nine  action points where the user can listen to streaming audio clips, with still images, in his/her mobile phone. This is sort of  augmented reality experience: The user sees the real world (where he/she is now bicycling) combined with the historic events (audio playback narrative). For example, at Linnansaari (a location on the route) you’ll see the actual 17th century castle ruins and the narrator tells how the castle exploded when fire, caused by a lighting, reached gunpowder warehouse… boom. The explosion caused stones fly over 400 meters.

Alternatively, the clips are available as podcasts from Oulu Tourism pages. You can download them into your iPod for offline listening and use in conjuction with a paper map. This demostrates interesting mix of multichannel publishing: paper, web, mobile and podcasts.

The tour is bilingual in Finnish and English.

There exists unreleased iPhone application, based on PhoneGap, which allows the user to track his/her location real-time on the web page. We didn’t see it worth of trouble to go through Apple iPhone application review process. When location based service support comes for the browser this feature is indended to be included as the standard HTML5 feature of the service.

There also exists Nokia Series 60 mobile application, based on PyS60 and Series 60 BrowserControl API, which allows the user to track his/her location in real-time. The application provides wrapper around Series 60 WebKit control and allows Javascript to access phone native functions (GPS) over localhost socket communication. Like with Apple, we didn’t see real-time tracking feature interesting enough to go through Symbian Signed process to get our application released. Also, BrowserControl had seriousquality problems and we didn’t consider it stable enough for the end users. Some work is available in PyS60 Community Edition repository.

The service is hosted on Python specific virtual server on Twinapex services server farm.

Features

• Premium content tailored for audio listening
• Dubbed in English and Finnish by a professional voice actor
• Bilingual: English/Finnish
• Adapts for smartphones (WebKit based browsers) and low end phones (XHTML mobile profile browsers)
• Streaming video and audio (RTSP / progressive HTTP download forv iPhone). Different audio quality is provided on depending on the handset features.
• Screen resolution detection based on user agent sniffing. Three different version of images are used.
• Custom Google Maps component for mobile is used. The component adapts for different mobile phones based on sniffing. Features include zoom, show action point, show the current location, search street address name. This component can be published on a request.
• Management interface features include video upload, video transcoding different mobile versions and editing bilingual content
• Apex Vertex handset database is used to detect the user’s mobile phone capabilities
• Apex Vertex logging and traffic analytics capabilities are used for the site statistics

Software stack

• Ubuntu 8.04 Hardy Heron virtual server
• Apache 2.2 / mod_python
• Python 2.5
• Django 1.0
• Django-CMS 1.0
• mobile.sniffer Python package to provide abstraction over different handset databases
• Apex Vertex streaming solution (RTSP based on Darwin streaming server by Apple)
• TinyMCE WYSIWYG editor
• Darwin streaming server
• ExtJS is extensively used in Apex Vertext management interface

Development effort

Development time: Around 100 hours. Three different developers where involved. Used development tools: Eclipse, PyDev, Subclipse, Subversion. There were around five meetings between the content provider and the technology provider. Few beta testing rounds using iPhone application were performed by bicycling in -10 celcius degrees weather (north and so on…). No polar bears were harmed during the creation of this mobile service.

The service is linked in from Oulu Tourism pages and thousands of paper brochures printed for Oulu summer season 2009.

About the author Mikko Ohtamaa

• LinkedIn
• Twitter