Profiling PostgreSQL database

This blog post will have some short notes about monitoring and profiling PostgreSQL databases.

pgtop

pgtop provides UNIX top command like user interface for PostgreSQL. pgtop command is available as Perl CPAN module.

How to install Perl CPAN modules as non-root user on Ubuntu (note: when it prompts to run sudo, answer no).

To install pgtop install following CPAN modules first: Term::ANSIColor, Term::ReadKey, DBD::Pg

perl -MCPAN -Mlocal::lib -e 'CPAN::install(DBD::Pg)'
perl -MCPAN -Mlocal::lib -e 'CPAN::install(Term::ReadKey)'
perl -MCPAN -Mlocal::lib -e 'CPAN::install(Term::ANSIColor)'

pgtop install instructions

pgtop manul

Running pgtop:

perl pgtop -d databasename -u yourdbuser -p yourdbuserpassword

pgfouine

pgfouine is a log analyzer for PostgreSQL.

Setting up multi-touch scrolling for Ubuntu 9.10 Karmic Koala Linux on Asus Eee 1005HA netbook

This post is specific to Asus Eee 1005HA netbook, but the technique explained here can be used on any computer having Synaptics touchpad.

Multi-touch gestures allow you to perform user interface actions by doing two finger gestures on touchpad. Apple introduced this feature on Macbooks and after you get used to it, it greatly enhances your web browsing on mouseless netbook. The most important gesture is scroll text by swiping the touchpad with two fingers.

Apple has also many patents related to the gestures so they are not enabled by default.

The real multi-finger touch support needs multi-finger aware (capacitive) touchpad. Most PC laptops are not equipped with one. Luckily some of the simple gestures, like two finger scrolling, can be emulated on normal pressure point sensitive touchpad via clever calculations and other tricks.

Note: Ubuntu HAL support for Synaptics seem to be broken. Only shell script at the end of the post will work. HAL options in FDI file are being ignored.

Setting up Synaptics driver

Type in terminal

gksudo gedit /etc/hal/fdi/policy/11-x11-synaptics.fdi

Create and save file with this content:

<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
 <device>
   <match key="info.capabilities" contains="input.touchpad">
       <merge key="input.x11_driver" type="string">synaptics</merge>
       <merge key="input.x11_options.SHMConfig" type="string">On</merge>

       <merge key="input.x11_options.EmulateTwoFingerMinZ" type="string">40</merge>
       <merge key="input.x11_options.VertTwoFingerScroll" type="string">1</merge>
       <merge key="input.x11_options.HorizTwoFingerScroll" type="string">1</merge>
       <merge key="input.x11_options.TapButton1" type="string">1</merge>
       <merge key="input.x11_options.TapButton2" type="string">3</merge>  <!--two finger tap -> middle clieck(3) -->
       <merge key="input.x11_options.TapButton3" type="string">2</merge>  <!--three finger tap -> right click(2). almost impossible to click -->
   </match>
 </device>
</deviceinfo>

This allows us to use synclient utility to watch touchpad real-time data in console window.

Now restart X

sudo /etc/init.d/gdm restart

And open terminal again.

Type in command

synclient -m 100

And you should see data like this scrolling in the terminal:

129.355  2912 3469  59 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 129.455  2952 3529  59 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 time     x    y   z f  w  l r u d m     multi  gl gm gr gdx gdy
 129.555  3283 3516  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.656  3928 3517  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.756  4364 3637  60 1  4  1 0 0 0 0  00000000   0  0  0   0   0
 129.856  4020 3329  49 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 129.956  3634 3122  58 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.057  3320 2957  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.157  2779 3312  61 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.257  2557 3739  61 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.358  2636 3485  39 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.458  2659 3104  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0
 130.558  2671 2988  60 1  4  0 0 0 0 0  00000000   0  0  0   0   0

f column tells the number of fingers. w is the touched area width. z is the pressure.

If you put two fingers on touchpad and you see value f=2 then your hardware has multi-touch aware touchpad. Unfortunately Asus Eee 1005HA doesn’t seem to have one

Emulation approach

Synaptics driver can emulate two-finger touch with the following conditions

• Touched area width exceeds certain threshold (min width)
• Touch pressure exceeds certain thresholds

When the conditions are met the driver thinks “Wow looks this guy is pressing us really hard. maybe he is using two fingers?” Note that touchpad values are touchpad specific and values applying for one model don’t work on another computer.

Synaptics driver settings are described here. Synaptic driver settings can be modified run-time using xinput command. Run synclient -m 100 in one terminal window and change threshold values in other until you find correct emulation parameters for your laptop. Below is my xinput tests. Test scrolling on Firefox and any long web page.

moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 7
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 280
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 11
moo@huiskuttaja:~$ xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 50
moo@huiskuttaja:~$

Looks like the following parameters are good for two finger emulation for Asus Eee 1005HA:

• Width: 8
• Pressure (Z): 10

You can also use command synclient -l to dump the current settings.

Below is the final script you need to run during log-in (see note about broken HAL at the beginning of the post):

#!/bin/sh

#!/bin/sh
#
# Use xinput --list-props "SynPS/2 Synaptics TouchPad" to extract data
#

# Set multi-touch emulation parameters
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Pressure" 32 10
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Width" 32 8
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Two-Finger Scrolling" 8 1
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Two-Finger Scrolling" 8 1 1

# Disable edge scrolling
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Edge Scrolling" 8 0 0 0 

# This will make cursor not to jump if you have two fingers on the touchpad and you list one
# (which you usually do after two-finger scrolling)
xinput set-int-prop "SynPS/2 Synaptics TouchPad" "Synaptics Jumpy Cursor Threshold" 32 110

Jumpy cursor after two finger scroll

When you do a two-finger scroll and lift your one finger before the other the mouse cursor/scrolling may jump. Synaptics driver does not seem to have an option to filter out this bad event. If anyone knows solution for this please comment.

Other resources

• Linux Multi-Touch is a project to collect set of Perl scripts related to multi-touch events. If you want to customize your multi-touch experience and you have some development insight, this is the place to begin.
• Notes about multi-touch emulation (see the last comment).
• Diagnosing multitouch support and editing HAL files.
• Related idea on Ubuntu Brainstorm.
• Synaptics X11 driver source. Synaptics driver project page. Looks like there is no one central figure behind it, except for maintaining, and big bunch of patches from distributors.

Autoconfiguring dual monitors on Ubuntu

The following shell script is a helper script for laptop users who connect an external monitor now and then to their laptop. Since Ubuntu does not provide clever ways to arrange desktop or detect connected displays, you need to run the script from terminal when you change your monitor configuration.

• detect the numbers of monitors you have attached
• Activate the second (external) monitor if there are two monitors
• Use the external monitory as the primary display, otherwise use internal display as a primary display
• Move your taskbars to the primary display

It is based on disper tool by Willem van Engen. For now (2009), it’s nVidia only. ATI support could be possible.

#!/bin/sh
#
# Detect displays and move panels to the primary display
#
# Copyright 2009 Twinapex Research
#
# Author <mikko.ohtamaa@twinapex.com>
#

# disper command will detect and configure monitors
disper --displays=auto -e

# parse output from disper tool how many displays we have attached
# disper prints 2 lines per displer
lines=`disper -l|wc -l`

display_count=$((lines / 2))

echo $display_count

echo "Detected display count:" $display_count

# Make sure that we move panels to the correct display based
# on the display count
if [ $display_count = 1 ] ; then
   echo "Moving panels to the internal LCD display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "0"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "0"
else
   echo "Moving panels to the external display"
   gconftool-2 \
        --set "/apps/panel/toplevels/bottom_panel_screen0/monitor" \
        --type integer "1"
   gconftool-2 \
        --set "/apps/panel/toplevels/top_panel_screen0/monitor" \
        --type integer "1"
fi

How to encode h264 video files for Nokia Series 60 standalone playback

Bored with Spiderman 3 which came with your Nokia N95 8 GB? This guide shortly tells how to get movies into your N95 on Ubuntu Linux using ffmpeg video encoder. The aim is to encode video suitable for playback from Nokia N-series (N95, N78, others) mobile phone memory card. We use h264 + AAC codecs which provides the best quality/compression rate for Nokia phones currently.

Ubuntu does not distribute proprietary codes. First thing you need to do is to rebuild ffmpeg.  Since Ubuntu 8.04 Hardy Heron ships with ffmpeg from 2007, which is aeons old in video codec years, you need to build libx264 and ffmpeg from SVN sources. Here are detailed, valid, instructions. Note that FFMPEG trunk is not currently stable (September 2008), so you need to use revision 15261 which needs this little patch. Indeed, this is a very difficult month to start your career in the dark world of video encoders.

To make it legal and support open source codec development,  please pay for your codecs.

Then we use this guide by Robert Swain. We have a tiny sub 2,4″ screen, we do not care about the quality and do one pass encoding. By empirical research, I have found that the following MPEG-4 profile parameters are compatible with N95 8 GB and provide the optimal result. You can vary video and audio bitrate depending on your taste.

Here is a script which recursivelu encodes all detected video files suitable for mobile format:

#!/bin/sh
#
# Optimal movie encoding for Nokia N-series mobile phones
#
# Copyright 2008 Red Innovation Ltd.
#
# Say hi if you find this useful.
# We do some professional mobile video publishing, so if you
# need a helping hand please call us.
#
# Usage: Run encode.sh in any folder and all video files are recursively converted to mobile phone suitable format
#
# Note: We expect all the source material be in 16:9 aspect ration
#
# Also see http://www.nseries.com/index.html#l=support,search,faq,general,video%20encoding,53848
#

VIDEO_BITRATE=300k

AUDIO_BITRATE=72k

# Assume locally build ffmpeg + x264 in /usr/local/bin
# http://ubuntuforums.org/showthread.php?t=786095
export LD_LIBRARY_PATH=/usr/local/lib

# Search all source AVI, MPG and WMV video files
# Place all encoded files to the same folder with the source, with added .mp4 extension
find . -iname "*.avi" -or -iname "*.wmv" -or -iname "*.mpg" | while read src ; do
        srcfile=`basename "$src"`
	srcfolder=`dirname "$src"`
	dstfile="$srcfolder"/"$srcfile".mp4

	# The magical string!
	# Size and cropping is for 16:9 source material, so that 320:240 display will have black bars.
	# Fex pixels off... note that h264 sizes must be multiplies of 16, use 256x144 for streaming
	# N95 RealMedia player does not seem to respect MPEG-4 embedded aspect ration info.
	/usr/local/bin/ffmpeg -y -i "$srcfile" -acodec libfaac -ab $AUDIO_BITRATE -s 320x176 -aspect 16:9 -vcodec libx264 -b $VIDEO_BITRATE -qcomp 0.6 -qmin 16 -qmax 51 -qdiff 4 -flags +loop -cmp +chroma -subq 7 -refs 6 -g 250 -keyint_min 25 -rc_eq 'blurCplx^(1-qComp)' -sc_threshold 40 -me_range 12 -i_qfactor 0.71 -directpred 3 "$dstfile"

done

Perfect dual boot crypted hard disk setup with Truecrypt and LUKS

I have a work laptop used in Symbian and web development. I need to be able to boot both Vista and Linux. Due to client privacy, both operating systems must be crypted for the case of lost laptop. Even if I do not use Windows actively, its web browser data may contain stored password for client systems and it would be catastrophic to leak them accidentally.

Here are instructions how to encrypt your hard disk in safely but performance effective manner with Ubuntu 8.04 Hardy Heron and Windows Vista. These instructions can be applied for any version of Vista, since we use third party open source Truecrypt suite to encrypt the Windows partition. The instructions also give priority for Grub boot loader, so that the computer will boot to Linux if there is no user interaction during the boot.

1. Install Windows Vista from the factory first boot installer
2. Download Ubuntu 8.04 alternative install CD. The alternative install CD contains installer menus to encrypt your HD using LVM and LUKS.
3. For the sake of performance, we only crypt /home directory on Linux partition which contains all user editable files. All other files in Linux, maybe excluding configuration files in /etc, are open source and encrypting them only slows your application start-up times. It is possible to encrypt /home after install, but it is much easier during the install time. Here are instructions how to set up encrypted home partition with alternative install CD.
4. After this comes the exciting part. You must encrypt the Windows system partition using Truecrypt. Since Truecrypt is going to overwrite Ubuntu’s Grub bootloader on Master Boot Record (MBR), some magic is needed (detailed instructions).
   1. Install Truecrypt and overwrite MBR.
   2. Boot Ubuntu from live CD. Alternative install CD doesn’t work as it does not have grub binary. You could also try to boot from your Linux partition by giving out manual kernel root file system parameters for the CD boot loader.
   3. Back-up Truecrypt’s MBR to a file on /boot partition using dd
   4. Add Truecrypt’s MBR as a chain boot loader in Grub
   5. Rewrite MBR using Grub

For foreigners: You might want to keep the US keymap in hand, since the installer environment has not necessarily keymap set up correctly.

Note: Since my HP Pavilion dv9000 laptop has two 250 GB hds, the actual setup is following: windows system partition, windows data partition, rest is set up for Linux using LVM in stripe RAID containing the root partition and the crypted home. This effectively gives near 100 MB/s read speed from two 5400 RPM hds.

Zope Zeo vs. standalone setups

We do some Plone development here at Redi. As known, Plone is a powerful, but unfortunately quite a heavy CMS which is best suited for Intranets. Thus, we are always looking for speed increase.

Enter Zeo cluster – a feature that nowadays comes bundled with Zope and allows one database (practically Data.fs) to be used by multiple Zope instances, or more accurately Zeo clients. In standalone installation only one CPU / CPU core can be used for processing requests (as Zope / Python implementation is single-threaded AFAIK). So if there are any concurrent requests the database (ZODB, the Zope Object Database) usually has to wait for the request processing before it is asked for the data and only part of the processing power is used as requests are queued. Using Zeo server-client architecture however, each Zeo client can do the processing on their own CPU/core (thus efficiently using the whole CPU prosessing power available) and also minimize the hard disk idle time by asking for data in an ~asynchronous manner (in separate queues). Actually ZODB even serves the same object simultaneously to different client processes for performance reasons. This might raise database ConflictErrors, which are nothing to fear of, however, as noted some paragraphs below.

Similarly, you could also deploy Zeo clients on different computers in local network (or wherever you want), but that’s not the scope of this article. Having clients running on different machines is a similar case with the same performance basis, but there are connection lags, bandwith limits and such that decrease performance.

Theory vs. practice

Deploying a Zeo cluster instead of standalone Zope instance should theoretically increase the performance by factor of extra available CPUs / CPU cores. There might be some overheads from this setup though, so we tested it out using ApacheBenchmark – the benchmarking module that comes bundled with Apache nowadays. But first something about…

Setting up Zeo & converting from standalone mode

In the easiest scenario, setting Zeo up is rather easy: the unified installer supports Zeo-server setup out of the box (=there is a recipe for it). Just run the unified installer like:

$ ./install.sh zeo

Luckily, the unified installer uses buildout from Plone 3.1 onwards. Thus, converting your current buildout instances to Zeo cluster is nothing but change of buildout configuration. Where you would normally need ‘instance’ section in your buildout.cfg you will now need the following:

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = 127.0.0.1:12000
#effective-user = __EFFECTIVE_USER__

[client1]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
# The line below sets only the initial password. It will not change an
# existing password.
user = admin:mysecretpassword
http-address = 12001
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on

# If you want Zope to know about any additional eggs, list them here.
# This should include any development eggs you listed in develop-eggs above,
# e.g. eggs = ${buildout:eggs} ${plone:eggs} my.package
eggs =
    ${buildout:eggs}
    ${plone:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml =

products =
    ${buildout:directory}/products
    ${productdistros:location}
    ${plone:products}

To add more clients (which is quite the point here), append as many times the extra client sections like this:

[client2]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
user = ${client1:user}
http-address = 12002
#effective-user = __EFFECTIVE_USER__
#debug-mode = on
#verbose-security = on
eggs = ${client1:eggs}
zcml = ${client1:zcml}
products = ${client1:products}

That minimizes the need for retyping user names, password etc. These examples were taken from Plone unified installer buildout.cfg with ports changed.

Starting, stopping & restarting

Now, to start your Zeo-powered Plon clients you could type:

bin/zeoserver start
bin/client1 start
bin/client2 start
...same for all the clients...

However, the unified installer has a recipe which automatically generates nice and simple shell scripts to control your cluster. In the end of your buildout.cfg, add:

[unifiedinstaller]
recipe = plone.recipe.unifiedinstaller
user = ${client1:user}
primary-port = ${client1:http-address}

That should generate the scripts. In fact, it propably does also something else, something which I’m not aware of. However, I didn’t bump into any problems, yet Anyway, to start the whole cluster (server & clients), type:

bin/startcluster.sh

And that does it (it start server and the clients). Shut it down via:

bin/shutdowncluster.sh

And restart:

bin/restartcluster.sh

ConflictErrors – not that errerous

As noted before, in Zeo mode the ZODB might serve the same objects to two more clients at the same time. If one client manipulates the object before others (ie. edits values and saves changes) the other requests will propably fail. This raises ConflicError which looks like this:

ConflictError: database conflict error (oid 0x0f39, class HelpSys.HelpSys.ProductHelp)

In this case ZODB tries to reprocess the failed requests. This should be common database approach and thus a feature, not a bug (although Zope might want to tell that in error message!). For more accurate explanation see Plone discussion.

Parsing it together with web server

The Zeo components (server and clients) talk to each other via standard Internet protocols (TCP or UDP, not sure). In the default setup, the Zeo server listens to port 8100 and Zeo clients to 8080, 8081, etc. Thus, to access the separate clients as ‘one site’ we need to serve the requests to multiple clients. This can be achieved with load balancers. Apache has at least one: mod_proxy_balancer which should do exactly what we need. Apache isn’t the best choice for achieving high requests per second values, but it will do for our tests (compare to more lightweight but also more limited lighttpd). Just remember that there are other alternatives/methods available, like using squid as load balancer.

Our configuration is as follows (inside VirtualHost-directive):

  <Proxy balancer://lb>
    BalancerMember http://127.0.0.1:12001/
    BalancerMember http://127.0.0.1:12002/
    BalancerMember http://127.0.0.1:12003/
    BalancerMember http://127.0.0.1:12004/
  </Proxy>

  <Location /balancer-manager>
    SetHandler balancer-manager
    Order Deny,Allow
    Allow from all
  </Location>

  ProxyPass /balancer-manager !
  ProxyPass             / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/
  ProxyPassReverse      / balancer://lb/http://localhost/VirtualHostBase/http/www.mydomain.com:80/plonesite/VirtualHostRoot/

This setup also allows us to use the balancer-manager (accessible at /balancer-manager) that comes with mod_proxy_balancer. It’s useful for checking if the configuration is working and balancer is dividing the requests equally. In my setup the balancer is using the default Request Counting -algorithm which divides the requests numerically equally between the instances, but you might want to also try Weighted Traffic Counting, which should be for actual use. In our test only the frontpage is accessed however, so each request’s data transfer is equal and the weighted traffic counting isn’t of use.

The test

The server machine

• Ubuntu 8.04 virtual server
• Intel Xeon 2.0Ghz (4 cores)
• 2 GB of RAM
• Hard disk drive (7200rpm?)

The setup

• Standalone Plone instance
• Plone via Zeo server with 4 clients (as many clients as cores in processor)
• Plone via Zeo server with 6 clients (for curiosity)

The tests where run locally in development environment to minimize the network lag (was 0-1ms).

The test commands

ApacheBenchmark commands:

$ ab -n N -c C myurl

where N was either 1000 or 9000 (requests) and C 1, 10, 100 or 1000 (concurrent requests).

The results

You can download the more in-depth test sheet Plone Standalone vs. Zeo installation (PDF).

To put it simple: theory and practise meet well – Zeo server is a lot more powerful with concurrent requests. On non-concurrent requests the results are about the same.

Having as many Zeo clients as CPUs / CPU cores can boost the performance up to number of extra CPUs/cores. For example, in our quad-core server with Zeo setup we gained nearly 4 times the requests per second of standalone installation (~370% to be accurate). Increasing Zeo clients to 6 didn’t help any as there’s no processing power left from 4 heavily stressed client processes. Also to be noted is that the waiting times for clients nearly tripled (median jumped from 126 to 305 ms) when raising concurrency from 1 to 10. This isn’t bad though – those are still low figures compared to standalone’s median of 1215 ms! Only when raising concurrency to 100 we began to see some 3,6 seconds waiting times (6 seconds for standalone). Increasing concurrency didn’t bring down the requests/second rates much (less than 5%) as expected.

Overall, the results were expected, but now we have evidence of it: under concurrent request load Zeo server is a good option to multiply the performance of your site. With very low traffic sites which rarely get more than 1 request at time this doesn’t matter.

One bad word about the resource requirements though: The used RAM increase for 6 client Zeo setup (standard Plone 3.1.2 + 12 additional Products) was whopping 621 MB (1132 MB -> 1753 MB). That means about 100 MB per Zeo client as the Zeo server memory intake was only about 12-15 MB. Thus, only use as many Zeo clients as absolutely necessary or you might find your beloved server machine under very serious Zope flu!

Tuning file system performance for Plone development

I recently read this article about tuning Ext3 file system for better performance. I was doing a fresh Ubuntu 7.10 install on my laptop, so I decided to see how much this would affect to my every day Plone development.

On Linux, every time a file is read, its access time attribute is rewritten. This causes a lot of unnecessary writes to file system. Since there are only few rare application needing this feature, turning of the feature can give a nice performance boost on systems dealing with large amount of files.

Plone 3.0 has 10000 files. A lot of them are read during the start-up. Maybe I am getting somewhere here…

When you are doing Plone development, you need to restart Plone often. I used this highly scientific method to measure Plone start-up time from issuing zopectl fg to getting the front page load completed in Firefox. I warmed the file system cache beforehand by doing two dry runs.

I also did some simple front page bombing with ab tool.

System setup

• HP nx9420 laptop (5400 RPM hard disk)
• Plone 3.0.2/Zope 2.10.4
• Intel Core 2 Duo, 2 Ghz
• Ubuntu 7.10
• Applied following Ext3 options: noatime, data=writeback

Out-of-the-box filesystem

Lap 1: 23s

Lap 2: 22s

Lap 3: 22s

ab stats:

Concurrency Level: 10
Time taken for tests: 11.805239 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 2058700 bytes
HTML transferred: 2030600 bytes
Requests per second: 8.47 [#/sec] (mean)

Tuned file system

Lap 1: 21s

Lap 2: 22s

Lap 3: Didn’t bother to do it…

ab stats:

Concurrency Level: 10
Time taken for tests: 12.102054 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 2058700 bytes
HTML transferred: 2030600 bytes
Requests per second: 8.26 [#/sec] (mean)

Conclusion

“Hooray.”

Though Plone/Zope crawls through of thousands of files during the start up (and thus touches their access times), the slow start-up process seem to be CPU bound. Magic file system tricks won’t make your everyday Plone development more effective.

Wrong swap UUID after hibernation in Feisty

I’ve been using uswsusp for suspending/hibernating my Ubuntu Feisty laptop but suddenly it failed to resume from disk hibernation (blank screen with blinking cursor). I booted up in restoration mode and Ubuntu reported that it couldn’t restore the snapshot. After pressing enter to continue, the system booted up just fine, skipping the snapshot restoration as supposed. The startup problem vanished, but it brought up a new one when trying to hibernate:

>>> sudo s2disk
Could not use the resume device (try swapon -a)

Of course, swapon also gives a problem:

>>> sudo swapon -a
swapon: cannot stat
/dev/disk/by-uuid/4a815ae8-fa5b-4265-826c-d777a723e87b:
No such file or directory

It seems that the UUID reference for swap is broken. Or is it the swap? At this point I did some Google research and it turned out the behaviour was because of an Ubuntu Feisty bug, which causes the swap UUID change occasionally. It is closely related to hibernation, yet the cause remains unclear. To fix it, let’s do:

>>> free -m | grep -i swap
Swap:            0          0          0

Which indicates that the system doesn’t find swap at all (because of wrong UUID). To find correct one:

>>> sudo fdisk -l | grep swap
/dev/sda6      10669   10917  2000061 82  Linux swap / Solaris

Find your swap there and go for:

>>> sudo vol_id /dev/sda6
ID_FS_UUID=083d41f0-de57-48d4-92eb-aefde8fd6ec9

Then you’ll just have to edit it in /etc/fstab and hibernation should work again. You could also try restoring the original snapshot by editing:

>>> sudo nano /etc/initramfs-tools/conf.d/resume

and correcting the reference there also. I didn’t test this myself, however.

That should do it, but don’t get too excited: some report that the UUID keeps changing even after the fix, and it has to be manually changed over and over again. Luckily, I haven’t experienced such behaviour (yet!) and it’s quick to fix (though you might lose ability to hibernate, which is a really bad thing).

Printing to Windows Vista printer from Linux

Today my head hurts. It appears that Linux clients cannot access Microsoft Windows Vista shares or printers, because Microsoft changed sharing protocol from SMB to SMB2 in Windows Vista. In theory, if the client doesn’t support SMB2, Vista server should fall back to old and faithful working SMB. In practice, Linux Samba clients have a bug (still in Ubuntu Feisty Fawn and Samba 3.0.24) and this doesn’t happen.

You get this error to CUPS logs:

E [20/Aug/2007:18:43:25 +0300] [Job 141]
No ticket cache found for userid=1000

E [20/Aug/2007:18:43:25 +0300] [Job 141]
Can not get the ticket cache for moo

E [20/Aug/2007:18:43:25 +0300] [Job 141]
Session setup failed: NT_STATUS_LOGON_FAILURE

E [20/Aug/2007:18:43:25 +0300] [Job 141]
Tree connect failed (NT_STATUS_ACCESS_DENIED)

E [20/Aug/2007:18:43:25 +0300] [Job 141]
Unable to connect to CIFS host, will retry in 60 seconds...

Even worse, CUPS and Gnome UI fail silently. The printer shows a status “still printing” and no error is reported to the user

You can still print to Windows Vista server if you install LPD printer sharing on Windows Vista. Both CUPS printer manager and Vista supports LPD protocol for printing. Follow these excellent instructions.

Now, back to the work after 3 hours of unneeded bang-my-head-to-wall.

EDIT: This seems to be fixed in Samba 3.0.25.