Preface

Please do not take this post too seriously if I am ranting tongue-in-cheek. This thinking does not reflect the official opinion of Sauna Sprint team any way, it is just my personal rambling.

Background

Plone is having popularity issues. This can be deducted by:

• PHP CMS are more popular (Google trends and so on)
• Plone lacks love
• By discussing with Python programmers outside Plone community: they see Plone as monster

Plone is a superior CMS product (feature wise, polish wise, tech wise, and so on). So how did this happen?

My theory, you are free to disagree, is that the community has been driving “wrong” ideologies

• Plone’s “buyers” are not users, but developers
• There has been no driving community factor to make Plone accessible for more developers. Plone community has been working to make Plone work for the existing community members, not new members. The consulting companies have been too busy to extract money from their existing big clients. It might even be that it’s the best interest of these companies to keep the barrier of entry high, as this way they protect the Plone business for themselves. Note that this strategy will fail in the point there is no longer Plone business.

The big boys are telling that customers are abandoning Plone, because there are no developers. I know a lot of Plone developers, looking for more Plone work.  I’ll translate this that there are not enough affordable Plone developers entering the market.

I am also not buying arguments like

• “don’t compare apples to oranges (Django vs. Plone)”
• “its history and evolved that way”
• “big tool must be complex”

The result

Plone is easy to use, but hard to develop. If you need to use code generators and boiler-plate in Python code, you are doing it wrong.

Plone is a soup of different, non-mainstream, technologies. To be a Plonista you need master: Python (ok docs), Plone user mode (good docs), Zope component architecture (some docs), ZCML (no useful docs), GenericSetup (no docs), TAL (some docs), Archetypes (basically no docs), buildout (aaaaaarghs) … You have too much to learn from material which does not exist. Today the material effectively does not exist if it’s not your first Google hit – I am against book.

New developers are not getting in. It is easier to duplicate basic Plone functionality in a hacky in-house Django CMS than customize Plone for your use case. Plone does not give any more added value to the new developers. There is no business motivation to adapt Plone as a technology platform in new ventures.

The quest

Let’s fix it, as I see the developer acceptance the most crucial challenge Plone community is facing if the community wants to survive.

What we can do here is

• Make Plone easier for developers
• Make more, better, learning material available
• Eliminate and protect against factors in Plone which hinder the goals of two above bullets (do not include new technologies or modules to core if they make developers sad)

To accomplish this we can

1. Promote “easy” technologies inside and outside Plone community (Dexterity over Archetypes, Grok over ZCML)
2. Making a standard workflow available for them (Django style tutorial that do 1, 2, 3 and you are a certified Plone developer)
3. Kill technologies which make the stack unnecessary complex and are not necessarily needed in newbie use cases (buildout, I am looking at you)
4. Replace old, badly documented and difficult solutions with easier ones (could we actually replace TAL with something more popular?)

Operational plan

Here are some concrete tasks what we could tackle in Sauna Sprint.

Make it easy department

• Plone auto-restart team: provide working and fast auto-restart á la Django or Tornado (hint: plone.reload doesn’t cut it – you need real restarts process-wise)
• Default custom product team: ship Plone with a file-system based custom product skeleton, so that when you need to include your first custom Python module you don’t need to learn about buildout and paster first . You actually can copy-paste in Python code and it just works.
• Through-the-web is happy again team: Make it possible to code and execute Python through-the-web so that it actually works.: a working replacement for RestrictedPython

Fix user-wise broken things department

• Buildout team: make buildout more user friendly: spit out error messages which are actually useful, have newbie buildout tutorial from the developer perspective. Nothing bad should happen when buildout is run. Make buildout to confirm version updates, etc.
• Anti-buildout team: how to install eggs without buildout. Why did we need the buildout in the first place?

This is how you do it department

• Tutorial team: Make “gapless” hand-holding tutorial for creating your first view, form and content-type. Make it so that you do not need to look outside the tutorial or learn anything unnecessary (don’t make me think)
• Picture team: draw big pictures of Plone architecture, how different parts are connected, so that new developers get hang on things easily
• Document generator team: Integrate collective.developermanual with Sphinx tool that generate the reference  documentation from the source code: views, viewlets, templates, CSS classes, portlets, etc.

Hasta la vista babe department

• Kill it with fire team: Hunt down all bad learning material and make it redirect to new learning material (no ExternalMethods, no Install.py, no plone_skins, etc.)
• Nuke it from orbit team: I am not sure what this team could do, but I like the feeling of the name

We hope that novice  developers attend the sprint. This way we can use you as guinea pigs and document the pain points on a road to become Plone developer. Don’t worry – we’ll inject you with proper medicine to numb the pain and forget bad memories.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Low-end VPS come with 512 MB memory or less
• Front front-end server Apache / Nginx / Varnish takes > 100 MB +  min. 20 MB for each child process
• Memecached takes its toll
• MySQL takes 200 – 400 MB
• Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
• Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

• MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
• MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

• Let’s halve key_buffer from 16M to 8M. It is used by MyISAM table cache.
• Halve query_cache_size. query_cache_size  = 8M. Also, decrease query_cache_limit to 512 K.
• Each connection, even if idle, will have 256 KB buffer. Decrease the number of max. connections. (XXX: not sure about this). Drop max connections from 100 -> 30, as we do not have that many concurrent visitors on the site. Also, set less aggressive thread_stack size.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

• Fine-tuning MySQL
• Showing what database engine MySQL tables are using
• MySQL server memory usage
• Reducing MySQL memory usage for low end boxes

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Prerequisites

What you need in order to benefit from these instructions

• Basic UNIX command-line knowledge
• SSH access to the server
• MySQL access to the database
• LAMP stack ready on the new server

These instructions have been tested on Ubuntu/Debian/Linux but they should work in other environments too.

Write down database access information

Get password from config.php file on the old server:

cd /var/www/phpBB3
cat config.php

Write down database name, username and password.

Copy files

Use rsync to remotely copy forum files to a new computer. On new computer, in /var/www folder

rsync -av --compress-level=9 user@oldserver.com:/var/www/phpBB3 .

Dump and copy database

Execute the following command on the new server. It takes SSH connection to the old server and dumps phpBB3 database to the new server over the SSH connection.

ssh user@oldserver.com -C -o CompressionLevel=9 mysqldump -u databaseuser --password=databasepassword --skip-lock-tables --add-drop-table databasename > phpbb3.sql

Create a new database

Use the old access information from config.php to create a database with identical access information on the new server. You need a MySQL root access to create new databases.

mysql -uroot -p

Create database and grant access to phpBB3 user for it.

mysql> create database databasename;

mysql> GRANT ALL ON databasename.* TO 'databaseuser'@'localhost' identified by 'databasepassword';

Load the database on the new server from the dump file:

mysql> connect databasename;

mysql> source phpbb3.sql

Configure Apache virtualhost for the new server

The last step is to set-up Apache virtual host on the new server, so you can access the phpBB3 using a domain name. Note that this doesn’t need to be a real domain name, but you can spoof the domain name using /etc/hosts file on your local workstation.

Add file /etc/apache2/sites-enabled/phpbb3.conf (or pick a filename based on forum name if you host multiple forums)

<VirtualHost *>
 ServerName yourdomainname.com

 DocumentRoot /var/www/phpBB3
 <Directory />
   Options FollowSymLinks
   AllowOverride None
 </Directory>

</VirtualHost>

Note that <virtualhost *> may change depending on how Apache has been set up to listen IP addresses and ports. Also if you are using a shared hosting package or VPS you might need to use the server control panel (cPanel) to do this step.

Then check if your new config file is ok and restart Apache:

apache2ctl configtest
apache2ctl graceful

Hosts spoofing trick

If you are not having a DNS server of your own which you can use for the copy you can always use /etc/hosts file trick to spoof domain names. This way you can make Apache to serve the forum from the server even if the forum is not connected to any real domain name yet.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Because of duck-typing, you should rigorously document how methods should be called (try epytext and its fields).
• Most open source Python projects do the exact opposite
• Even Python standard library is poorly documented and sets a very bad example (missing manual ???)
• Thus, programming in Python becomes nightmare of grepping through source code (the implementation) or stepping into it in pdb just to figure out how APIs should work (Plone/Zope, anyone?)

Should Python community stop in some point to focus on delivering better documentation instead of focusing on new features and goodies (like the syntax moratorium which was recently lifted)?

From my personal experience

• The best, and the only, person to document the code adequately is the person how originally wrote it
• Because the author already knows how to use the code he doesn’t need to care about the fact how to enable the code for other users.  Many libraries and projects are driven by “scratching your own need” mentality, not by “let’s make this a happy community” mentality. The exception is something like Facebook or Google whose sole purpose is to attract new users their platform bringing in new €€€.
• If you are developing a framework or community project make the documentation a requirement for deliverable and stick with it. If you let one person to skip one hour of writing documentation you are making 10 persons spending one hour figuring out how to use the damn thing.
• Doctests are not documentation. They are tests. They are extremely unreadable way to say “how I should use this thing”, because doctests are often executed in the context of test stubs which do not reflect connections to the other parts of the framework or real contexts.
• “Buy a book – it tells you everything” business model is not feasible in long run. Books get old. Books are not searchable. People don’t buy books.

The good documentation is a way to differentiate, and win, in the situation where there are competing frameworks. I believe the success of Django was mostly driven by its good documentation.

This points could be applied to other duck-typed, open source driven programming languages (PHP anyone?). With good documentation we can reduce the need of Valium recipes for everyone of us.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Use your favorite editor to edit php.ini file in /Applications/XAMPP/etc/php.ini – sudo priviledges needed, Smultron does it out of the box.

Change lines:

log_errors = Off

;error_log = filename

To:

log_errors = on

error_log = /tmp/php.log

Restart Apache using XAMPP controller in Finder -> Applications.

Now use the following UNIX command to see continuous log flow in your terminal:

tail -f /tmp/php.log

See also the earlier article about XAMPP and file permissions.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

This Google App Engine feature came me as a surprise, though it makes perfect sense. Your site is slow if it has low traffic.

Google App Engine runs Python code on instances. By default, instances are dynamic. Instances are shutdown if they do not have enough traffic (requests per minute). Thus, when you get the individual hits to App Engine now and then, App Engine must restart your instance every time for each hit.

When this happens, you see the following in App Engine console logs for every request on low volume traffic:

This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.

It is not always ok to add 500 – 2000 milliseconds processing delay on the top of the normal processing time. Google’s own recommendation was that each page should be served within 200 milliseconds.

There are three ways to optimize this issue

• Use App Engine premium feature “Always on” 0,30 $ / day which keeps your instance always running
• Use cron job or such to keep your instance alive (polling once in a minute seems to do the job)
• Optimize your imports and split your code to several modules with light amount of imports, so that start up is fast (modules are imported only once)

We are using Zabbix software to monitor our sites (sidenote: I don’t recommend Zabbix as the first monitoring software choice as it is very difficult to use and has bad user experience, alienating both sysadmins and developers away from it). This is what we had before optimizations – App Engine was starting a new process for every request:

… and this is output we got after optimizations:

Here is the corresponding diagram after optimizations from App Engine dashboard itself. These processing times are without network latency. As far as I know Google does not expose the endpoints of App Engine hosting, so you don’t know from which site of the world your responses come from. By comparing this diagram to the diagram above, you can see how Internet traffic is affecting to your App Engine application.

The PITA of dying instances

For some reason, App Engine instances misbehave sometimes. This causes the HTTP requests die ungracefully.

Normally it is not a problem as you lost few page loads now and then. People are used to “Internet grade” service and can hit the refresh button if they have problems opening a page.

However if you are monitoring your site and the site gives an unnecessary alarm in the middle of the night, waking up your bastard operator from Hell, he will be very angry next morning and tell you to migrate the crappy software from unreliable Python / App Engine to more reliable PHP servers

This is what you see in App Engine logs:

A serious problem was encountered with the process that handled this request, causing it to exit.
This is likely to cause a new process to be used for the next request to your application.
If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

After digging in deeper, you see that it is a problem of instating a new object in the database, exceeding 30 seconds hard limit for processing a HTTP request:

2011-03-09 05:06:20.794 / 500 30094ms 86cpu_ms 40api_cpu_ms
0kb Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;
.NET CLR 2.0.50727),gzip(gfe),gzip(gfe),gzip(gfe)

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
  File "/base/data/home/apps/mfabrikkampagne/1.347249742610459821/main.py", line 494, in main
    run_wsgi_app(application)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 97, in run_wsgi_app
    run_bare_wsgi_app(add_wsgi_middleware(application))
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 115, in run_bare_wsgi_app
    result = application(env, _start_response)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 515, in __call__
    handler.get(*groups)
  File "/base/data/home/apps/mfabrikkampagne/1.347249742610459821/main.py", line 296, in get
    try: self.session = Session()

So it looks like there is a temporary hick-up in Google App Engine’s Data Store (Big Table?). In the example above the error comes from gaeutilities‘s Session model, but it could be any other model.

It is possible to catch DeadlineExceededError and temporarily work-around it, as shown in App Engine documentation.

The best way to handle this situation is to adjust your monitoring software – Zabbix in our case. Zabbix allows you to configure triggers so that they don’t alarm on every bad item state change. Instead, you can use min() function and trigger the alarm after the trigger condition has failed every time during a monitoring period. Just make sure that the trigger period is at least twice long as the update interval of your web scenario: this way Zabbix can logs at least two item state changes and allows one of them to be failed one.

For example if

• Update interval of web scenario is 60 seconds
• Trigger function must check minimal failures of 1 during 2*60 seconds + some buffer = 150 seconds.

{xxx.fi:web.test.fail[de.mfabrik.com].min(150)}=1

This will allow one failed response before triggering the alarm.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

file_put_contents("/tmp/log.txt", "this is a test entry\n");

The proper method would be

• Configure error log in php.ini
• Use error_log()
• …. or use logging facility of your PHP framework

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Next: Plone overtakes WordPress in popularity.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Work as the site user

The site can be installed locally (under /home) or globally (under /var/www). If the site is installed globally you might need to set your effective user. You should perform the command as the same user who owns the site PHP files. You should not be using root use as the owner of the site files. If you run Joomla! site through Apache, the effective user is www-data. Use sudo command to switch to this user. Use ls -l command to figure out which user you are.

sudo -i -u www-data

Back up code files

Use tar command to pack the existing folder structure in the case the update will destroy critical files.

cd /var/www/yoursite
tar -cjf yoursite.tar.bz2 *

Back up database

mysqldump -umysql_user_name-p mysql_database_name > mysql_database_name.sql

Check Joomla! version

Download version specific Joomla! update pack

wget http://joomlacode.org/gf/download/frsrelease/12611/53380/Joomla_1.5.18_to_1.5.20-Stable-Patch_Package.tar.gz

tar -xvzf Joomla_1.5.18_to_1.5.20-Stable-Patch_Package.tar.gz

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter