mFabrik - mobile sites, apps, HTML5 and CMS software development » php

Reducing MySQL memory usage on Ubuntu / Debian Linux

Mikko Ohtamaa — Thu, 31 Mar 2011 10:56:46 +0000

If you are running your services on a low end virtual hosting every byte of memory you can save is important. The memory is often the limiting factor of how many applications you can run on VPS: CPUs are shared, memory not, on the same physical host.

Low-end VPS come with 512 MB memory or less
Front front-end server Apache / Nginx / Varnish takes > 100 MB + min. 20 MB for each child process
Memecached takes its toll
MySQL takes 200 – 400 MB
Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

Let’s halve key_buffer from 16M to 8M. It is used by MyISAM table cache.
Halve query_cache_size. query_cache_size = 8M. Also, decrease query_cache_limit to 512 K.
Each connection, even if idle, will have 256 KB buffer. Decrease the number of max. connections. (XXX: not sure about this). Drop max connections from 100 -> 30, as we do not have that many concurrent visitors on the site. Also, set less aggressive thread_stack size.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Copy/move phpBB3 forum from a server to another computer (Ubuntu/Linux)

Mikko Ohtamaa — Wed, 30 Mar 2011 13:40:11 +0000

Here are short instructions what you need to do in order to move / copy phpBB3 forum.

Prerequisites

What you need in order to benefit from these instructions

Basic UNIX command-line knowledge
SSH access to the server
MySQL access to the database
LAMP stack ready on the new server

These instructions have been tested on Ubuntu/Debian/Linux but they should work in other environments too.

Write down database access information

Get password from config.php file on the old server:

cd /var/www/phpBB3
cat config.php

Write down database name, username and password.

Copy files

Use rsync to remotely copy forum files to a new computer. On new computer, in /var/www folder

rsync -av --compress-level=9 user@oldserver.com:/var/www/phpBB3 .

Dump and copy database

Execute the following command on the new server. It takes SSH connection to the old server and dumps phpBB3 database to the new server over the SSH connection.

ssh user@oldserver.com -C -o CompressionLevel=9 mysqldump -u databaseuser --password=databasepassword --skip-lock-tables --add-drop-table databasename > phpbb3.sql

Create a new database

Use the old access information from config.php to create a database with identical access information on the new server. You need a MySQL root access to create new databases.

mysql -uroot -p

Create database and grant access to phpBB3 user for it.

mysql> create database databasename;

mysql> GRANT ALL ON databasename.* TO 'databaseuser'@'localhost' identified by 'databasepassword';

Load the database on the new server from the dump file:

mysql> connect databasename;

mysql> source phpbb3.sql

Configure Apache virtualhost for the new server

The last step is to set-up Apache virtual host on the new server, so you can access the phpBB3 using a domain name. Note that this doesn’t need to be a real domain name, but you can spoof the domain name using /etc/hosts file on your local workstation.

Add file /etc/apache2/sites-enabled/phpbb3.conf (or pick a filename based on forum name if you host multiple forums)


 ServerName yourdomainname.com

 DocumentRoot /var/www/phpBB3
 
   Options FollowSymLinks
   AllowOverride None

Note that may change depending on how Apache has been set up to listen IP addresses and ports. Also if you are using a shared hosting package or VPS you might need to use the server control panel (cPanel) to do this step.

Then check if your new config file is ok and restart Apache:

apache2ctl configtest
apache2ctl graceful

Hosts spoofing trick

If you are not having a DNS server of your own which you can use for the copy you can always use /etc/hosts file trick to spoof domain names. This way you can make Apache to serve the forum from the server even if the forum is not connected to any real domain name yet.

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Enable PHP log output (error_log) on XAMPP on OSX

Mikko Ohtamaa — Tue, 15 Mar 2011 21:29:18 +0000

If you are using XAMPP to develop PHP software (WordPress, Joomla!) on OSX you might want to get some advanced logging output from your code. PHP provides nice error_log() function, but it is silent by default. Here are short instructions how to enable it and follow the log.

Use your favorite editor to edit php.ini file in /Applications/XAMPP/etc/php.ini – sudo priviledges needed, Smultron does it out of the box.

Change lines:

log_errors = Off

;error_log = filename

To:

log_errors = on

error_log = /tmp/php.log

Restart Apache using XAMPP controller in Finder -> Applications.

Now use the following UNIX command to see continuous log flow in your terminal:

tail -f /tmp/php.log

See also the earlier article about XAMPP and file permissions.

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Ghetto logging for PHP

Mikko Ohtamaa — Thu, 03 Mar 2011 12:49:55 +0000

In the case you need to quickly see if certain code path is executed on a production server for which config cannot be changed.

file_put_contents("/tmp/log.txt", "this is a test entry\n");

The proper method would be

Configure error log in php.ini
Use error_log()
…. or use logging facility of your PHP framework

Get developers Subscribe mFabrik blog in a reader Follow me on Twitter

Now hiring (Developer, Helsinki)

veikko — Mon, 29 Mar 2010 09:14:31 +0000

We are looking for a new developer to join our production team in Helsinki, Finland. Check out the details at our web site:

http://mfabrik.com/about-us/jobs/developer-helsinki

Eclipse web developer plug-in memo

Mikko Ohtamaa — Tue, 27 Nov 2007 11:46:19 +0000

Currently I work in quite wide field of software development: Python (standalone, Plone, Zope, Django), PHP, Java, Symbian and embedded Linux. I am using Eclipse for development, since it’s pretty much the only consistent platform filling my needs. The nature of work also forces me to use different computers (Mac/Windows/Linux) with different clients. This drives me to reinstall Eclipse now and then.

Below are my personal notes what plug-ins are needed to get “perfect” Eclipse set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.

Eclipse setup

Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. You can use dummy text as the name of update site.