About

mFabrik Blog is about mobile and web software development, open source and Linux. We tell exciting tales where business, technology, web and mobile convergence.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

One-liner to copy remote MySQL database to local computer

Posted on July 23, 2011

The following commands dump a MySQL database from a remote server and create a corresponding database on the local computer.

The instructions have been tested on OSX and Linux (Ubuntu/Debian). On-line SSH compression is applied, so transferring SQL files, which are text content and compress well, should be around 6x faster than normal.

(Well… actually the script is six lines, but because this is my blog I’ll decide it doesn’t count)

The script

  • Remotely runs mysqldump and puts the result to a local file
  • Creates a MySQL database and corresponding user with full access to this database
  • Reads the content of mysqldump to the newly created database
 ssh user@dserver.com -C -o CompressionLevel=9 mysqldump -u YOURDATABASEUSER --password=YOURDATABASEPASSWORD --skip-lock-tables --add-drop-table YOURDATABASENAME > YOURDATABASENAME.sql
mysql -uroot -p
create database YOURDATABASENAME;
connect YOURDATABASENAME;
source YOURDATABASENAME.sql
GRANT ALL ON YOURDATABASENAME.* TO 'YOURDATABASEUSER'@'localhost' identified by 'YOURDATABASEPASSWORD';

Leave out create database and GRANT for the subsequent runs – all data on the local computer will be replaced.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Testing mobile websites with Firefox Mobile for PC (Fennec desktop)

Posted on July 8, 2011

Firefox Mobile (Fennec) has also desktop builds. They are very useful for mobile web site testing as the browser is fast, has real keyboard and is only one mouse click away.

Here are instructions how to run Firefox Mobile on Ubuntu Linux (tested on 32-bit Ubuntu 10.10)

wget http://releases.mozilla.org/pub/mozilla.org/mobile/releases/latest/linux-i686/fennec-5.0.en-US.linux-i686.tar.bz2
tar -xjf fennec-5.0.en-US.linux-i686.tar.bz2
cd fennec
./fennec

.. and thats all you need. It works out of the box! 400x times faster than using Android emulator browser.

There are also OSX and Windows builds available.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Sticky session load balancing with Apache and mod_balancer on Ubuntu Linux

Posted on April 15, 2011

Apache 2.2 can do load balancing with sticky sessions. However, there is a catch. You need to use mod_headers module to set a cookie based on the chosen balancer member for the first request and then route the subsequent requests to this client.

Use cases

The method described here works in every situation and does not rely on client IP address, etc. The only downside is that if one balancer member goes down all subsequent requests for it will die. So this method can be only used for load balancing, not for high availability (I am not sure if BALANCER_ROUTE_CHANGED environment variable is set when a balancer member is lost and would redirect the clients to a new balancer member).

This requests were tested on Ubuntu Linux, but may as well work in other environments.

Setting route configution in virtual host

Create a balancer

<Proxy balancer://yourlb>
 BalancerMember http://127.0.0.1:13001/ route=1
 BalancerMember http://127.0.0.1:13002/ route=2
 BalancerMember http://127.0.0.1:13003/ route=3
 BalancerMember http://127.0.0.1:13004/ route=4
</Proxy>

Set the cookie using mod_headers. Note that the cookie must be in format [session name].[route id] (the dot is required). It seems to be possible to leave session name empty.

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

Make ProxyPass  follow the cookie (Zope virtual host monster style with HTTPS)

ProxyPass / balancer://lbsits/http://localhost/VirtualHostBase/https/yoursite.org:443/yourplonesiteid/VirtualHostRoot/ stickysession=ROUTEID

Note: Hard restart is required. apache2ctl graceful is not enough to make new balancer rules effective.

Testing

Use wget

wget -S https://yoursite.org/

See that Set-Cookie: ROUTE_ID is present and it contains a valid value (is not empty)

HTTP/1.1 200 OK
 Date: Wed, 13 Apr 2011 15:21:52 GMT
 Server: Zope/(Zope 2.10.9-final, python 2.4.5, linux2) ZServer/1.1 Plone/3.3.3
 Content-Length: 23197
 Expires: Sat, 01 Jan 2000 00:00:00 GMT
 Content-Type: text/html;charset=utf-8
 Content-Language: en
 Set-Cookie: I18N_LANGUAGE="en"; Path=/
 Set-Cookie: ROUTEID=.1; path=/

More info

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Reducing MySQL memory usage on Ubuntu / Debian Linux

Posted on March 31, 2011

If you are running your services on a low end virtual hosting every byte of memory you can save is important. The memory is often the limiting factor of how many applications you can run on VPS: CPUs are shared, memory not, on the same physical host.

  • Low-end VPS come with 512 MB memory or less
  • Front front-end server Apache / Nginx / Varnish takes > 100 MB +  min. 20 MB for each child process
  • Memecached takes its toll
  • MySQL takes 200 – 400 MB
  • Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
  • Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

  • MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
  • MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter