The following commands dump a MySQL database from a remote server and create a corresponding database on the local computer.

The instructions have been tested on OSX and Linux (Ubuntu/Debian). On-line SSH compression is applied, so transferring SQL files, which are text content and compress well, should be around 6x faster than normal.

(Well… actually the script is six lines, but because this is my blog I’ll decide it doesn’t count)

The script

• Remotely runs mysqldump and puts the result to a local file
• Creates a MySQL database and corresponding user with full access to this database
• Reads the content of mysqldump to the newly created database

 ssh user@dserver.com -C -o CompressionLevel=9 mysqldump -u YOURDATABASEUSER --password=YOURDATABASEPASSWORD --skip-lock-tables --add-drop-table YOURDATABASENAME > YOURDATABASENAME.sql
mysql -uroot -p
create database YOURDATABASENAME;
connect YOURDATABASENAME;
source YOURDATABASENAME.sql
GRANT ALL ON YOURDATABASENAME.* TO 'YOURDATABASEUSER'@'localhost' identified by 'YOURDATABASEPASSWORD';

Leave out create database and GRANT for the subsequent runs – all data on the local computer will be replaced.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• Low-end VPS come with 512 MB memory or less
• Front front-end server Apache / Nginx / Varnish takes > 100 MB +  min. 20 MB for each child process
• Memecached takes its toll
• MySQL takes 200 – 400 MB
• Each Python / PHP process takes at least 15 MB and you need parallel processes for paraller HTTP requests (FCGI, pre-fork, others… )
• Operating system processes need some memory (SSH, cron, sendmail)

As you can see it gets very crowded in 512 MB.

It’s especially troublesome since the memory is allocated lazily and the memory usage builds up slowly. In some point caches are no longer caches, but swapped to a disk – virtual memory usage grows beyond available RAM. To keep the server response, everything time critical should fit to RAM once and if the processes themselves don’t know how to release memory in this situation you need to tune a memory cap for them.

MySQL memory consumption

MySQL can be a greedy bastard what comes to memory consumption. Here on this server MySQL seems to take 417M virtual memory which seems to be little excessive for just running two WordPress instances and one Django / Python application:

1310 mysql     20   0  417M 21100  2776 S  0.0  1.2  0:00.00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/v

After some tuning I was able to bring it down a bit

3354 mysql     20   0  276m  19m 2848 S    0  1.2   3:41.19 mysqld

A reduction of 130 MB, or 1/4 of the server total memory. Not bad.

Use mtop to monitor running MySQL, its querieries, etc. so you know what’s going on. As you can see this MySQL has very good cache rate meaning that basically it is keeping everything in memory. If the content of the sites is less than 10 MBytes total, 400 MB contains plenty of space to cache the content:

load average: 0.05, 0.08, 0.16 mysqld 5.0.51a-3ubuntu5.8-log up 1 day(s), 19:47 hrs                                                             
2 threads: 1 running, 6 cached. Queries/slow: 187.1K/0 Cache Hit: 99.39%

What eats memory

I am not an expert on MySQL, so I hope someone with more insight could post comments regarding how to tune MySQL for low memory situations and how it is expected to behave.

Some ideas I run through my head

• MySQL default cache settings are not too tight on Ubuntu/Debian, making it suitable for moderate loads, not low loads. If you don’t have much content, everything is just kept in memory (even if not needed)
• MySQL uses round robin for connections and if there is 100 max connections it will allocate a thread stack for each connection (someone please confirm this – I found contracting infos).

Configuring MySQL

Here are listed some methods how to reduce the memory usage. This is what I done on this little box

MySQL is mostly configured in /etc/mysql/my.cnf on Ubuntu / Debian.

• Let’s halve key_buffer from 16M to 8M. It is used by MyISAM table cache.
• Halve query_cache_size. query_cache_size  = 8M. Also, decrease query_cache_limit to 512 K.
• Each connection, even if idle, will have 256 KB buffer. Decrease the number of max. connections. (XXX: not sure about this). Drop max connections from 100 -> 30, as we do not have that many concurrent visitors on the site. Also, set less aggressive thread_stack size.

The final adjustments

key_buffer              = 8M
max_connections         = 30
query_cache_size        = 8M
query_cache_limit       = 512K
thread_stack            = 128K

More info

• Fine-tuning MySQL
• Showing what database engine MySQL tables are using
• MySQL server memory usage
• Reducing MySQL memory usage for low end boxes

Send in more tips please! Is 32-bit better than 64-bit for low end VPS, how much this affects MySQL?

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Prerequisites

What you need in order to benefit from these instructions

• Basic UNIX command-line knowledge
• SSH access to the server
• MySQL access to the database
• LAMP stack ready on the new server

These instructions have been tested on Ubuntu/Debian/Linux but they should work in other environments too.

Write down database access information

Get password from config.php file on the old server:

cd /var/www/phpBB3
cat config.php

Write down database name, username and password.

Copy files

Use rsync to remotely copy forum files to a new computer. On new computer, in /var/www folder

rsync -av --compress-level=9 user@oldserver.com:/var/www/phpBB3 .

Dump and copy database

Execute the following command on the new server. It takes SSH connection to the old server and dumps phpBB3 database to the new server over the SSH connection.

ssh user@oldserver.com -C -o CompressionLevel=9 mysqldump -u databaseuser --password=databasepassword --skip-lock-tables --add-drop-table databasename > phpbb3.sql

Create a new database

Use the old access information from config.php to create a database with identical access information on the new server. You need a MySQL root access to create new databases.

mysql -uroot -p

Create database and grant access to phpBB3 user for it.

mysql> create database databasename;

mysql> GRANT ALL ON databasename.* TO 'databaseuser'@'localhost' identified by 'databasepassword';

Load the database on the new server from the dump file:

mysql> connect databasename;

mysql> source phpbb3.sql

Configure Apache virtualhost for the new server

The last step is to set-up Apache virtual host on the new server, so you can access the phpBB3 using a domain name. Note that this doesn’t need to be a real domain name, but you can spoof the domain name using /etc/hosts file on your local workstation.

Add file /etc/apache2/sites-enabled/phpbb3.conf (or pick a filename based on forum name if you host multiple forums)

<VirtualHost *>
 ServerName yourdomainname.com

 DocumentRoot /var/www/phpBB3
 <Directory />
   Options FollowSymLinks
   AllowOverride None
 </Directory>

</VirtualHost>

Note that <virtualhost *> may change depending on how Apache has been set up to listen IP addresses and ports. Also if you are using a shared hosting package or VPS you might need to use the server control panel (cPanel) to do this step.

Then check if your new config file is ok and restart Apache:

apache2ctl configtest
apache2ctl graceful

Hosts spoofing trick

If you are not having a DNS server of your own which you can use for the copy you can always use /etc/hosts file trick to spoof domain names. This way you can make Apache to serve the forum from the server even if the forum is not connected to any real domain name yet.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

• You are using MySQL installation from mysql.com (and not e.g. from Macports)
• You want to install MySQL-python egg using easy_install or buildout

You’ll get an error

Getting distribution for 'MySQL-python'.
install_dir /Users/moo/project/eggs/tmp8-j13e
sh: mysql_config: command not found
Traceback (most recent call last):
 File "<string>", line 1, in ?
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1855, in main
 with_ei_usage(lambda:
 File "/Users/moo/code/collective.buildout.python/python-2.4/lib/python2.4/site-packages/distribute-0.6.14-py2.4.egg/setuptools/command/easy_install.py", line 1836, in with_ei_usage
 return f()  
...

 31, in <lambda>
 {'__file__':setup_script, '__name__':'__main__'}
 File "setup.py", line 15, in ?
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 43, in get_config
 File "/var/folders/O8/O8pt7q52F7Oi+P3O0pNqq++++TI/-Tmp-/easy_install-g5cvxl/MySQL-python-1.2.3/setup_posix.py", line 24, in mysql_config
EnvironmentError: mysql_config not found
An error occured when trying to install MySQL-python 1.2.3.Look above this message for any errors thatwere output by easy_install.
While:
 Installing instance.
 Getting distribution for 'MySQL-python'.
Error: Couldn't install: MySQL-python 1.2.3

The reason is that MySQL installer puts MySQL under /usr/local and does not expose MySQL utilites directly to path. MySQL-python expects to find utility mysql_config to configure compile time paths when building native bindings using GCC.

Workaround:

sudo -i
cd /usr/bin
ln -s /usr/local/mysql-5.1.46-osx10.5-x86_64/bin/mysql_config .

Then rerun installer.

Though it is a bit harsh solution to modify system folder, it gets the thing done in simple manner.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

By default XAMPP users nobody user for running Apache. If you try to load any file created by your normal system user Apache will give access denied error. This means that either you must make all PHP files readable for everyone, or change the file owner. This is painful if you need to repeat the process for every file.

Instead, it is easier to make XAMPP run under the permissions of your local user.

Edit /Applications/XAMPP/etc/httpd.conf:

<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
User yourusername
Group staff
</IfModule>
</IfModule>

<IfModule !mpm_winnt_module><IfModule !mpm_netware_module>

DocumentRoot "/Users/yourusername/yoursiteroot"

You also need to change the location of setting default folder execution rights htdocs location to your development location

#
# This has been changed from /Applications/XAMPP/htdocs
#
<Directory "/Users/moo/code/joomla/mobilejoomla_dev">
    Options Indexes FollowSymLinks ExecCGI Includes
   AllowOverride All

</Directory>

And now Apache will happily eat all files you create or edit.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Prerequisitements

• How to deal with UNIX command line programs

Use MySQL command to login to running MySQL instance. If this is on local host you can try

mysql -uroot # no password

or

mysql -uroot -p # asks for password

Use -H option to connect to external MySQL host (on shared hosting).

Then create a new database using CREATE DATABASE command. Here we create a database called joomla.

mysql> create database joomla;
Query OK, 1 row affected (0.00 sec)

And then we create a MySQL user whose access is limited to this database only. In this case we create admin user with admin password. Note that MySQL accepts local connections only by default, so MySQL or this user is not exposed to Internet (and potential attacks).

Username is joomla and password is joomladbpassword.

GRANT ALL ON joomla.* TO 'joomla'@'localhost' identified by 'joomladbpassword';

… and now you can proceed to install Joomla.

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Work as the site user

The site can be installed locally (under /home) or globally (under /var/www). If the site is installed globally you might need to set your effective user. You should perform the command as the same user who owns the site PHP files. You should not be using root use as the owner of the site files. If you run Joomla! site through Apache, the effective user is www-data. Use sudo command to switch to this user. Use ls -l command to figure out which user you are.

sudo -i -u www-data

Back up code files

Use tar command to pack the existing folder structure in the case the update will destroy critical files.

cd /var/www/yoursite
tar -cjf yoursite.tar.bz2 *

Back up database

mysqldump -umysql_user_name-p mysql_database_name > mysql_database_name.sql

Check Joomla! version

Download version specific Joomla! update pack

wget http://joomlacode.org/gf/download/frsrelease/12611/53380/Joomla_1.5.18_to_1.5.20-Stable-Patch_Package.tar.gz

tar -xvzf Joomla_1.5.18_to_1.5.20-Stable-Patch_Package.tar.gz

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

The instructions may apply for other versions too.

Prerequisitements

What you need to have in order to use this how to

• Basic UNIX file permissions knowledge
• Basic UNIX shell knowledge
• You have a Linux server (Ubuntu / Debian) for which you have root user access and you plan to use this server to host one or several Joomla! sites
• Apache and MySQL instaleld on your server

User setup

Set-up an UNIX user on a dedicated server for Joomla! hosting. The user can SSH in the box and write to his home folder, /tmp and /var/www site folder.

We create a user called “user” in this instructions. Replace it with the username you desire. We also use the example site name (www).yoursite.com.

sudo adduser user # Asks for the password and created /home/user

sudo mkdir /var/www/user
sudo chmod -R user:user /var/www/user # Only user has writing access to this folder

Setup MySQL user account

Install MySQL as per Debian/Ubuntu instructions.

Login as MySQL admin user (may vary depending how your MySQL is configured). Note that first you will be asked for sudo password, then for MySQL administrative user password.

sudo mysql -u admin -p

CREATE DATABASE user DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON user.* TO 'user'@'localhost' identified by 'zxc123zxc'; 

Extract Joomla! installation files

Enter the folder which will contain web site PHP files.

sudo -i -u user # pose yourself as UNIX user who runs the site

cd /var/www/user

wget http://joomlacode.org/gf/download/frsrelease/12350/51111/Joomla_1.5.18-Stable-Full_Package.zip

unzip Joomla_1.5.18-Stable-Full_Package.zip

exit

Set file permission

Note that during Joomla’s browser based installation Apache’s www-data must have write access to folder in order to create configuration.php file. We will later remove this access right.

We will set Joomla! files under UNIX group group www-data so that Apache can read them. Certain files are set to be writable. This must be done as root user.

sudo chown -R user:www-data /var/www/user # Make user group to www-data

sudo chmod g+wrx /var/www/user # Read only access to www-data user. Write access for installation, will be later removed.

Now ls -l command in /var/www/user should give you something like this for fil masks:

drwxr-xr-x 11 user www-data    4096 2010-05-28 10:22 plugins
-rwxr--r--  1 user www-data     304 2010-05-28 10:21 robots.txt
drwxr-xr-x  6 user www-data    4096 2010-05-28 10:22 templates

Creating Apache configuration

sudo nano /etc/apache2/sites-enabled/yoursite.conf

   ServerName yoursite.com
   ServerAlias www.yoursite.com
   ServerAdmin info@yourcompany.com

   LogFormat       combined
   TransferLog     /var/log/apache2/yoursite.log

   # Make sure this virtual host if capable of executing PHP5
   Options +ExecCGI
   AddType application/x-httpd-php .php .php5

   # Point to www folder where Joomla! is extracted
   DocumentRoot /var/www/yoursite

   # Do not give illusion of safety
   # as PHP safe_mode really is a crap
   # and only causes problems
   php_admin_flag safe_mode off

   #
   # This entry will redirect traffic www.yoursite.com -> yoursite.com
   # Assume mod_rewrite is installed and enabled on Apache
   # 301 is HTTP Permanent Redirect code
   RewriteEngine On
   RewriteCond %{HTTP_HOST} ^www\.yoursite\.com [NC]
   RewriteRule (.*) http://yoursite.com$1 [L,R=301]

</VirtualHost>

Faking the DNS entry

sudo gedit /etc/hosts

# Force this hostname to go to your server public IP address from your local computer

123.123.123 yoursite.com www.yoursite.com

Start Joomla! browser based installation

http://yoursite.com

Secure the configuration

sudo chmod -R g-w /var/www/user # Remote write permission
sudo rm -rf /var/www/user/installation # Remove installation directory
# Add write permission to folders which contain writable files
sudo chmod -R g+x /var/www/user/logs
sudo chmod -R g+x /var/www/user/images
sudo chmod -R g+x /var/www/user/tmp
sudo chmod -R g+x /var/www/user/images

Setting up htaccess files

Joomla! comes with a sample htaccess file which has some security measurements by having RewriteRules to prevent malformed URL access.

To install this file do the following

sudo -i
cd /var/www/user
cp htaccess.txt .htaccess
chmod user:www-data .htaccess # Set file permission to be readable by Apache and writable by the UNIX user

Then we create a .htaccess file which we will place in all folders with Joomla! write access to prevent execution of PHP files in these folders. First we create htaccess.limited file which we use as a template.

sudo -i
cd /var/www/user
nano htaccess.limited # Open text editor

Use the following htaccess.limited content

# secure directory by disabling script execution
AddHandler cgi-script .php .pl .py .jsp .asp .htm .shtml .sh .cgi
Options -ExecCGI -Indexes

And put the master template htaccess.limited  to proper places

cp htaccess.limited media/.htaccess
chown -R user:www-data media/.htaccess 

cp htaccess.limited tmp/.htaccess
chown -R user:www-data tmp/.htaccess 

cp htaccess.limited logs/.htaccess
chown -R user:www-data logs/.htaccess

cp htaccess.limited images/.htaccess
chown -R user:www-data images/.htaccess

Start using the site

Setting outgoing email

Test outgoing email

Maintaining file permission

sudo usermod -a -G www-data user # Add user to www-data group so that it can set group permissions

sudo -i -u user # Login as your UNIX user
chgrp -R www-data templates libraries # Fix group ownership
chmod -R g+rx libraries templates # Set read access for the group

Get developers  Subscribe mFabrik blog in a reader Follow me on Twitter

Here is a workaround based on xinetd daemon. These are sample commands for Ubuntu/Debian.

Go to root

sudo -i

Install xinetd

apt-get install xinetd

Add a new xinetd mapping

pico /etc/xinetd.d/mysql

service mysql
{
    only_from	   = localhost mansikki.redinnovation.com 80.75.108.108 server213-171-218-5.livedns.org.uk 213.171.218.5
    flags          = REUSE
    socket_type    = stream
    wait           = no
    user           = root
    redirect       = 127.0.0.1 3306
    log_on_failure += USERID
    interface 	   = 84.34.147.68
}

Restart xinetd

/etc/init.d/xinetd restart

To debug xinetd:

/etc/init.d/xinetd stop

xinetd -d

xinetd only_from directive also gives an access control by allowed source IP addresses. This protects your MySQL against bots and brute force attacks.

Note that iptables DNAT translation doesn’t work (easily). Localhost packets don’t travel PREROUTING and POSTROUTING chains.