Tag: production

Windows Azure is offering Extra Small Virtual Machines (1 vCPU, 768 MB RAM) at $9/mo, free for first 3 months. Yes, you read it right. You can get a Windows VMs at $9 (until May 31st) and Linux VMs at $15, at the same rate as Amazon Micro Instance, which is free for first year (boo Microsoft!). However, MSDN Ultimate subscribers get 1500h worth of Small VM hours for free! It is a good time to grab a VM or two and setup your own servers, where you can host your own blog and showcase your awesome open source projects. VMs are your own mini server, where you get full remote desktop access and do whatever you like. You don’t have to be limited to a web based control panel. It’s your own baby server on the cloud, fully scalable and redundant.

In this article, I will show you show to setup a rock solid Linux instance and configure some useful security and maintenance features:

• Tune Apache, MySql, PHP to run within an Extra Small VM within the memory limits.
• Secure your server from common brute force attacks.
• Harden a WordPress instance and make it resilient to common failures.
• Optimize your website for maximum client side caching and improve page load time.
• Some handy Linux scripts and cron jobs that can take care of your server most of the time, identifying and fixing common problems and let you have your peace of mind.

This article is written from a Windows user point of view, who has never used Linux before. So, Linux gurus, don’t roll your eyes at me.

Create a Linux VM

Let’s first create a new Ubuntu server.

The admin user name on the Linux server is testadmin. We will use this to login to the Linux server.

Let’s select the “Provide Password” option instead of “Upload SSH key” to make the configuration simpler. If you use password option, then every time you login to the Linux VM using Putty (a client for working on Linux servers remotely from Windows), you will have to enter username and password. But if you use the certificate option, you won’t have to enter password always. It is definitely the better way to do it, but it is just a lot of extra steps to configure it. For simplicity, let’s ignore this for now.

Let’s name the server and choose a location where you want the server. You should choose a location where most of your users are going to come from. If you are hosting sites for Asia, do choose an Asian location. You will get much better website load performance because the network roundtrip time from your users computer to the server will be significantly lower.

Once the VM is created, it will look like this:

Go to the “Endpoints” tab, and see the public port that has been opened for SSH. SSH is the Remote Desktop Protocol (RDP) equivalent for Linux.

So, the server’s DNS is ubuntu23052013.cloudapp.net and SSH public port is 54423.

Connect to the VM

Let’s get Putty and configure it to connect to this VM:

Put the DNS and the port. Then put a name on “Saved Sessions” and click Save. Then go to “Appearance” and change the font to Consolas, 15. Come back to this, and click “Save” again.

Now click “Open” and you will be taken to the remote session on the server. You will be asked to accept the server, just click Yes. Then use testadmin as username and the password you have configured while creating the VM.

Congratulations! You are now into a Linux server. Don’t be shocked, there’s no graphical client like Windows Remote Desktop. It is all command line based console. But you will be amazed how quickly you can get things done on Linux via command line. Windows Command Line isn’t even remotely close to being as powerful as Linux.

First, we will go to elevated command line, using the linux “sudo su” command.

When you see “root@Test”, that means you are in “root” mode (Windows equivalent of Administrator privilege).

Setup upgrade

Let’s do a quick check to see if there’s any upgrade pending. Run the command “do-release-upgrade”.

root@Test:/home/testadmin# do-release-upgrade
Checking for a new Ubuntu release
No new release found

Best to do the upgrade first before you configure anything.

By the way, you can copy some text and paste it into Putty by just right clicking on the Putty window.

Now let’s setup automatic security patch installation by running this command.

unattended-upgrade

Add swap file to get virtual memory

Linux VM doesn’t come with Swap file. That means you only have the 768 MB RAM to play with. If by any chance any application exceeds the RAM, it crashes. So, you need to add a swap file so that you get plenty of virtual memory to play with.

Run the command “swapon –s” to check if there’s any swap file. If none, it will look like this:

root@Test:/home/testadmin# swapon -s
Filename                                Type            Size    Used    Priority
root@Test:/home/testadmin#

Let’s create one:

dd if=/dev/zero of=/swapfile bs=1024 count=512k

This will create a 512 MB empty file.

You should see an output like this if everything works out:

root@Test:/home/testadmin# dd if=/dev/zero of=/swapfile bs=1024 count=512k
524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 20.4811 s, 26.2 MB/s

Now run these two commands to make that file the swap file:

mkswap /swapfile
swapon /swapfile

Let’s check again if the swap file is now working:

root@Test:/home/testadmin# swapon -s
Filename                                Type            Size    Used    Priority
/swapfile                               file            524284  0       -1

Now the swap file is there and turned on.

Now the problem is, when you VM will reboot, the swap will be gone. So, to make the swap permanent, we need to do something more.

nano /etc/fstab

This will open a text editor which should look like this:

Welcome to the world of Text based computing! This is usually a shocker for Windows users. There’s no Notepad guys.

Now, go to the end of the file and add this line:

/swapfile       none    swap    sw      0       0

It will then look like this:

Now here’s the awesome thing. How do you save and exit?

Press Ctrl X, then Y, then ENTER. Yes, it is not Windows. It takes a while to get used to it.

After exiting, run this command:

chown root:root /swapfile 
chmod 0600 /swapfile

This will set proper permission to the swap file.

Now let’s reboot and make sure the swap file is there.

reboot

After the system is back and you have logged back in and run the “sudo su” command to get into root access, run “swapon –s” to check if the swap is working.

You can also use “free” to see how much RAM and swap is getting used. It is a very handy command to check if your server is suffering from memory issue:

It shows swap is there for use.

Install LAMP stack

LAMP means Linux, Apache, Mysql and PHP. That’s the pre-requisite for getting WordPress to run.

Run these two commands:

apt-get install tasksel
tasksel

It will start a Wizard to configure the LAMP server and install all the stuff it needs.

Select LAMP.

You should be asked for password for the MySQL root user account:

Once it is complete, you should have a LAMP stack ready and there will be a webserver running that you can test.

You need to open the HTTP port 80 on your VM from Azure firewall:

Go to Endpoint and create a new endpoint and map port 80.

Wait at least 5 mins after Azure says it is done. I could not hit the site immediately after getting complete message. I had to wait for some time.

Now, the moment of truth:

http://ubuntu23052013.cloudapp.net/

The web folder is at /var/www. This is the equivalent of c:inetpubwwwroot in Windows.

You will find an index.html file there, which is what you see when you browser your server.

Let’s create a test.php file there to make sure PHP is working.

nano test.php

Then put this into the file:

<?php phpInfo() ?>

Ctrl+X, Y, ENTER to get out of nano.

Now hit this file on browser:

http://ubuntu23052013.cloudapp.net/test.php

You should see this:

This means PHP is ready and you are now ready to install WordPress.

Tune MySQL to consume less memory

On an extra small VM, we have very limited RAM. So, we need to tune MySQL to use as little RAM as possible. So, we need to change the mysql config files to make some adjustments.

First, backup the config file.

cp /etc/mysql/my.cnf my-original.cnf

Now let’s make some changes. I usually change the following values:

# * Fine Tuning
#
key_buffer              = 1M
max_allowed_packet      = 1M
thread_stack            = 64K
thread_cache_size       = 8
innodb_buffer_pool_size = 1M

The last entry, innodb_buffer_pool_size may not be there. In that case, you will have to add that line. All of these lines are inside the [mysqld] section. Make sure you put them inside the [mysqld] section and not after any other section.

You need to use nano to open the /etc/mysql/my.cnf file and make these changes.

Install SMTP server

You need to install a SMTP server in order to send email from your server. Postfix is the name of the server. Here’s how to install this:

apt-get install postfix

You should see a config screen:

Then you will be asked to put the fully qualified DNS for your website. If you are finally planning to host your site on a domain like example.com, then put that here. Or you can put the Azure DNS for now and later on add your final domain in the Postfix config file.

Let’s test sending email. First install a mail utility to send email:

apt-get install mailutils

Once it is installed, test sending an email:

echo "Test mail from postfix" | mail -s "Test Postfix" youremail@email.com

You should receive an email. if not, then Gmail doesn’t accept mails from your server. If you aren’t receiving email, then check the last entries from the log file to see what’s wrong:

tail /var/log/mail.log

Configure a website to be served on a domain

Say you have a website, example.com to be served from your server. You need to create something called “Virtual Host” in Apache’s term. It is equivalent to IIS Website.

First create the folder where all the files for the website will reside:

mkdir -p /var/www/example.com/public_html
chown -R $USER:$USER /var/www/example.com/public_html
chmod -R 755 /var/www
cp /etc/apache2/sites-available/default /etc/apache2/sites-available/example.com

The last line creates a sample configuration for the website. Then edit the /etc/apache2/sites-available/example.com file to configure the virtual host with the correct configuration using nano:

nano /etc/apache2/sites-available/example.com

After editing the file will look like this:

First you add the ServerName and ServerAlias. Then you change the DocumentRoot from /var/www to the /var/www/example.com/public_html

Then you change the three occurrences of AllowOverride to AllowOverride All. This will come handy when we will setup a .htaccess file.

Now enable the website:

a2ensite example.com
a2dissite 000-default

Now let’s install some modules that you will need to boost site performance and enable caching.

a2enmod rewrite
a2enmod headers
a2enmod expires

Finally, restart Apache.

service apache2 restart

If you see a warning message like this:

* Restarting web server apache2
 apache2: Could not reliably determine the server's fully qualified domain name, using 100.88.124.9 for ServerName

Then open the apache config file:

nano /etc/apache2/apache2.conf

And add this line:

ServerName example.com

Then try restarting apache again. It should fix the problem.

Let’s verify the site is configured properly. Run this command:

grep -R AllowOverride /etc/apache2

You should see the output should contain these three lines:

/etc/apache2/sites-enabled/example.com:         AllowOverride All
/etc/apache2/sites-enabled/example.com:         AllowOverride All
/etc/apache2/sites-enabled/example.com:         AllowOverride All

This means the site is now enabled (as it is in sites-enabled folder) and the AllowOverride is done properly.

Now we need to create a file on the virtual host (website) and see if the site works:

nano /var/www/example.com/public_html/index.html

Put something there and then hit the URL.

Before hitting URL, you need to put the Public IP of the server in your local PC’s hosts file. In Windows, it is c:WindowsSystem32driversetchosts. You should add something like this:

137.117.135.68    example.com

Now you can hit the http://example.com on browser and it should show:

This means we have got virtual host configured. Now you just have to map your domain to the IP of the VM.

Install WordPress

Now we will install the WordPress engine. First go to your /home/testadmin directory and then run the following commands.

cd /home/testadmin/
wget http://wordpress.org/latest.tar.gz
tar -xzvf latest.tar.gz
cp ./wordpress/wp-config-sample.php ./wordpress/wp-config.php

Now create a MySQL database for wordpress:

mysql -u root -p

Enter the password you used while installing LAMP.

Then run these commands in the MySQL command prompt:

create database wordpress;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, 
  DROP, INDEX, ALTER, CREATE TEMPORARY TABLES, 
  LOCK TABLES ON wordpress.* 
  TO 'wordpress'@'localhost' 
  IDENTIFIED BY 'Wordpress123';
FLUSH PRIVILEGES;

This will create a database named wordpress, then create a database user named wordpress with password WordPress123 and give permission to that wordpress database. To confirm the user can access the database, quit from mysql command prompt and try to login using the wordpress user account:

 mysql -u wordpress -p wordpress

If this works, then you have created the wordpress database and user properly.

Now we need to configure the WordPress config file:

nano ./wordpress/wp-config.php

There are 3 entries to fix:

Ctrl X, Y, ENTER out of nano.

Now copy the whole wordpress folder to the virtual host folder:

cp -r ./wordpress/* /var/www/example.com/public_html/
cd /var/www/
chown www-data:www-data * -R 
usermod -a -G www-data testadmin

In the last line, we have the admin user name. Make sure you have got it right.

Remove the index.html file and your wordpress will be up and running.

rm /var/www/example.com/public_html/index.html

Do the configuration. Change the “admin” user to something else. You will be saved from many brute force attacks.

Tuning wordpress installation

In Apache, there’s something called .htaccess file, where you can define site specific settings like – caching of static files, handling 404, rewriting URL etc.

I have created a .htaccess file that is suitable for wordpress and ensures site produces properly cached static files and handles 404 properly. You can get the file content here:

https://gist.github.com/oazabir/5637290

It also makes WordPress support extensionless URL for posts.

Copy the raw content and then create a .htaccess file in the public_html folder and paste it:

 nano .htaccess

Right click on nano and the content will be pasted. Ctrl X Y ENTER out of nano.

Make sure your wordpress site is still running. If not, then you have incorrectly copied .htaccess file content.

Install some essential WordPress plugins

You should install the following plugins:

• WP Security Scan
• BackWPup
• Configure SMTP
• Facebook
• Jetpack by WordPress.com

Just copy and paste these names into the search box while adding new plugin.

Monitoring traffic on Apache

Apachetop is an awesome tool that shows you traffic hitting your website in near real time. Install it using:

apt-get install apachetop

Then run apachetop and you will see traffic like this:

Checking for missing and failing URLs

You should regularly check for 404 and 500 errors on your website. This will help you identify missing links and correct those links on your WordPress posts. HTTP 500 pages will give you an indication what is failing on your website and whether you are under some attack.

tail -n 10000 /var/log/apache2/access.log | awk '($9 !~ /200|304/)' | awk '{print $9,$7}' | sort | uniq

This will process the last 10k lines from the Apache access log (equivalent to IIS website log) and show you URLS which aren’t 200 and 304. It’s a good starting point to check which URLs are failing and fix them one by one.

Detect site failure and auto restart webserver

You can create a cron (equivalent to Windows Task Scheduler) to run say every 10 minutes and check if your site is still running. If not, it will restart Apache and MySQL.

FILE=/home/siteup.log
SITE=http://quranerkotha.com
SIZE=`stat -c %s $FILE`

# Delete the file if it is over 1MB already
if [ $SIZE -ge 1000000 ]; then
	rm FILE
fi

# IF the site doesn't response within 15 sec, time to kill apache and mysql
if wget --timeout=15 --read-timeout=15 -O /dev/null -q $SITE
then
	echo "Site is up" >> $FILE
	date >> $FILE
else
	echo "Site is down" >> $FILE
	date >> $FILE
	/usr/bin/killall apache2 >> $FILE
	/usr/sbin/service mysql restart >> $FILE
	/usr/sbin/service apache2 restart >> $FILE
	wget --timeout=30 --read-timeout=30 -O /dev/null -q $SITE
fi

Save this in a file inside the /home/testadmin folder. Say the file name is “checksite”.

First, you need to give this file execute permission. In Linux, you can’t just write a batch file like file and run it. You have to give that file execute permission.

chmod +x checksite

Now you can test the file by running ./checksite command.

If it is running fine, then let’s setup a cron which will make this script run every 10 minutes.

Run:

crontab -e

This will open a sample file. Go to the end of the file and put these lines:

SHELL = /bin/bash
PATH = /usr/bin/:/bin:/usr/sbin

*/10 * * * * /home/testadmin/checksite >> $HOME/uptime.log 2>&1

Save the file and exit. Wait for 10 minutes. It should run the script and produce a file in /root/uptime.log folder.

You can view the content of the file using cat command.

cat /root/uptime.log

This will show you if the site was down at some time.

Detect rogue process and kill them automatically

Sometimes there might be some process that keeps consuming 100% CPU all the time and brings down your server. Sometimes it happens to Apache process if you are under attack and the attacker has found a way to hit som expensive page that consumes high CPU. In that case, you will need some automatic way to kill top CPU consuming process and recover the server. Here’s a script to do that:

#!/bin/bash

CPU_LOAD=$(uptime | cut -d"," -f5 | cut -d":" -f2 | cut -d" " -f2 | sed -e "s/$
CPU_THRESHOLD=100

if [ $CPU_LOAD -gt $CPU_THRESHOLD ] ; then
  echo $CPU_LOAD exceeded threshold
  ps -eo pcpu -eo cmd | head -n 5
  kill -9 $(ps -eo pid | sort -k 1 -r | grep -v PID | head -n 1)
  echo Top process killed
  CPU_LOAD=$(uptime | cut -d"," -f4 | cut -d":" -f2 | cut -d" " -f2 | sed -e "s$
  echo CPU load after kill $CPU_LOAD
fi

exit 0

Save this file and use chmod to give it executable permission. Then use crontab –e command to add the following line:

*/10 * * * * /home/testadmin/killtop >> $HOME/killtop.log 2>&1

It will run this script every 10 minutes and the script checks if the server has been consuming 100% CPU on average in last 10 minutes. If it is, then it will find the top CPU consuming process and kill it.

Now the CPU_THRESHOLD value needs some discussion. The “100% utilization” mark is 100 on a single-core system, 200, on a dual-core, 400 on a quad-core, etc. This article has a very nice discussion on this.

Tuning PNG, JPEG for faster site load performance

You can use OptiPNG and JpegTran, two utlities to heavily compress PNG and JPEG files in your website. Sometimes these tools save 70% file size.

First install these tools:

apt-get install opting
apt-get install libjpeg-progs

Now use this command to compress all files inside your web directory. You need to first get into the web directory and run these commands:

cd /var/www/example.com/
find . -type d -exec sh -c "cd "{}" && pwd && optipng -o7 *.png" \;
find . -name '*.jpg' -exec jpegtran -optimize -verbose -outfile {} {} \;

You can save these commands in a script file and run the periodically. They will take a very long time to run, but they might save a lot of download bandwidth on your website and increase your page load performance significantly.

Protecting your website from brute force attack

Someone can write a simple script to continuously hit your virtual server and consume 100% CPU on the server. VMs aren’t as powerful as physical servers. So, they run out of capacity pretty quickly, epecially Extra Small VMs. So, you need to install mod_evasive module in Apache to prevent someone from continuously hitting your website and consume all available system resource.

First install the module:

apt-get install libapache2-mod-evasive

Now you need to create a log folder where the blocked IPs will be logged:

mkdir /var/log/apache2/mod_evasive
chown www-data:www-data /var/log/apache2/mod_evasive

Open the mod_evasive configuration file at

/etc/apache2/mods-enabled/mod-evasive.conf

and put this:

<ifmodule mod_evasive20.c>
   DOSHashTableSize 3097
   DOSPageCount  1
   DOSSiteCount  20
   DOSPageInterval 10
   DOSSiteInterval  10
   DOSBlockingPeriod  60
   DOSLogDir   /var/log/mod_evasive
   DOSEmailNotify notify@email.com
   DOSWhitelist   127.0.0.1
</ifmodule>

This is a *VERY* restrictive setting. It will only allow someone to hit the same page on your blog site once every 10 seconds. I started with the default settings, but my site was going down from some brute force attack or some crawler, which was continuously hitting some page on my site. So, I kept lowering it down to this value. It’s been 3 days. So, far no downtime.

Conclusion

In this article, we have learnt you to setup a Linux VM in Windows Azure. We have also learnt how to install LAMP, configure and tune Apache, MySQL, WordPress. We have learnt some handy tricks to monitor the server, do regular housekeeping and also setup some automated job that can save you from waking up at 3 AM in order to recover your server.