Secure Ubuntu web server (16.04 xenial) with LEMP stack

Setting up a new web server can be a difficult task. The challenge is to find the right balance between security and usability. Searching online I didn't find any good guides in setting up a new server with the main focus being security.

I have to clarify and quantify what I mean with "secure", because how far will we go in protecting our server? I will try to protect my server against an adversary with a budget lower then $50.000. This is based on the price of a zero day exploit in PHP to gain remote code execution. This tutorial is also based on my own list of principles and attack vectorsPrinciples
1. When a bug is fixed in Ubuntu, my server will be patched in max. 24 hours.
2. Using different users for different tasks.
3. Principle of least privilege: only give a user or application the privileges it needs to execute it's task.
4. Install the least possible applications to narrow the attack surface.
5. Run the least possible services to narrow the attack surface.
6. Activate the least possible features in configs of applications to narrow the attack surface.

Attack vectors
1. Using software which is written with the focus on performance and backwards compatibility instead of security: Linux kernel, Ubuntu, PHP, MariaDB, Nginx.
2. From the moment that a security bug is patched in open source software that I use, it's out in the open an attacker can exploit it until my system is patched. This window of opportunity is inevitable. 3. When my private key is stolen and the passphrase on my private key is stolen, they will have access to my server.
.

This tutorial is meant to be a work in progress. I hope security experts will contribute to this tutorial at the bottom of the article.

I think it's important that every instruction used in this tutorial is explained and that the source is referenced. This way you are not simply copy pasting and hope for the best.

There are two modes in which you can read this tutorial, "beginner" or "expert" (top right switch). In beginner mode every step is explained in detail. If you are an expert user, a lot of this information is hidden from you, so you can go pretty fast through this tutorial.

Starting point

The server is a VPS hosted by a third party and is pre-installed with Ubuntu 16.04 (codename Xenial). This is where we jump-off.

Let's get going, I assume your hosting company gave you a login (I will use "root") and an IP address (In my examples I will use 192.0.2.1) and the server is accessible with SSH. So we start by accessing the server.

Connect to the server

(maybe you are prompted to enter a password, which your hosting company should have provided)

Create a user account

First let's create a user account because we don't want to do everything with the root account. The root account has all privileges on the server and we shouldn't be messing around with it. You don't want to be hanging around at the console as a user with all privileges, because when your account is compromised the attacker can do everything. We are going to introduce an extra layer of security to access privileged commands.

So let's create a user account with name "darius", with the following command.

Now let me explain something else. We will need a passphrase to enter privileged commands. If you didn't already switch to using passphrases instead of passwords, this is the day. Passphrases are stronger and better for humans to remember than the typical passwords we need to choose for a lot of our daily logins with capitals, numbers and special characters crap.

How to create a good passphrase?

You probably can think of a good and strong passphrase right now. Now do you? You've got one right now? Well probably the one that comes in mind that is "easy" to remember isn't the strongest one. The words you chose are probably not really random and have a lot of grammar rules. An attacker can use this to greatly decrease the number of possibilities when trying to brute-force your passphrase.

But don't worry, there is a solution. My favorite one is called diceware. Diceware is a method of creating a passphrase by rolling dice. You will need a list of about 7000 simple words numbered with 5 digits corresponding with 5 dice rolls. Roll the dice 5 times, choose the word you get, and repeat that step at least 6 times so you have 6 random words. Keep this to yourself, this is your fresh and strong passphrase!

Set sudo password

Alright, let's install this passphrase on our server for our sudo privileges. We are still logged in as root so let's give user darius a passphrase.

Logging in with a public/private key pair

Using public/private key pairs offers considerably more protection than using passwords or password lists which can be captured if the client, the server or the secure session is compromised.

We need to login with our account on the remote server via SSH. We just did that by logging in as root using the password that was given to us by our VPS provider, but we are going to replace the password login with a public key login.

Why do I choose public key authentication over password authentication?

1. Your private key can be a lot stronger than any password you can remember (unless you have the brain of Chuck Norris). Choosing a weak password will leave your server weak for bots that brute force attack your server to ultimately gain control.
2. Your password will travel over the network to the server, which makes it possible for interception and decryption by either a man in the middle or a compromised server. Your private key will only be used in your local environment and is never transmitted over the network.
3. You don't need to trust the server you are connecting to that they will treat your key confidential (in contrast to passwords), because you are only sharing your public key.
4. When one server gets compromised, you don't want them to have access to your other servers. With passwords, you should have a different password for every login which is hard to remember. With public keys you are only sharing your public key which may be stolen.

Create a private and public key

You can use the application ssh-keygen which is already installed on your Ubuntu machine. Let's create a private and a public key.

After running this command you will be asked where to save the key (it will default to /home/darius/.ssh/id_rsa which is good) and we are prompted to enter a passphrase. Enter your fresh passphrase!

Get the public key to our server

You have the private and public key locally and we need to copy the public key to the server.

We are still on the server and need to go to our users home directory. I'm making the assumption you are familair with the commands cd, mkdir, touch, chmod. You can concatenate two commands with a double &, with the advantage that the command only runs when the preceeding command is executed successfully

Here we need to create a directory called .ssh and a file in it called authorized_keys with our public key in it. The directory and file may only be readable and writable by you (if you don't set it this strict, the file will not be readFrom the ssh manual:

~/.ssh/authorized_keys Lists the public keys (DSA, ECDSA, Ed25519, RSA) that can be used for logging in as this user. The format of this file is described in the sshd(8) manual page. This file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others. ).

Set the directory and file permissions.

Copy the public key from your local machine and paste it in .ssh/authorized_keys. If this is your first time you can use nano (beginner) to open your files (vi for expert users).

SSH hardening

We are going to tighten the ways to connect to and from this server to another server. We only want to accept logins with private/public key authentication and only first class crypto is allowed here. This is a great source for explanation about the SSH configs. The following command can be run to append some configuration rules to the ssh_config, which is used to connect from your machine to another machine with SSH.

When we and other people connect to this machine we are only going to use our brand new host key and only allow the best crypto algorithms. The sshd_config is the config for the SSH daemon running on our server.

Open up the sshd_config.

You should remove the following two lines.

Append the new settings to the config by running this on the console.

Remove all the old keys and generate new keys with the right algorithms.

Test if the config is working.

Now slap yourself in the face and stay awake, because this is very important. You can lock yourself out from your server if you are not careful. Keep your current SSH logged in session open. Reload the new SSH settings and open a new connection to your server in a new window on your local machine.

Open a new window with a new session and check if you can connect to your server. You should not be asked for a password, but be logged in immediately.

If you can't, check all the settings again and make sure you followed every step.

Disable login with root

We remove the password from the root account. The idea is that we always login with an account with less privileges and for (administrative) tasks that require more privileges we use the "sudo" Sudo is short for "superuser do", which runs an instruction as a superuser (a user with the most privileges). When for example installing new programs we need to run sudo.

When our account get’s compromised for some reason, the adversary is still limited to the privileges our account has, and can not run administrative instructions.

The system logs when sudo is run by a user. When working with multiple users, this is useful when holding users responsible for certain actions.

Source:
Wikipedia command.

Firewall settings

We are going to set our firewall to only allow incoming connections on SSH and reject all incoming traffic. We run the following commands.

Updates and sources list

We can make our server to check automatically for updates on software you install with apt The apt command is a powerful command-line tool, which works with Ubuntu’s Advanced Packaging Tool (APT) performing such functions as installation of new software packages, upgrade of existing software packages, updating of the package list index, and even upgrading the entire Ubuntu system.

Source:
Ubuntu Server Guide .

There is a list of source URL’s where APT will check for updates (called repositories). There are different types of sources you can specify. We like especially free and open software so we only need those repositories. Open the file where all our repositories are specified.

We are going to comment out all sources by adding a # in front of it and simply add these 3 lines at the bottom:

Note that universe is not officially supported by the Ubuntu team, but has lot's and lot's of more applications which we need. It doesn't mean they are not secure, but aren't being checked by the guys from Canonical. Of course you should always be cautious when installing new applications since every extra line of code running on your machine adds to the chance of bugs and security vulnerabilities. Always check if applications aren't too bloated, unsecure or not widely supported.

If you don’t like free and open software and want to include other repositories, you can customize your preferences for your repositories by using the Ubuntu repositories page.

Unused programs, updating and upgrading

We will use the command line tool apt-get to manage our packages. This is a back-end tool that uses APT in the background. Ok, now first let's remove this remote administration tool of canonical (landscape-common). We do this with the purge command. The -y flag means we want to enter "yes" to all questions ("are your sure you...").

We are going to update and upgrade our software. With the following command the list of all applications and versions are retrieved from our repositories.

To upgrade all applications run the following. After that cleanup all libraries, that were dependencies of programs that are now removed. These dependencies are removed with the autoremove command.

Configuring the upgrade process

We are going to configure our server to automatically install security updates. We don't want our server to do other updates because we don't know the consequences. A security update can be considered safe to install, without breaking changes. Turn off installing other recommended software by editing this file.

Change the following line.

Automatically installing updates

We use a package to automatically install new security updates called unattended upgrades.

We configure it to run on low prioirty.

We configure which updates may be upgraded unattended. This is done in the following file.

As we said, only security updates. Furthermore we want to receive an email when our system will be automatically upgraded. Remove unused dependencies. Comment out everything except for this.

Let's configure APT to check periodically for security updates. Open the file.

Set these options.

Now what happens? In the background is periodically checked if there are security updates. If there are some, they will be installed. When a reboot is required, there will be a file created at /var/run/reboot-required. What we are going to do is create a cron job Cron is the name of program that enables unix users to execute commands or scripts (groups of commands) automatically at a specified time/date. Source:
Wikipedia . We add a job to the cron file of the root user.

For rebooting we have two options

1. Reboot automatically every night at a specific time
   Well this is what I use, just reboot at a time I have low traffic on my server. Add the following line to the crontab. 30 4 * * * root if [ -f /var/run/reboot-required ]; then echo ‘security reboot at 04:30’ | mail -s ‘security reboot’ root; /sbin/shutdown -r 04:30 security update; fi
   
2. Send an email to me and I will reboot manually when I see fit
   30 4 * * * root if [ -f /var/run/reboot-required ]; then echo ‘security reboot required’ | mail -s ‘security reboot required’ root;

The time of a cron job is set by the first 5 numbers. This is how it works.

 ┌───────────── min (0–59)
 │ ┌────────────── hour (0–23)
 │ │ ┌─────────────── day of month (1–31)
 │ │ │ ┌──────────────── month (1–12)
 │ │ │ │ ┌───────────────── day of week (0–6) (Sunday to Saturday;
 │ │ │ │ │ 7 is also Sunday)
 │ │ │ │ │
 │ │ │ │ │
 * * * * * command to execute

So when you want to run your cron job every day at 00:30 you will do it this way.

Next you have the user which needs to run the command which is root followed by the command. We use an if statement to check if the file /var/run/reboot-required exists and then echo a text to the system (logged in users will see that). Then we will mail the user root with a subject (-s option). When you have chosen for option 1, then also the program /sbin/shutdown will be run, with options -r, not to shutdown, but to reboot, with a time to schedule the shutdown and a text to echo to logged in users.

Installing time synchronization NTP

We need to install an application to synchronise the server clock. The standard ntpd daemon in Ubuntu is very accurate, but adds a lot of code complexity. I prefer to use openntpd which is less accurate but a lot simpler and considered more secure. When you don't need deadly time precision on this server, I would prefer openntpd.

Configure your time servers.

When you want time servers closest to your servers country, find the addresses here. The following general servers can also be used.

Setting a static DNS server

Your hosting company should provide you with the IP addresses of their DNS resolvers. Those servers are needed by your server in order to transform a domain name to an IP address. You can manually set them once.

Open the follwing file

Add the IP addresses of the resolvers of your hosting company.

We can remove the application that retrieves the DNS servers on startup.

Setting a static IP address

You should have a dedicated IP from your hosting company. We can set this static IP address so we don't need DHCP or anything else to be installed on our server. To set the static ip, we first need to find out what the name is of our primary network interface.

What you see here are all your network adapters. On the most left there is the name. You don't need the lo name (this is a loopback device), but the other one. Probably the name is "ens3" (for Ubuntu 14.04 it was default "eth0"). It is also possible your hosting company already installed a static IP address.

We are going to set a static IP address to this network interface. So we are going to edit the interfaces file.

For every interface there is a configuration and for every interface you can have an IPv4 address and an IPv6 address. We start by creating the following block (you can use 4 spaces as indent).

The start of the file says autoLines beginning with the word "auto" are used to identify the physical interfaces to be brought up when ifup is run with the -a option. (This option is used by the system boot scripts.) Source:
Ubuntu manpage which basically means this network interface is brought up on startup.

All IP addresses should be given to you by your hosting company. The address key is used to assign our own IP address. The netmask is needed to split up the IP address into subnets. The gateway is the IP address which connects you to the internet. Since we have static DNS servers, everything is handled by /etc/resolv.conf and /etc/hosts.

When you have retrieved an IPv6 address from your hoster with a gateway, we like to setup that IP too. Add the following block to the file with your own IPv6 address, netmask and gateway.

Setting a hostname

Of course every little baby needs a name. You can give your server a name on your network. For example you can assign a registered domain name to your server. Let's say we have the domain qqq.is in our possession and want our server to be named "production.qqq.is". It can be done with hostnamectl which will set the /etc/hostname file and reloads it. Run this command.

Then you need to also add this line to your /etc/hosts file. This file is used to link IP addresses to hostnames. So open it.

Add the following line.

Sending emails from the server

You want to make sure your server can properly deliver mail, at least for sending out cron job errors and other system alerts. I prefer to use opensmtpd. It is considerabely easy to install and build with security in mind (that's where OpenBSD is famous for).

In our cron job script we had this line mail -s ‘security reboot required’ root. We should make sure that mail to the root user is delivered to your personal mailbox. To set your e-mail address, edit this file.

You can enter the name of the user, followed by your email address.

Load the aliases table in SMTP with the following command.

That's it.