Thorough Tutorials

Read modes

Secure Ubuntu web server (16.04 xenial) with LEMP stack

Setting up a new web server can be a difficult task. The challenge is to find the right balance between security and usability. Searching online I didn't find any good guides in setting up a new server with the main focus being security.

I have to clarify and quantify what I mean with "secure", because how far will we go in protecting our server? I will try to protect my server against an adversary with a budget lower then $50.000. This is based on the price of a zero day exploit in PHP to gain remote code execution. This tutorial is also based on my own list of principles and attack vectorsPrinciples
1. When a bug is fixed in Ubuntu, my server will be patched in max. 24 hours.
2. Using different users for different tasks.
3. Principle of least privilege: only give a user or application the privileges it needs to execute it's task.
4. Install the least possible applications to narrow the attack surface.
5. Run the least possible services to narrow the attack surface.
6. Activate the least possible features in configs of applications to narrow the attack surface.

Attack vectors
1. Using software which is written with the focus on performance and backwards compatibility instead of security: Linux kernel, Ubuntu, PHP, MariaDB, Nginx.
2. From the moment that a security bug is patched in open source software that I use, it's out in the open an attacker can exploit it until my system is patched. This window of opportunity is inevitable. 3. When my private key is stolen and the passphrase on my private key is stolen, they will have access to my server.
.

This tutorial is meant to be a work in progress. I hope security experts will contribute to this tutorial at the bottom of the article.

I think it's important that every instruction used in this tutorial is explained and that the source is referenced. This way you are not simply copy pasting and hope for the best.

There are two modes in which you can read this tutorial, "beginner" or "expert" (top right switch). In beginner mode every step is explained in detail. If you are an expert user, a lot of this information is hidden from you, so you can go pretty fast through this tutorial.

Starting point

The server is a VPS hosted by a third party and is pre-installed with Ubuntu 16.04 (codename Xenial). This is where we jump-off.

Let's get going, I assume your hosting company gave you a login (I will use "root") and an IP address (In my examples I will use 192.0.2.1) and the server is accessible with SSH. So we start by accessing the server.

Connect to the server

ssh root@192.0.2.1OpenSSH SSH client (remote login program) SSH is used to connect to a remote server. Via the command line you can specificy which user and IP address you would like to use. To specify a user prefix the IP address with the username follow by a @. When not specifying a user, the username of your local system is used. We used root. 192.0.2.1 is the IP address of the remote server Source:
Ubuntu manpage

(maybe you are prompted to enter a password, which your hosting company should have provided)

Create a user account

First let's create a user account because we don't want to do everything with the root account. The root account has all privileges on the server and we shouldn't be messing around with it. You don't want to be hanging around at the console as a user with all privileges, because when your account is compromised the attacker can do everything. We are going to introduce an extra layer of security to access privileged commands.

So let's create a user account with name "darius", with the following command.

useradd -mUG sudo,adm -s /bin/bash dariususeradd - create a new user or update default new user information Let me explain what each option does: -m create a home dir for this user in /home/darius -G add the user to the groups "sudo" and "adm" -U create a group with your username (this will be useful when setting directory permissions) -s set the default shell to /bin/bash darius the name of the new user Source:
Ubuntu manpage

Now let me explain something else. We will need a passphrase to enter privileged commands. If you didn't already switch to using passphrases instead of passwords, this is the day. Passphrases are stronger and better for humans to remember than the typical passwords we need to choose for a lot of our daily logins with capitals, numbers and special characters crap.

How to create a good passphrase?

You probably can think of a good and strong passphrase right now. Now do you? You've got one right now? Well probably the one that comes in mind that is "easy" to remember isn't the strongest one. The words you chose are probably not really random and have a lot of grammar rules. An attacker can use this to greatly decrease the number of possibilities when trying to brute-force your passphrase.

But don't worry, there is a solution. My favorite one is called diceware. Diceware is a method of creating a passphrase by rolling dice. You will need a list of about 7000 simple words numbered with 5 digits corresponding with 5 dice rolls. Roll the dice 5 times, choose the word you get, and repeat that step at least 6 times so you have 6 random words. Keep this to yourself, this is your fresh and strong passphrase!

Set sudo password

Alright, let's install this passphrase on our server for our sudo privileges. We are still logged in as root so let's give user darius a passphrase.

passwd dariuspasswd - change user password The passwd command changes passwords for user accounts. A normal user may only change the password for his/her own account, while the superuser may change the password for any account. The passwords you enter are hashed and saved in /etc/shadow Source:
Ubuntu manpage

Logging in with a public/private key pair

Using public/private key pairs offers considerably more protection than using passwords or password lists which can be captured if the client, the server or the secure session is compromised.

We need to login with our account on the remote server via SSH. We just did that by logging in as root using the password that was given to us by our VPS provider, but we are going to replace the password login with a public key login.

Why do I choose public key authentication over password authentication?

  1. Your private key can be a lot stronger than any password you can remember (unless you have the brain of Chuck Norris). Choosing a weak password will leave your server weak for bots that brute force attack your server to ultimately gain control.
  2. Your password will travel over the network to the server, which makes it possible for interception and decryption by either a man in the middle or a compromised server. Your private key will only be used in your local environment and is never transmitted over the network.
  3. You don't need to trust the server you are connecting to that they will treat your key confidential (in contrast to passwords), because you are only sharing your public key.
  4. When one server gets compromised, you don't want them to have access to your other servers. With passwords, you should have a different password for every login which is hard to remember. With public keys you are only sharing your public key which may be stolen.

Create a private and public key

You can use the application ssh-keygen which is already installed on your Ubuntu machine. Let's create a private and a public key.

ssh-keygen -b 4096ssh-keygen - authentication key generation, management and conversion Default ssh-keygen creates an RSA private key. -b 4096 The size of the private key in bits. Source:
Ubuntu manpage

After running this command you will be asked where to save the key (it will default to /home/darius/.ssh/id_rsa which is good) and we are prompted to enter a passphrase. Enter your fresh passphrase!

Get the public key to our server

You have the private and public key locally and we need to copy the public key to the server.

We are still on the server and need to go to our users home directory. I'm making the assumption you are familair with the commands cd, mkdir, touch, chmod. You can concatenate two commands with a double &, with the advantage that the command only runs when the preceeding command is executed successfully

cd /home/darius

Here we need to create a directory called .ssh and a file in it called authorized_keys with our public key in it. The directory and file may only be readable and writable by you (if you don't set it this strict, the file will not be readFrom the ssh manual:

~/.ssh/authorized_keys Lists the public keys (DSA, ECDSA, Ed25519, RSA) that can be used for logging in as this user. The format of this file is described in the sshd(8) manual page. This file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others.
).

mkdir .ssh && touch .ssh/authorized_keys

Set the directory and file permissions.

chmod 700 .ssh && chmod 600 .ssh/authorzied_keys

Copy the public key from your local machine and paste it in .ssh/authorized_keys. If this is your first time you can use nano (beginner) to open your files (vi for expert users).

nano .ssh/authorized_keys

SSH hardening

We are going to tighten the ways to connect to and from this server to another server. We only want to accept logins with private/public key authentication and only first class crypto is allowed here. This is a great source for explanation about the SSH configs. The following command can be run to append some configuration rules to the ssh_config, which is used to connect from your machine to another machine with SSH.

cat /etc/ssh/ssh_config >> <<EOF HashKnownHosts yes Host * ConnectTimeout 30 KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr ServerAliveInterval 10 EOFcat is used to read content from a file. The >> operator is used to append content to the end of a file. With <<EOF and EOF we can work with multiline strings and don't need all kinds of escaping of the special characters in our content. The following options are used to harden our config:
HashKnownHosts
Your known host file is the file where all outgoing hosts are stored with their address. When we hash this file, an attacker can't see to which other machines we connected with SSH. Your tab completeion for hosts will not work though.
Host
Restrict to which hosts this config is applied. When using '*' as pattern, this config is used as default for all hosts.
ConnectTimeout

Specifies the timeout (in seconds) used when connecting to the SSH server, instead of using the default system TCP timeout.

KexAlgorithms
These are the host key algorithms used to do the key exchange between two hosts.
MACs

Specifies the MAC (message authentication code) algorithms in order of preference. The MAC algorithm is used for data integrity protection.

Ciphers

Symmetric ciphers are used to encrypt the data after the initial key exchange and authentication is complete.

ServerAliveInterval

Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server.

When we and other people connect to this machine we are only going to use our brand new host key and only allow the best crypto algorithms. The sshd_config is the config for the SSH daemon running on our server.

Open up the sshd_config.

nano /etc/ssh/sshd_config

You should remove the following two lines.

HostKey /etc/ssh/ssh_host_dsa_key HostKey /etc/ssh/ssh_host_ecdsa_key

Append the new settings to the config by running this on the console.

cat /etc/ssh/sshd_config >> <<EOF Protocol 2 HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_rsa_key KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com PermitRootLogin no PasswordAuthentication no X11Forwarding no ChallengeResponseAuthentication no KbdInteractiveAuthentication no EOF cat is used to read content from a file. The >> operator is used to append content to the end of a file. With <<EOF and EOF we can work with multiline strings and don't need all kinds of escaping of the special characters in our content.
HostKey
The host is our server and this way we tell the SSH daemon to only use these keys when other machines connect to us.
PermitRootLogin
We don't want anyone logging in directly with the root account.
PasswordAuthentication
We don't want anyone logging in with a password. Only private/public keys.
X11Forwarding

The X Window System (X11, or shortened to simply X) is a windowing system for bitmap displays, common on UNIX-like computer operating systems.

I assume you did not install a desktop version, so we don't need this.

ChallengeResponseAuthentication
This is login via one time passwords. We also don't want that.
KbdInteractiveAuthentication

Specifies whether to allow keyboard-interactive authentication.

The rest of the options are already explained for the SSH config.

Remove all the old keys and generate new keys with the right algorithms.

cd /etc/ssh rm ssh_host_* ssh-keygen -t ed25519 -f ssh_host_ed25519_key -N "" < /dev/null ssh-keygen -t rsa -b 4096 -f ssh_host_rsa_key -N "" < /dev/null

Test if the config is working.

sudo sshd -t

Now slap yourself in the face and stay awake, because this is very important. You can lock yourself out from your server if you are not careful. Keep your current SSH logged in session open. Reload the new SSH settings and open a new connection to your server in a new window on your local machine.

service ssh reload

Open a new window with a new session and check if you can connect to your server. You should not be asked for a password, but be logged in immediately.

ssh darius@192.0.2.1

If you can't, check all the settings again and make sure you followed every step.

Disable login with root

We remove the password from the root account. The idea is that we always login with an account with less privileges and for (administrative) tasks that require more privileges we use the "sudo" Sudo is short for "superuser do", which runs an instruction as a superuser (a user with the most privileges). When for example installing new programs we need to run sudo.

When our account get’s compromised for some reason, the adversary is still limited to the privileges our account has, and can not run administrative instructions.

The system logs when sudo is run by a user. When working with multiple users, this is useful when holding users responsible for certain actions.

Source:
Wikipedia
command.

sudo passwd -l root

Firewall settings

We are going to set our firewall to only allow incoming connections on SSH and reject all incoming traffic. We run the following commands.

sudo ufw logging off sudo ufw default reject incoming sudo ufw limit OpenSSH sudo ufw enable

Updates and sources list

We can make our server to check automatically for updates on software you install with apt The apt command is a powerful command-line tool, which works with Ubuntu’s Advanced Packaging Tool (APT) performing such functions as installation of new software packages, upgrade of existing software packages, updating of the package list index, and even upgrading the entire Ubuntu system.

Source:
Ubuntu Server Guide
.

There is a list of source URL’s where APT will check for updates (called repositories). There are different types of sources you can specify. We like especially free and open software so we only need those repositories. Open the file where all our repositories are specified.

sudo nano /etc/apt/sources.list

We are going to comment out all sources by adding a # in front of it and simply add these 3 lines at the bottom:

deb http://archive.ubuntu.com/ubuntu xenial main universe deb http://archive.ubuntu.com/ubuntu xenial-updates main universe deb http://security.ubuntu.com/ubuntu xenial-security main universedeb These repositories contain binaries or precompiled packages. These repositories are required for most users. http://archive.ubuntu.com/ubuntu The URI (Uniform Resource Identifier), in this case a location on the internet. See the official mirror list or the self-maintained mirror list to find other mirrors. xenial is the release name or version of your distribution main universe This are the section names or components. There can be several section names, separated by spaces. main The main component contains applications that are free software, can be freely redistributed and are fully supported by the Ubuntu team. universe The universe component is a snapshot of the free, open-source, and Linux world. It houses almost every piece of open-source software, all built from a range of public sources. Canonical does not provide a guarantee of regular security updates for software in the universe component, but will provide these where they are made available by the community. Users should understand the risk inherent in using these packages. Popular or well supported pieces of software will move from universe into main if they are backed by maintainers willing to meet the standards set by the Ubuntu team. Source:
Ubuntu Repositories/CommandLine

Note that universe is not officially supported by the Ubuntu team, but has lot's and lot's of more applications which we need. It doesn't mean they are not secure, but aren't being checked by the guys from Canonical. Of course you should always be cautious when installing new applications since every extra line of code running on your machine adds to the chance of bugs and security vulnerabilities. Always check if applications aren't too bloated, unsecure or not widely supported.

If you don’t like free and open software and want to include other repositories, you can customize your preferences for your repositories by using the Ubuntu repositories page.

Unused programs, updating and upgrading

We will use the command line tool apt-get to manage our packages. This is a back-end tool that uses APT in the background. Ok, now first let's remove this remote administration tool of canonical (landscape-common). We do this with the purge command. The -y flag means we want to enter "yes" to all questions ("are your sure you...").

apt-get -y purge landscape-common

We are going to update and upgrade our software. With the following command the list of all applications and versions are retrieved from our repositories.

apt-get update

To upgrade all applications run the following. After that cleanup all libraries, that were dependencies of programs that are now removed. These dependencies are removed with the autoremove command.

apt-get -y dist-upgrade apt-get -y autoremove

Configuring the upgrade process

We are going to configure our server to automatically install security updates. We don't want our server to do other updates because we don't know the consequences. A security update can be considered safe to install, without breaking changes. Turn off installing other recommended software by editing this file.

sudo nano /etc/apt/apt.conf.d/00InstallRecommends

Change the following line.

APT::Install-Recommends “false”;

Automatically installing updates

We use a package to automatically install new security updates called unattended upgrades.

sudo apt-get install unattended-upgrades

We configure it to run on low prioirty.

sudo dpkg-reconfigure --priority=low unattended-upgradesSource:
Automatic Security Updates - Ubuntu Community Help

We configure which updates may be upgraded unattended. This is done in the following file.

nano /etc/apt/apt.conf.d/50unattended-upgrades

As we said, only security updates. Furthermore we want to receive an email when our system will be automatically upgraded. Remove unused dependencies. Comment out everything except for this.

// Automatically upgrade packages from these (origin:archive) pairs Unattended-Upgrade::Allowed-Origins { “${distro_id}:${distro_codename}”; “${distro_id}:${distro_codename}-security”; // “${distro_id}:${distro_codename}-updates”; // “${distro_id}:${distro_codename}-proposed”; // “${distro_id}:${distro_codename}-backports”; }; Unattended-Upgrade::Mail "root"; Unattended-Upgrade::Remove-Unused-Dependencies "true";Unattended-Upgrade::Mail To which user should we sent mail about unattended upgrades? Unattended-Upgrade::Remove-Unused-Dependencies Like the name says, remove unused dependencies

Let's configure APT to check periodically for security updates. Open the file.

nano etc/apt/apt.conf.d/10periodic

Set these options.

APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Download-Upgradeable-Packages "0"; APT::Periodic::AutocleanInterval "7"; APT::Periodic::Unattended-Upgrade "1";APT::Periodic::Update-Package-Lists Update the package list, yes ofcourse APT::Periodic::Download-Upgradeable-Packages Download all packages that could be upgraded. We don't need that since we only need security updates, not other updates. APT::Periodic::AutocleanInterval When is the package download list cleaned up? Every week please. APT::Periodic::Unattended-Upgrade Boolean to turn on unattended upgrades.

Now what happens? In the background is periodically checked if there are security updates. If there are some, they will be installed. When a reboot is required, there will be a file created at /var/run/reboot-required. What we are going to do is create a cron job Cron is the name of program that enables unix users to execute commands or scripts (groups of commands) automatically at a specified time/date. Source:
Wikipedia
. We add a job to the cron file of the root user.

nano /etc/crontab

For rebooting we have two options

  1. Reboot automatically every night at a specific time
    Well this is what I use, just reboot at a time I have low traffic on my server. Add the following line to the crontab. 30 4 * * * root if [ -f /var/run/reboot-required ]; then echo ‘security reboot at 04:30’ | mail -s ‘security reboot’ root; /sbin/shutdown -r 04:30 security update; fi
  2. Send an email to me and I will reboot manually when I see fit
    30 4 * * * root if [ -f /var/run/reboot-required ]; then echo ‘security reboot required’ | mail -s ‘security reboot required’ root;

The time of a cron job is set by the first 5 numbers. This is how it works.

 ┌───────────── min (0–59)
 │ ┌────────────── hour (0–23)
 │ │ ┌─────────────── day of month (1–31)
 │ │ │ ┌──────────────── month (1–12)
 │ │ │ │ ┌───────────────── day of week (0–6) (Sunday to Saturday;
 │ │ │ │ │ 7 is also Sunday)
 │ │ │ │ │
 │ │ │ │ │
 * * * * * command to execute

So when you want to run your cron job every day at 00:30 you will do it this way.

30 0 * * *

Next you have the user which needs to run the command which is root followed by the command. We use an if statement to check if the file /var/run/reboot-required exists and then echo a text to the system (logged in users will see that). Then we will mail the user root with a subject (-s option). When you have chosen for option 1, then also the program /sbin/shutdown will be run, with options -r, not to shutdown, but to reboot, with a time to schedule the shutdown and a text to echo to logged in users.

Installing time synchronization NTP

We need to install an application to synchronise the server clock. The standard ntpd daemon in Ubuntu is very accurate, but adds a lot of code complexity. I prefer to use openntpd which is less accurate but a lot simpler and considered more secure. When you don't need deadly time precision on this server, I would prefer openntpd.

apt-get -y install openntpd

Configure your time servers.

nano /etc/openntpd/ntpd.conf

When you want time servers closest to your servers country, find the addresses here. The following general servers can also be used.

server 0.pool.ntp.org server 1.pool.ntp.org server 2.pool.ntp.org server 3.pool.ntp.org

Setting a static DNS server

Your hosting company should provide you with the IP addresses of their DNS resolvers. Those servers are needed by your server in order to transform a domain name to an IP address. You can manually set them once.

Open the follwing file

nano /etc/resolv.conf

Add the IP addresses of the resolvers of your hosting company.

nameserver 3.4.5.6 nameserver 3.4.5.7

We can remove the application that retrieves the DNS servers on startup.

apt-get -y purge resolvconf apt-get -y autoremove

Setting a static IP address

You should have a dedicated IP from your hosting company. We can set this static IP address so we don't need DHCP or anything else to be installed on our server. To set the static ip, we first need to find out what the name is of our primary network interface.

ifconfigIfconfig is used to configure the kernel-resident network interfaces. It is used at boot time to set up interfaces as necessary. After that, it is usually only needed when debugging or when system tuning is needed Source:
Ubuntu manpage

What you see here are all your network adapters. On the most left there is the name. You don't need the lo name (this is a loopback device), but the other one. Probably the name is "ens3" (for Ubuntu 14.04 it was default "eth0"). It is also possible your hosting company already installed a static IP address.

We are going to set a static IP address to this network interface. So we are going to edit the interfaces file.

nano /etc/network/interfaces

For every interface there is a configuration and for every interface you can have an IPv4 address and an IPv6 address. We start by creating the following block (you can use 4 spaces as indent).

# The primary network interface auto ens3 iface ens3 inet static address 192.0.2.1 netmask 255.255.255.0 gateway 192.0.2.100

The start of the file says autoLines beginning with the word "auto" are used to identify the physical interfaces to be brought up when ifup is run with the -a option. (This option is used by the system boot scripts.) Source:
Ubuntu manpage
which basically means this network interface is brought up on startup.

All IP addresses should be given to you by your hosting company. The address key is used to assign our own IP address. The netmask is needed to split up the IP address into subnets. The gateway is the IP address which connects you to the internet. Since we have static DNS servers, everything is handled by /etc/resolv.conf and /etc/hosts.

When you have retrieved an IPv6 address from your hoster with a gateway, we like to setup that IP too. Add the following block to the file with your own IPv6 address, netmask and gateway.

iface ens3 inet6 static address 2a02:0202:02:0:0:0202:0202:0202 netmask 64 gateway 2a02:0202:02::1

Setting a hostname

Of course every little baby needs a name. You can give your server a name on your network. For example you can assign a registered domain name to your server. Let's say we have the domain qqq.is in our possession and want our server to be named "production.qqq.is". It can be done with hostnamectl which will set the /etc/hostname file and reloads it. Run this command.

hostnamectl set-hostname production

Then you need to also add this line to your /etc/hosts file. This file is used to link IP addresses to hostnames. So open it.

nano /etc/hosts

Add the following line.

192.0.2.1 production.qqq.is production

Sending emails from the server

You want to make sure your server can properly deliver mail, at least for sending out cron job errors and other system alerts. I prefer to use opensmtpd. It is considerabely easy to install and build with security in mind (that's where OpenBSD is famous for).

apt-get -y install opensmtpd

In our cron job script we had this line mail -s ‘security reboot required’ root. We should make sure that mail to the root user is delivered to your personal mailbox. To set your e-mail address, edit this file.

nano /etc/aliases

You can enter the name of the user, followed by your email address.

root: my@emailaddress.is

Load the aliases table in SMTP with the following command.

smtpctl update table aliasessmtpctl: control the Simple Mail Transfer Protocol daemon update table Guess what, this updates a table aliases The name of the table Source:
OpenBSD manpage

That's it.