We have covered connecting containers across several hosts previously with plain GRE tunnels and Tinc, which is an awesome little program to build secure mesh networks and VPNs. Today we are going to look at connecting containers across several LXC host with IPSEC. For most users I would recommend Tinc for ease of use, flexibility and performance however there may be some cases where using IPSEC may make sense.

The main difference between IPSEC and Tinc or OpenVPN is ipsec operates in kernel space and thus would be faster and more efficient. But this performance gain can only be realized by pretty good networking equipment with hardware offload with support for encryption. For the average user the difference in performance between Tinc and Ipsec would most likely be minimal.

IPSEC is more widely used and supported across the industry by leading vendors like Cisco, Juniper etc and considered very secure. In Linux IPSEC is supported in the kernel. There are number of tools available to use IPSEC built into the kernel depending on distribution. Openswan and StrongSwan seem  to the more popular ones. For this tutorial we are going to use Openswan on Debian, and we will shortly follow up with Strongswan on Alpine Linux, but the configuration would be similar.

First you need to decide your network topography and how you want to build a vpn. We are going to build a site to site VPN tunnel to connect 2 LXC hosts to give containers on either host access to each other.

We are also going to show you how you can use a container as a VPN endpoint, as in the VPN will be installed in the container rather than the host to connect 2 or more containers across LXC hosts. The use case of connecting containers rather than hosts is limited however its interesting to be able to use IPSEC in containers and there may be a number of ways this could prove useful.

This is a fairly standard config and there are tons of guides online. We are going to try to make it simpler and easier to understand. First let's understand the network and what we are trying to do.

1. Host A is on public IP 1.1.1.1
2. Host A LXC containers are on default subnet 10.0.3.0/24
3. Host B is on public IP 2.2.2.2
4. Host B LXC containers are also on default LXC subnet 10.0.3.0/24

We want to connect containers in Host A and Host B with an IPSEC VPN. Both hosts are on Debian Wheezy and containers are using the default LXC NAT bridge lxcbr0 with subnet 10.0.3.0/24. First let's change the lxc subnet on Host B to 10.0.4.0/24 so there is no clash and the IPSEC tunnel can easily route across hosts.

Next let's change the LXC subnet of Host B to 10.0.4.0/24. To change the default LXC subnet in Debian, edit the /etc/init.d/lxc-net script and change lxc_addr, lxc_network and lxc_dhcp_range values for the new subnet like shown below. If you are on Ubuntu edit the /etc/default/lxc-net script.

USE_LXC_BRIDGE="false"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.4.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.4.0/24"
LXC_DHCP_RANGE="10.0.4.2,10.0.4.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE="/etc/lxc/dnsmasq.conf"
varrun="/var/run/lxc"
LXC_DOMAIN="lxc"

After making the change restart the lxc-net service. Make sure no containers are running before doing this.

service lxc-net restart

So now containers on Host B are on the 10.0.4.0/24 network. Good, let's start configuring the tunnel.

The following steps should be replicated on both Host A and B. Install Openswan.

apt-get install openswan lsof

The installer will offer to generate certificates for you. You can omit this step and generate certificates as required later. The main Openswan config files are /etc/ipsec.conf and /etc/ipsec.secret

Append the following to the ipsec.conf file

conn lxc
#               # Left security gateway, subnet behind it, nexthop toward right.
                authby=secret
                type=tunnel
                keyexchange=ike
                esp=3des-md5
                pfs=no
                left=1.1.1.1
               #leftsourceip=10.0.3.1
                leftid=@hosta
                leftsubnet=10.0.3.0/24
#               leftnexthop=%defaultroute
#               # Right security gateway, subnet behind it, nexthop toward left.
                right=2.2.2.2
                rightid=@hostb
               #rightsourceip=10.0.4.1
                rightsubnet=10.0.4.0/24
#               rightnexthop=10.101.102.103
#               # To authorize this connection, but not actually start it, 
#               # at startup, uncomment this.
                auto=start

Let's look at the whole ipsec.conf file and understand some values.

In the first section of the file uncomment the following 3 configs:

nat_traversal=yes
protostack=auto
plutostderrlog=/var/log/pluto.log

In this particular case we are not using NAT traversal as both our VPN endpoints are public IPs but its ok to leave it on. Protostack refers to the ipsec protocol used. The default in most modern Linux distributions is netkey. It's ok to leave it on auto. The plutostderrlog is the error log file. Its imporrtant to enable this to debug errors.

conn lxc
The value after 'conn' is the name of the VPN connection. This should be identical on both hosts.

authby=secret
This means we are using a secret to authenticate the 2 hosts. You can also use certificates. More on that here.

Type=tunnel
This means we want to build a secure tunnel across the 2 hosts. Ipsec also operates in 'transport' mode. More on that here.

keyexchange, esp and pfs
These configure the encryption to be used. The are a largish number of options here depending on security levels desired. Remember various encryption levels have a cost in processing power required and hence speed of the tunnel. We have used basic options here to show how to make the tunnel work. Please configure encryption as per your requirements. Learn more about this here.

left, leftid and leftsubnet
In ipsec configuration left normally refers to the local host and right to the remote hosts. So 'left' configures the host IP, the host id and the host subnet, in our case the host IP is 1.1.1.1, the host id is @hosta (this can be anything as per your choice) and the host subnet is our lxc container subnet 10.0.3.0/24.

right, rightid and rightsubnet
refers to the remote host details and in our case Host B's IP is 2.2.2.2, id is @hostb and right LXC container subnet is 10.0.4.0/24. Remember we changed it earlier.

autostart
Autostart configures the tunnel connection to autostart with the ipsec daemon. More options on this here.

Repeat the same steps on Host B. The Host B config would look like this. As you would have guessed when configuring Host B the configuration for 'left' will be accordingly be Host B's IP and subnet and 'right' will be Host A IP and subnet like below.

conn lxc
#               # Left security gateway, subnet behind it, nexthop toward right.
                authby=secret
                type=tunnel
                keyexchange=ike
                esp=3des-md5
                pfs=no
                left=2.2.2.2
                leftid=@hostb
               #leftsourceip=10.0.4.1
                leftsubnet=10.0.4.0/24
#               leftnexthop=%defaultroute
#               # Right security gateway, subnet behind it, nexthop toward left.
                right=1.1.1.1
                rightid=@hosta
               #rightsourceip=10.0.3.1
                rightsubnet=10.0.3.0/24
#               rightnexthop=10.101.102.103
#               # To authorize this connection, but not actually start it, 
#               # at startup, uncomment this.
                auto=start

A couple of important points. Host A and B are on public IPS 1.1.1.1 and 2.2.2.2. These are publicly addressable IPs ie can be reached from the internet. Please change these value to reflect your Host A and Host B public IPs.

Now let's set up the /etc/ipsec.secrets file. Append the following to the file first in Host A changing the value in "yoursecret" to a password of your choosing.

@hosta @hostb: PSK "yoursecret"

Then in Host B append the same line but switch the order of @hosta and @hostb and change "yoursecret" to the pasword your chose earlier.

@hostb @hosta: PSK "yoursecret"

One last thing to do. We need to disable send_redirects and accept_redirects on both Hosts.

echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/accept_redirects

To make this permanent edit /etc/sysctl.conf and uncomment the line below

net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

Now run ipsec verify to check if everything is in order. You should not get any errors.

ipsec verify

The moment of truth, now we are now ready to start the tunnel. On both hosts start the ipsec service.

service ipsec start

This should initiate the tunnel on both ends and if everything is working you will be able to ping containers on either side from each other. Please note pings will not work from the gateway itself but will from the containers. To enable this you can uncomment leftsourceip and rightsourceip values in the configs above. The ipsec service automatically sets up routing via netkey so you do not have to set any routes.

The lxc-net script which sets up the lxcbr0 bridge and network sets up masquerading so that containers on the lxcbr0 network can access the internet.

This is the blanket masquerading set up by the lxc-net script

iptables -t nat -A POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE

This blanket masquerading may interfere with the ipsec vpn traffic so you may need to add an iptables rule like below.

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.3.0/24 -d 10.0.4.0/24 -j ACCEPT

If your containers are not able to ping each other that means a configuration error. Check the status of the tunnel with ipsec status command

service ipsec status

This should give you the status of the tunnel. When configured properly you will get a message like below.

1 tunnels up
some eroutes exist.

ipsec auto --status

This will give you more information on the status of the tunnel

The /var/log/pluto.log file is a good place to look for hints on errors as is kernel messages in syslog or dmesg.

The IPSEC netkey module sets up routing automatically with what is know as xfrm tunnels. To check this run

ip xfrm state

This should give you the tunnel route between the 2 public IPs.

To check if encrypted tunnel is working ping one of the remote containers from inside the container from Host A to Host B or use a program like curl to get some data from the containers, or anything to create some traffic while using the tcpdump tool

tcpdump -f esp

If the connectivity is encrypted and traveling through the tunnel your should see some ESP traffic.

In part II of our LXC IPSEC VPN guide we are going to directly connect 2 LXC containers across 2 hosts with a ipsec vpn tunnel. This is a more complex configuration as both containers are in a NAT network behind the hosts and IPSEC depends on kernel modules which are not available in containers unless loaded in the host. The containers are going to be the VPN endpoints and not the host.

More from the Flockport LXC networking series

Connect LXC containers across hosts with IPSEC VPNs

Connect LXC hosts with Tinc VPNs

Connect LXC containers across hosts with Tinc VPNs

Connect LXC hosts with GRE tunnels

LXC networking deep dive - Extending layer 2 across hosts