In part I of this guide we connected LXC hosts with a IPSEC VPN tunnel so containers on each host were part of the VPN. This would be the best way to connect LXC containers across hosts securely. In part II we are going to directly connect 2 LXC containers across 2 hosts with a ipsec vpn tunnel,

Please ensure you have gone through the LXC networking guide so you are familiar with the concepts. This is still work in progress, there have been questions if it is possible to run ipsec in a container but as we will show it works and could be useful for some cases.

This is a more complex configuration as both containers are in a NAT network behind the hosts and IPSEC depends on kernel modules which are not available in containers unless loaded in the host. The containers are going to be the VPN endpoints and not the host.

This is quite possibly the first tutorial connecting containers over an IPSEC VPN. Our hosts and containers are Debian Wheezy. Containers on Host A are on the default LXC subnet while on Host B the subnet has been changed to 10.0.4.0/24. Before we start let's have a look at the network topology.

1. Host A is on public IP 1.1.1.1
2. Host A LXC subnet is 10.0.3.0/24
3. Container A on Host A IP is 10.0.3.15
4. Host B is on public IP 2.2.2.2
5. Host B LXC subnet is on 10.0.4.0/24
6. Container B on Host B IP is 10.0.4.20

We want to connect Container A on Host A with container B on remote Host B over an Ipsec tunnel. Before we can use IPSEC in containers we need to load a number of kernel modules. In both Host A and B.

Load the ipsec modules

modprobe af_key ah4 ipcomp esp xfrm4 xfrm_tunnel tunnel

Now in both Container A and Container B install Openswan and a few needed utilities.

apt-get install Openswan lsof iptables

The installer will offer to create a host certificate for you. You can omit this step and create a certificate later as required.

Next let's change the LXC subnet of Host B to 10.0.4.0/24. To change the default LXC subnet in Debian, edit the /etc/init.d/lxc-net script and change lxc_addr, lxc_network and lxc_dhcp_range values for the new subnet like shown below. If you are on Ubuntu edit the /etc/default/lxc-net script.

USE_LXC_BRIDGE="false"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.4.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.4.0/24"
LXC_DHCP_RANGE="10.0.4.2,10.0.4.254"
LXC_DHCP_MAX="253"
LXC_DHCP_CONFILE="/etc/lxc/dnsmasq.conf"
varrun="/var/run/lxc"
LXC_DOMAIN="lxc"

After making the change restart the lxc-net service. Make sure no containers are running before doing this.

service lxc-net restart

Now in container A append the following to the /etc/ipsec.conf file

conn lxc
#               # Left security gateway, subnet behind it, nexthop toward right.
                authby=secret
                type=tunnel
                keyexchange=ike
                esp=3des-md5
                #pfs=yes
                left=10.0.3.15
                leftid=@containera
                leftsubnet=10.0.3.0/24
#               leftnexthop=%defaultroute
#               # Right security gateway, subnet behind it, nexthop toward left.
                right=2.2.2.2
                rightid=@containerb
                rightsubnet=10.0.4.0/24
#               rightnexthop=10.101.102.103
#               # To authorize this connection, but not actually start it, 
#               # at startup, uncomment this.
                auto=start

Let's look at the  ipsec.conf file and understand some values.

In the first section of the file uncomment the following 3 configs:

nat_traversal=yes
protostack=auto
plutostderrlog=/var/log/pluto.log

In this particular config we are using NAT traversal as both our VPN endpoints are behind public IPs. Notice in ipsec.config the host B switches to the respective host's public IP and not the container IP as that's the only way to make the tunnel work. Protostack refers to the ipsec protocol used. The default in most modern Linux distributions is netkey. It's ok to leave it on auto. The plutostderrlog is the error log file. Its important to enable this to debug errors.

conn lxc
The value after 'conn' is the name of the VPN connection. This should be identical on both hosts.

authby=secret
This means we are using a secret to authenticate the 2 hosts. You can also use certificates. More on that here.

Type=tunnel
This means we want to build a secure tunnel across the 2 hosts. Ipsec also operates in 'transport' mode. More on that here.

keyexchange, esp and pfs
These configure the encryption to be used. The are a largish number of options here depending on security levels desired. Remember various encryption levels have a cost in processing power required and hence speed of the tunnel. We have used basic options here to show how to make the tunnel work. Please configure encryption as per your requirements. Learn more about this here.

left, leftid and leftsubnet
In ipsec configuration left normally refers to the local host and right to the remote hosts. So 'left' configures the host IP, the host id and the host subnet, in our case the left host IP is container IP 10.0.3.15, the host id is @containera (this can be anything as per your choice) and the host subnet is our lxc container subnet 10.0.3.0/24.

right, rightid and rightsubnet
This refers to the remote host details and in our case Host B's IP will NOT be container B IP 10.0.4.20 but Host's B public IP 2.2.2.2, id is @containerb and right LXC container subnet is 10.0.4.0/24. Remember we changed it earlier.

autostart
Autostart configures the tunnel connection to autostart with the ipsec daemon. More options on this here.

Repeat the same steps on Host B. The Host B config would look like this. As you would have guessed when configuring Host B the configuration for 'left' will be accordingly be Host B's container IP and subnet and 'right' will be Host A public IP and subnet like below.

conn lxc
#               # Left security gateway, subnet behind it, nexthop toward right.
                authby=secret
                type=tunnel
                keyexchange=ike
                esp=3des-md5
                #pfs=yes
                left=10.0.4.20
                leftid=@containerb
                leftsubnet=10.0.4.0/24
#               leftnexthop=%defaultroute
#               # Right security gateway, subnet behind it, nexthop toward left.
                right=1.1.1.1
                rightid=@containera
                rightsubnet=10.0.3.0/24
#               rightnexthop=10.101.102.103
#               # To authorize this connection, but not actually start it, 
#               # at startup, uncomment this.
                auto=start

A couple of important points. We are building a VPN between containers with private IPs behind hosts with public IPs. Host A and B are on public IPS 1.1.1.1 and 2.2.2.2. Container A and B are private IPs 10.0.3.15 and 10.0.4.20 respectively. We are thus going to have to port forward the ipsec udp ports 500 and 4500 from the Host public IP to the container private IP to make the tunnel work. You need to do this on both containers. So on host A the iptables command to port forward udp port 500 and 4500 will look like below.

For Host A

iptables -t nat -D PREROUTING -p UDP -d 1.1.1.1 --dport 4500 -j DNAT --to 10.0.3.15:4500
iptables -t nat -D PREROUTING -p UDP -d 1.1.1.1 --dport 500 -j DNAT --to 10.0.3.15:500

For Host B

iptables -t nat -D PREROUTING -p UDP -d 2.2.2.2 --dport 4500 -j DNAT --to 10.0.4.20:4500
iptables -t nat -D PREROUTING -p UDP -d 2.2.2.2 --dport 500 -j DNAT --to 10.0.4.20:500

Please change the values of these configs to your own host and container IPs.

Now let's set up the /etc/ipsec.secrets file. Append the following to the file first in container A changing the value in "yoursecret" to a password of your choosing.

@containera @containerb: PSK "yoursecret"

Then in container B append the same line but switch the order of @containera and @containerb and change "yoursecret" to the pasword your chose earlier.

@containerb @containera: PSK "yoursecret"

One last thing to do. We need to disable send_redirects and accept_redirects on both containers.

echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/accept_redirects

To make this permanent edit /etc/sysctl.conf and uncomment the line below

net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0

Now run ipsec verify to check if everything is in order. You should not get any errors.

ipsec verify

If your containers are not able to ping each other that means a configuration error. Check the status of the tunnel with ipsec status command

service ipsec status

This should give you the status of the tunnel. When configured properly you will get a message like below.

1 tunnels up
some eroutes exist.

ipsec auto --status

This will give you more information on the status of the tunnel

The /var/log/pluto.log file is a good place to look for hints on errors as is kernel messages in syslog or dmesg.

The IPSEC netkey module sets up routing automatically with what is know as xfrm tunnels. To check this run

ip xfrm state

This should give you the tunnel route between the 2 public IPs.

To check if encrypted tunnel is working ping one of the remote containers from inside the container from Host A to Host B or use a program like curl to get some data from the containers, or anything to create some traffic while using the tcpdump tool

tcpdump -f esp

If the connectivity is encrypted and traveling through the tunnel your should see some ESP traffic.

Since we are in containers make sure the kernel modules are loaded on the host

modprobe af_key ah4 ipcomp esp xfrm4 xfrm_tunnel tunnel

You may need to set up additional routes or iptables rules depending on your networking environment.

More from the Flockport LXC networking series

Connect LXC hosts with IPSEC VPNs

Connect LXC hosts with Tinc VPNs

Connect LXC containers across hosts with Tinc VPNs

Connect LXC hosts with GRE tunnels

LXC networking deep dive - Extending layer 2 across hosts