LXC containers, like KVM and other virtual machine systems, provides a number of networking options from NAT, bridged mode to assigning static IPs. We cover this in the LXC networking guide.

In this article we are going to look at options for multi host connectivity. For those who already connect multiple VM hosts it's the same for LXC containers, there are no special requirements. Containers behave like VMs, and you can continue to use those options.

Typically in your own network, connecting multiple LXC or VM hosts is easily done by bridging VMs/containers network directly to the LXC hosts physical network connected to a router/switch. This way all containers are on the same network. The network can be managed at a higher level than the LXC/VM hosts, and it handles IPs, DHCP, subnets and inter host connectivity.

Alternatively if LXC hosts are on the same network you can also simply add a route if you do not wish to bridge the containers to the Host network. But for this to work the LXC subnets ie 10.0.3.0/24 on each LXC host must be different.

For instance if LXC Host 1 is IP 192.168.1.2 and LXC Host 2 is IP 192.168.1.3, containers on Host 2 can be changed to operate on subnet 10.0.4.0/24. Then the containers across the 2 hosts can access each other by a simple routing rule on each host.

On Host 1
ip route add 10.0.4.0/24 via 192.168.1.3

On Host 2
ip route add 10.0.3.0/24 via 192.168.1.2

Now containers on both Hosts 1 and 2 should be able to ping each other.

Connecting multiple LXC hosts in the cloud

A lot of users in the cloud may not have the same freedom or choice. Users may have dedicated servers or KVM instances with little control of the networking layer. How do you connect LXC containers across multiple LXC hosts?

Here we are going to look at 4 ways to create a private network of LXC hosts across networks in the cloud.

GRE tunnel
This is the simplest solution. A GRE tunnel connects 2 LXC hosts and lets containers on both sides connect to each other. You can build multiple GRE tunnels to connect more hosts.

Tinc VPN
Tinc is a fantastic little program to create private networks. We will use Tinc to let containers on both sides talk to each other. The improvement over a plain jane GRE tunnel is Tinc provides an encrypted connection. Tinc is a bit special and among the Internet's best kept secrets. It's a wonderful tool to create distributed private mesh networks with tremendous flexibility. Unlike other VPN solutions Tinc is not a hub and spoke model with single choke or failure points. And we are just scratching the surface here. Tinc affirms the old adage; simplicity works, simplicity scales.

Tinc container to container connection
We use Tinc to connect 2 containers behind NAT networks across the Internet. This is unique because both containers are behind a NAT and we try not to touch the host networking too much. This is a mesh network enabled in the containers networking space and the only setting on the host is a single port forward of tcp/udp port 655 on one of the hosts, to a container. In this scenario containers will have 2 IPs, the container's basic lxcbr0 NAT network and the Tinc vpn network

Open vSwitch (OVS) bridges connected by GRE tunnels
LXC supports multiple virtual network cards. You can create a new bridge interface and create a second network for your LXC containers. You can create a second OVS bridge on LXC hosts and connect the bridges with GRE tunnels. We are going to a link to a guide by Serge Hallyn, LXC lead developer and another by Scott Lowe. OVS is what the networking gurus uses to build networks with things like openflow, vxlan and other fancy functionality. Open vSwitch is a major component of the Openstack networking layer. Option 1,2,3 are easier to setup and use but for more advanced functionality OVS is a great choice.

A GRE tunnel is a simple point to point IP tunnel connecting 2 public IPs and networks. But it's not encrypted. For private networks across the public Internet, encryption is often standard but if your traffic does not have high security requirements a GRE tunnel is a simple just works solution.

Setting it up
Connecting 2 LXC hosts with a GRE tunnel will enable your LXC containers to access each other. LXC containers will be using their default NAT bridge lxcbr0.

We are going to change the subnet provided by lxcbr0 on one side which is normally 10.0.3.0/24 to 10.0.4.0/24 so there is no clash of IPs on either side and we can route both ways.

So here is the network map. We are going to call our GRE tunnel 'neta' on Host A and 'netb' on Host B (You can call this anything you want)

Host A has public IP 1.2.3.4
Host B has public IP 2.3.4.5
Containers in Host A are on subnet 10.0.4.0/24 via default lxcbr0 nat bridge
Containers in Host B are on subnet 10.0.3.0/24 via default lxcbr0 nat bridge

Change the subnet on Host A
To change the subnet edit the /etc/init.d/lxc-net script. Change the subnet entries from 10.0.3.0/24 to 10.0.4.0/24.

Create the GRE tunnel on Host A and B
ip tunnel add neta mode gre remote 2.3.4.5 local 1.2.3.4 ttl 255
ip tunnel add netb mode gre remote 1.2.3.4 local 2.3.4.5 ttl 255

On Host A

ip link set neta up
ip addr add 10.0.4.254 neta
ip route add 10.0.3.0/24 dev neta

On Host B

ip link set netb up
ip addr add 10.0.3.254 netb
ip route add 10.0.4.0/24 dev netb

Congratulations! You spanking new GRE tunnel is up and your containers on both sides can ping each other. You can do a traceroute and you will notice the IP address we added to the tunnel on each side 10.0.3/4.254 is being used as the gateway to reach either side. This is a random link IP, you can use anything 10.0.0.1/2 for instance.

To remove the tunnel

ip link set netb down
ip tunnel del netb

Tinc is more secure than a GRE tunnel as connections are encrypted. Tinc also gives you the ability to built a distributed mesh network. We are not covering Tinc's extensive capabilities here, please visit tinc-vpn.org

For the networking gurus Tinc can operate as a router in layer 3 or a switch in layer 2 mode, for this example we are using Tinc in its default router mode.

To avoid container IP clash we are going to change the default lxcbr0 subnet 10.0.3.0/24 on one side, let's do it on Host A

Change the subnet on Host A
Edit the /etc/init.d/lxc-net script to change the LXC subnet on lxcbr0 NAT network from 10.0.3.0/24 to 10.0.4.0/24. Before doing this stop containers on Host A, stop the lxc-net service, make the change and then restart the lxc-net service.

service lxc-net stop

Edit the lxc-net script

service lxc-net start

So here is the network map.

Host A has public IP 1.2.3.4
Host B has public IP 2.3.4.5
Containers in Host A are on subnet 10.0.4.0/24 via default lxcbr0 nat bridge
Containers in Host B are on subnet 10.0.3.0/24 via default lxcbr0 nat bridge

We are going to use 10.0.0.1 and 10.0.0.2 as the interface IPs for Tinc.

Install Tinc on both Host A and B

apt-get install tinc

Tinc operates on a concept of network names for the private VPN. Let's call our network 'flockport'.

In /etc/tinc/ on both Host A and Host B create a folder called 'flockport' and do the following.

mkdir /etc/tinc/flockport

This will hold our configuration for this network.

Create a 'hosts' folder in the flockport folder

mkdir /etc/tinc/flockport/hosts

Create the following files in the flockport folder - tinc.conf, tinc-up, tinc-down

touch tinc.conf tinc-up tinc-down

Configure Tinc on Host A

nano /etc/tinc/flockport/tinc.conf

Name = hosta (You can use any name for your hosts)
AddressFamily = ipv4
Interface = tun0

nano tinc-up

#!/bin/bash
ifconfig $interface 10.0.0.1 netmask 255.255.255.0
ip route add 10.0.3.0/24 dev $INTERFACE

nano tinc-down

#!/bin/bash
ifconfig $INTERFACE down
ip route del 10.0.3.0/24 dev $INTERFACE

nano /etc/tinc/flockport/hosts/hosta

Address 1.2.3.4
Subnet 10.0.4.0/24

Configure Tinc on Host B

nano /etc/tinc/flockport/tinc.conf

Name = hostb
AddressFamily = ipv4
Interface = tun0
ConnectTo = hosta

nano tinc-up

#!/bin/bash
ifconfig $interface 10.0.0.2 netmask 255.255.255.0
ip route add 10.0.4.0/24 dev $INTERFACE

nano tinc-down

#!/bin/bash
ifconfig $INTERFACE down
ip route del 10.0.4.0/24 dev $INTERFACE

nano /etc/tinc/flockport/hosts/hostb

Subnet 10.0.3.0/24

Generate keys on both Host A and Host B

tincd -n flockport -K

This will generate private key files in the flockport folder and append public keys to the host files /etc/tinc/flockport/hosts/xxx

Exchange host files on either side
Copy the hosts file with the public keys from /etc/tinc/flockport/hosts/xxx on host A to the hosts folder n Host B and vice versa.

So now your /etc/tinc/flockport/hosts folder on Host A and Host B should have both 'hosta' and 'hostb' files in them

The moment of truth! Run the tincd command on both Host A and Host B

tincd -n flockport

If you followed the guide accurately your containers on both Host A and B should now be able to ping each other

To ensure the Tinc private network starts on reboot edit the /etc/tinc/nets.boot file on Host A and B and add the network name ie in this case flockport. This ensures that the Tinc network startup on boot and is available.

You can easily add more LXC hosts to the network. Tinc has a number of options on optimizing connectivity - compression etc, and choosing the security cipher. Please visit the Tinc website and go through the documentation for more options.

This is unique as we are going to connect 2 (or more) containers behind a NAT network with Tinc. With this config we try not to touch the host networking layer too much. This is a mesh network enabled in containers networking space and the only setting on the host is a single port forward of tcp/udp 655 on one of the hosts, to a container. In this scenario containers will have 2 IPs, the containers basic lxcbr0 NAT network and the Tinc vpn network

The setup is the same as setting up the Tinc LXC network in option 2, only in this case we are going to do it in the containers. Choose 2 random containers from Host A and Host B, let's call them 'debian' from Host A and 'ubuntu' from Host B for this guide.

Installing Tinc in containers need the tun device to be available. So lets create a tun device on both containers first. Default LXC configurations allow the creation of tun devices in containers. Please ensure you are using default lxc templates for this guide.

mkdir /dev/net
mknod /dev/net/tun c 10 200
chmod 666 /dev/net/tun

With the devices create on both containers we are ready to proceed. This the network map.

Host A has public IP 1.2.3.4
Host B has public IP 2.3.4.5
Containers in Host A are on subnet 10.0.3.0/24 via default lxcbr0 nat bridge
Containers in Host B are on subnet 10.0.3.0/24 via default lxcbr0 nat bridge
Debian container is IP 10.0.3.4
Ubuntu container is IP 10.0.3.26

We should put them on different subnets to avoid ip conflicts and facilitate routing but we are building an overlay mesh network and containers will have 2 sets of IPs and actually be available on the network on the subnet 10.0.0.0/24

We are going to use 10.0.0.1 and 10.0.0.2 as the interface IPs for Tinc network

Install Tinc on both Host A and B

apt-get install tinc

Tinc operates on a concept of network names for the private VPN. Let's call our network 'flockport'.
In /etc/tinc/ on both debian and ubuntu containers create a folder called 'flockport' and do the following.

mkdir /etc/tinc/flockport

This will hold our configuration for this network.

Create a 'hosts' folder in the flockport folder

mkdir /etc/tinc/flockport/hosts

Create the following files in the flockport folder - tinc.conf, tinc-up, tinc-down

touch tinc.conf tinc-up tinc-down

Configure Tinc on debian container
nano tinc.conf

Name = debian (You can use any name)
AddressFamily = ipv4
Interface = tun0

nano tinc-up

#!/bin/bash
ifconfig $interface 10.0.0.1 netmask 255.255.255.0

nano tinc-down

#!/bin/bash
ifconfig $INTERFACE down

nano /etc/tinc/flockport/hosts/debian

Address 1.2.3.4
Subnet 10.0.0.1

Configure Tinc on ubuntu container

nano /etc/tinc/flockport/tinc.conf

Name = ubuntu
AddressFamily = ipv4
Interface = tun0
ConnectTo = debian

nano tinc-up

#!/bin/bash
ifconfig $interface 10.0.0.2 netmask 255.255.255.0

nano tinc-down

#!/bin/bash
ifconfig $INTERFACE down

nano /etc/tinc/flockport/hosts/hostb

Subnet 10.0.0.2

Generate keys on both Host A and Host B

tincd -n flockport -K

This will generate a private key in the flockport folder and append a public key to the host file /etc/tinc/flockport/hosts/xxx

Exchange host files on either container
Copy the hosts file with the public keys from /etc/tinc/flockport/hosts/xxx on debian to the hosts folder in ubuntu and vice versa.

So now your /etc/tinc/flockport/hosts folder on debian and ubuntu should have both 'debian' and 'ubuntu' files in them

Since we are in containers how is tinc in ubuntu going to connect to tinc in debian, both behind nat networks? Tinc uses udp/tcp port 655, so we are going to port forward 655 udp and tcp on Debian's host which is Host A public ip 1.2.3.4. This is the ONLY network state we are putting on the host in this configuration.

iptables -t nat -I PREROUTING -i eth0 -p TCP -d 1.2.3.4/32 --dport 655 -j DNAT --to-destination 10.0.3.4:655
iptables -t nat -I PREROUTING -i eth0 -p UDP -d 1.2.3.4/32 --dport 655 -j DNAT --to-destination 10.0.3.4:655

The moment of truth! Run the tincd command on both debian and ubuntu containers

tincd -n flockport

If you followed the guide accurately debian and ubuntu containers should now be able to ping each other.

Add more containers to the network
To add more containers to this network, follow the same process, lets say you want to add container C 'debclone' from Host A to the vpn network. Let's assign a Tinc network IP 10.0.0.3 to debclone and in the tinc.conf for debclone. All the steps remain the same, only add the config below.

Since Container C is on Host A we are using the ConnectTo setting to connect to the Debian container. For a container on Host B we would use this setting to connect to the ubuntu container.

ConnectTo = debian

And in the host file of the Debian container in debclone add the debian container NAT address

Address = 10.0.3.4

Copy the debclone host file to the debian and ubuntu containers and vice versa to debclone.

At this point you have added debclone to the network and debian, ubuntu and debclone will be able to ping each other.

To ensure the TINC private network starts on reboot edit the /etc/tinc/nets.boot file on Host A and B and add the network name ie in this case flockport. This ensures that the tinc network starts up on boot and is available.

Follow the same process to add more containers from other hosts to the Tinc mesh.

Tinc has a number of options to optimize connectivity - compression etc, and choosing the security cipher. Please visit the Tinc website for more options.

With Tinc and GRE tunnels you create layer 3 overlay networks. You can also use IPSEC tunnels to create layer 3 overlays, and since IPSEC is in kernel space it could be more efficient. See our IPSEC guides below.

Connect LXC hosts with IPSEC VPNs

Connect LXC containers across hosts with IPSEC VPNs

You can also create layer 2 overlay networks as we show in the guide below.

LXC extending layer 2 across hosts