Flockport labs - Using LXC containers as routers
LXC containers support multiple network interfaces providing flexibility and allowing for interesting use cases. For instance with 2 interfaces you can use your LXC container as a router.
This has limited use but does let you simulate networks and experiment with network namespaces which LXC uses.
In this guide we are going to cover using an LXC container as a router. This is quite similar to using VMs of software routers like Vyatta, Vyos or Pfsense.
These software routers provide ready to use VMs that can be deployed in KVM, Vmware, Xen etc, but given this interesting paper shows LXC outperforms KVM for routing perhaps they should considers providing containers.
The objective here is to make a single LXC container act as a router for a network spanning hosts, containers and VMs.
The network topology is 2 physical hosts. Host A and Host B. Host A will host the router container and has 2 physical network interfaces; eth0 bridged to br0 connected to the gateway internet router and eth1 bridged to br5 and connected to a switch.
Host B has one physical network interface eth0 bridged to br0 and connected to the switch. The router container on Host A will be the router for anything connected to br5 bridge and the physical switch.
Let's start. We are going to configure a single container on Host A and give the container 2 network interfaces, one connected to the wan ie Host A's br0, and the other connected to the LAN ie br5, a standalone bridge which will be the network the router container will serve.
This is how the LXC container config file will look for networking.
lxc.network.type = veth lxc.network.flags = up lxc.network.link = br0 lxc.network.name = eth0 lxc.network.hwaddr = 00:16:3e:ab:f9:65 lxc.network.mtu = 1500 lxc.network.type = veth lxc.network.flags = up lxc.network.link = br5 lxc.network.name = eth1 lxc.network.hwaddr = 00:16:3e:ab:f5:45 lxc.network.mtu = 1500
As you can see we have connected the container's eth0 to br0 bridge on the host so its on the host network and gets an IP from the router.
(If you don't want to bridge your Hosts A eth0 you can use LXC's default lxcbr0 bridge as it provides internet connectivity via iptables masquerading so it can be your wan. However a direct connection to wan is preferred for this use case and simplicity)
lxc.network.type = veth lxc.network.flags = up lxc.network.link = lxcbr0 lxc.network.name = eth0 lxc.network.hwaddr = 00:16:3e:ab:f9:65 lxc.network.mtu = 1500 lxc.network.type = veth lxc.network.flags = up lxc.network.link = br5 lxc.network.name = eth1 lxc.network.hwaddr = 00:16:3e:ab:f5:45 lxc.network.mtu = 1500
To go hardcore you can even pass through your host's eth0 directly to the container using LXC's 'phys' network type. So far we have been using the veth network type. LXC also supports vlan, macvlan, phys, empty and none network types. See ContainerOps excellent post on this here.
Host configuration
First let's bridge eth0 to br0. If this is your primary internet connection and you do not have physical access to the host or router this can potentially lock you out so experiment locally. Note your eth0's IP before doing this, as you will need to move it from eth0 to br0. These instructions presume your router hands out IPs by DHCP. For instance if your eth0 IP is 192.168.1.5
brctl addbr br0 brctl addif br0 eth0 ip addr del 192.168.1.5/24 dev eth0 ip addr add 192.168.1.5/24 dev br0 ip link set br0 up
Now for LAN how will other containers, VMs or hosts connect to your container router to get their IP and routing?
This is where br5 comes in. br5 is a standalone bridge on Host A which we will connect to the router container's eth1. You can of course call the bridge anything. Let's create and bring it up on the host. We are not going to give br5 an IP
brctl addbr br5 ifconfig br5 up
Now in our LXC container's config we can add this as a second interface and map it to eth1 in the container as seen in the config. Next start the container and you should get an IP and internet access for container's eth0 connected to br0. Container's eth1 connected to br5 will have no IP. Check and ensure the container has internet access.
Container configuration
The next steps will happen inside the container. Log in or lxc-attach into the container and Install dnsmasq and iptables.
apt-get install dnsmasq iptables
Now configure dnsmasq to serve the container's eth1 interface. In /etc/dnsmasq.conf configure the values below.
interface=eth1 bind-interfaces listen-address=10.0.2.1 dhcp-range=10.0.2.2,10.0.2.254,12h
Now still inside the container let's give an IP to eth1 and bring it up
ifconfig eth1 10.0.2.1 netmask 255.255.255.0 up
We are choosing the subnet 10.0.2.0/24 for our LAN. This is the subnet our LXC router container will manage.
Now restart dnsmasq and ensure it starts without error.
service dnsmasq restart
Now pay attention, any container or VM on Host A connecting to br5 bridge will get an IP from the dnsmasq instance we just configured inside the container for eth1. (which as we know is connected to br5) Neat!
What we have essentially done here conceptually is take the lxcbr0 bridge setup by the lxc-net script on the host to inside the container.
Start another container or VM on Host A (you can connect your kvm or xen instances usually attached to virbr0 or xenbr0 to the br5 bridge) configured to the br5 bridge and you should see it getting an IP in the 10.0.2.0/24 range from your router container.
lxc.network.type = veth lxc.network.flags = up lxc.network.link = br5 lxc.network.name = eth0 lxc.network.hwaddr = 00:16:3e:ab:e9:25 lxc.network.mtu = 1500
While basic connectivity is up there is no routing. None of the containers connected to the br5 bridge have any routing or access to the Internet. Let's change that. In your router container add this iptables rule
iptables -t nat -A POSTROUTING -s 10.0.2.0/24 ! -d 10.0.2.0/24 -j MASQUERADE
Now log into any container or VM connected to the br5 interface and they should have internet access. In some hosts with differently configured firewalls you may need to add a few more rules. Now basic routing is in place, you can use iptables and other Linux utilities to fine tune your router container to improve performance and offer more services.
Extending the bridge to multiple hosts
This is where you will need a second physical network interface on Host A, and the physical switch. You can get gigabit unmanaged switches for less than $30. What we will do is add the second physical interface eth1 to br5 on Host A
Any bridge on your host can be extended this way.
brctl addif br5 eth1
Now any Host connecting to this switch will get routing and dhcp services from br5 and our router container. If you want to connect VMs and containers on these hosts to br5 then you need to bridge the interface connecting to the physical switch on those hosts, and connect containers and VMs to the bridge.
Running a software router like Vyos or Pfsense as a VM
We noticed confusion online with a lot of folks unsure how to run a VM as a router. If you want to use a virtual instance of a software router like Vyatta, Vyos or Pfsense you can use the same principle of the router container to connect the router VM to br5 so that it becomes the router.
For our tests we used the Vyos VM image in KVM and connected it to the br5 bridge, similar to how we used the LXC router container and it worked perfectly. Vyos is a community fork of Vyatta. Please look at the Vyos user guide for basic configuration.
Make it stick
The instructions we used so far are ephemeral. You will lose the settings on reboot. To make the bridges persistent you need to configure them in your distributions networking config, for instance on Debian or Ubuntu to make bridges persistent across reboot you need to add them to /etc/network/interfaces
auto lo iface lo inet loopback auto eth0 iface eth0 inet manual auto br0 iface br0 inet dhcp bridge_ports eth0 bridge_stp off bridge_fd 0 bridge_maxwait 0 auto eth1 iface eth1 inet manual auto br5 iface br5 inet manual bridge_ports eth1 bridge_stp off bridge_fd 0 bridge_maxwait 0
As you can see in the config above eth0 is bridged to br0 and eth1 to br5. For a standalone bridge you can comment out the 'bridge_port eth1' section. You can also assign a static IP to the br5 bridge when required.
For the router container all we need to do is add a static IP to eth1 using the /etc/network/interfaces file and install iptables-persistent to save the masquerading rule. Dnsmasq via /etc/dnsmasq.conf will take care of DHCP.
In Debian or Ubuntu to make iptables rules persistent you need to install the iptables-persistent package.
apt-get install iptables-persistent
Also for any persistent router related functionality you need to enable network related kernel settings in /etc/sysctl.conf. For instance IP forwarding needs to be enabled for any kind of routing functionality and needs to be uncommented in the syctl.conf file.
If you do not have a second network interface on Host A and a switch you can use the standalone bridge br5 to route local containers or VMs. To extend across hosts you could connect Host B to the gateway internet router and use Ethernet over GRE or L2tpv3 tunnels to extend layer 2 across both hosts by as discussed in Part I of this guide, but this increases complexity and reduces your options.
Stay updated on Flockport news
