Gluster is a distributed file system that's robust, powerful and surprisingly easy to use. We like that. Gluster allows easy file replication across your nodes in real time and can be configured to provide redundancy and high availability through distribution and replication or enhanced performance by striping data across your nodes.

It allows you to pool your data in various configurations similar to raid. You can have distributed volumes or replicated volumes for redundancy and fail over or stripe data across volumes to increase performance, making it a great fit for typical scale out scenarios.

Gluster is an excellent alternative to expensive proprietary storage systems. There are other approaches to distributed storage and high availability like DRBD and Ceph but Gluster is far easier to setup and use. Gluster doesn't need any specific setup, it sits above the file system layer and thus works on any file system (XFS recommended, we use btrfs) and installs like any other app.

In a typical app like WordPress for example apart from the db, data is stored in /wp-content/upload. You can use rsync to keep things in sync between 2 or more instances, but that's not a real time or scalable solution.

Combine Gluster with LXC and you have got yourself a fantastic set of capabilities for a highly scalable architecture. You can have data in Gluster storage bricks inside containers, these can be like your portable scalable instances, or store it outside containers in Gluster storage nodes with Gluster mounts or even LXC bind mounts. The possibilities are endless

For this guide we are going to focus on installing Gluster in containers and use it to replicate data across 2 containers.

Select 2 containers for your Gluster install. For this guide let's call them Gluster 1 and Gluster 2. GlusterFS is going to keep the selected folder in sync between the 2 containers via the network.

You can download the latest Gluster packages as per your container OS. You need the Glusterfs-server and Gluster-fs client packages.

For this guide we are going to use Debian Wheezy 64 bit containers. So let's start by adding the Gluster Wheezy repo to the containers.

wget -O - http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/Debian/wheezy/pubkey.gpg | apt-key add -

echo deb http://download.gluster.org/pub/gluster/glusterfs/3.5/3.5.2/Debian/wheezy/apt wheezy main > /etc/apt/sources.list.d/gluster.list

apt-get update

Gluster depends on Fuse and since we are installing in containers we need to create the fuse device in our containers before installing. Make sure /dev/fuse is loaded in the host, and you are using default LXC OS templates for your containers.

In both containers
mknod /dev/fuse c 10 229

Once this is done install Gluster on both containers

apt-get install glusterfs-server glusterfs-client

If you are in Ubuntu and incase you get an apparmor error, disable apparmor in the container temporarily by using the LXC flag

lxc.aa_profile = off

Reboot the container with the profile off, create the device. Once this is successful you can stop the container, remove the line and reboot the container to proceed.

GlusterFS can use host names or IPs to connect to nodes. We will use host names to configure our replicated storage units. Get the container IPs and add host names in their respective /etc/hosts file.Let's use gluster1 and gluster2 as host name for the 2 containers.

We are using a single LXC host for this guide, in actual use Gluster will typically span multiple hosts. For production environments it's a good idea to configure your internal DNS server to resolve container names. If you use our advanced networking guide to connect containers across several hosts, the /etc/hosts file will suffice.

Let's connect the Gluster nodes. We are using only 2 nodes for this guide. You can initiate the peering from any node. Let's do it from Gluster 1.
gluster peer probe gluster2

You should get a successful peering confirmation
peer probe: success

To check peer status you can use the command
gluster peer status

This should give you a status message like below

Number of Peers: 1

Hostname: gluster2
Port: 24007
Uuid: 7fbha954-9d2a-8n7f-73ga-5ccj42f8932f
State: Peer in Cluster (Connected)

Now that the nodes are peered let's add a Gluster replication volume (from Gluster 1 container node)

gluster volume create volume1 replica 2 transport tcp gluster1:/var/gluster-storage gluster2:/var/gluster-storage

Let's start the volume
gluster volume start volume1

You can query for volume info
gluster volume info

The Gluster client is used to mount volumes in nodes. Let's mount volume 1 in our Gluster 1 container node.
mount -t glusterfs gluster1:/volume1 /var/www/data

Do the same in Gluster 2
mount -t glusterfs gluster1:/volume1 /var/www/data

/var/www/data is the mount point for your replicated Gluster volume. This mount point can of course can be any location folder of your choice. Any data placed here will be replicated across the nodes.

Now that everything works the next step is to ensure these volumes are automatically mounted at boot. You could add a line to the containers fstab but with recent versions of Gluster I have found this often doesn't work.

An alternative is to use an init script. Here is a working init script from the fine folks at the infosec blog. You need to change the 'gluster1' in the script to match your chosen host names for the containers.

### BEGIN INIT INFO
# Provides:          mount-glusterfs
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO

#! /bin/sh
# /etc/init.d/mount-glusterfs
#

# Some things that run always at boot
mount -t glusterfs gluster1:/volume1 /var/www/data

# Uncomment this line if you need to start Apache after mount glusterFS volume
# service apache2 start

# Carry out specific functions when asked to by the system
case "$1" in
  start)
    echo "Mounting glusterfs volumes "
    mount -t glusterfs gluster1:/volume1 /var/www/data
    ;;
  stop)
    echo "Unmount glusterfs volumes"
    umount /var/mount
    ;;
  *)
    echo "Usage: /etc/init.d/mount-glusterfs {start|stop}"
    exit 1
    ;;
esac

exit 0

This is a very simple use case and barely touches the huge feature set and capabilities at play. It's important to understand the concepts and the Gluster website is a good place to start.

However since we love our readers we are going to give a high level conceptual overview and hope it is useful.

Gluster uses the concept of pools, nodes, volumes, bricks and mounts. Depending on your architecture you can bring these together in various ways to give you maximum resilience, performance, availability and flexibility.

A pool is a collection of nodes/peers that form the storage pool, these could be nodes dedicated to storage that are consumed by other nodes or directly the nodes delivering services.

A lot of the use cases we cover are in the cloud, Gluster like LXC does not need us to focus on underlying file systems or storage which is what makes them truly flexible. However you could for instance dedicate underlying hardware specifically to your Gluster pool and create volumes out of them which are consumed by your nodes.

A Gluster volume is created on your storage nodes. It could be a replicated or distributed volume depending on how its configured and it spans nodes. A volume is typically made up of individual bricks on the nodes. Volumes are what is finally consumed. Volumes are mounted in a chosen directory on node/nodes that consume Gluster storage services.

For the use case this guide covered, our storage nodes were our actual application service nodes, and had a single volume with 2 bricks in the 2 nodes (containers) configured for replication.