all posts tagged rackspace

by on March 15, 2013

Glorious Gluster – How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10

A few of our projects recently called for a distributed file-system that provided high availability and redundancy. After a tip off from a fellow techie and a quick browse around the net it appeared that a solution called GlusterFS appeared to tick all the boxes for what we were wanting.

However setting it up turned out not to be as trivial as I had originally anticipated. I’m going to try and put down the process we have evolved for setting it up on Ubuntu in the cloud

A couple of things to clear up first.

  1. We are using Rackspace for our cloud but beyond the setup of the servers it should still be relevant
  2. There are a number of ways to interact with Rackspaces set up but for this we are going to use the cloud control panel
  3. We use Ubuntu as our preferred server which means that our config tends to be all over the place compared to other guides
  4. You will need to set up a minimum of 2 servers and a separate block storage device for each.
  5. We have set up and broken a few different variations of gluster setup so far and make no guarantees that the setup in this blog is infallable but its the best wehave so far.

Setting up the hardware

First things first. We are going to need to set up are some servers.

Feel free to create any size server you want. Just make sure to select Ubuntu 12.10 (or whatever version you may have that is newer).

You will also need to define a new network to work with. We use this to isolate the traffic between the nodes of our new gluster.

You can create a new network when creating the first of your servers. On the creation page under the networks heading you can find a “Create Network” button.

create-networkHopefully this should be quite self explanatory. Now when you create subsequent servers you will then have the option to attach your new network (“GlusterNet” in my example).

Once the two starting nodes have been created then you need to add some additional block storage to store your data on. Make sure that you create blocks that have sufficient capacity for your needs. Something else to consider is using High Performance SSD storage. Its a little on the pricy side but well worth the expense if you are trying to eak out every ounce of performance from the implementation.


You will then need to attach one to each of your servers.


Once attached you will be able to see the details of the block mount point from the block storage details page.


Make a note of the mount point (in this case “/dev/xvdb”) as we will need that in a minute.

Prepare the Server

Now that we have a the hardware ready we can shell into a server to set it up.

First you need to shell into your server and update its OS as the images provided by most cloud supplier tends not to have the latest patches and updates. In our case it’s as simple as:

apt-get update 
apt-get upgrade

Once that’s done we then need to prepare the Block Storage device ( henceforth refered to as a “brick”)

if you run

fdisk -l
  you should see that an entry that looks something like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This indicates that our brick needs a partition table and formatting. We can achieve this be doing the following

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe7da4288.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I’ve highlighted the prompts and my responses. All we are doing here is creating a default partition table that has a single partition which uses up the whole disk.

now running

fdisk -l
  should give us something that looks like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
43 heads, 44 sectors/track, 110843 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe7da4288

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1            2048   209715199   104856576   83  Linu

As you can now see we have a valid device of  /dev/xvdb1 that we can mountHowever we need to create a valid filesystem on the new brick before we can mount it. I have been doing this with Ext4 rather than XFS (which is the recommened filesystem from gluster), this is mainly down to the fact that when i tried using XFS I kept getting some issues with performance and access. I’m sure that with further investigation I could resolve this but as of yet haven’t had chance to. So far though I have had zero issues using Ext4. To create the filesystem we run:

mkfs.ext4 -j /dev/xvdb1

Next, create a folder to mount to, easily done by executing:

mkdir -p /glusterfs/brick

Finally, the simplest way to mount the device is via your /etc/fstab by adding the line

/dev/xvdb1       /glusterfs/brick ext4 defaults 1 2

and running

mount -a
  as root (this will also mean that it mounts on boot for you automatically as well.)

Next we need to install the latest gluster version. At the time of writing this was v3.3.1. You can find a version to suit your OS at If you are using Ubuntu you can do the following

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-server glusterfs-client

By this point you will now have a single working server to continue on your going to need to set up your second server ready to create your new volume.

Once you have your second (or third, fourth, etc) setup its a good idea to add a reference to each one of them to your

  file. This is not really necessary and you can just use the IP addresses of each server but it saves you having to remember each IP and makes it easier to identify.

Remember that we are going to be working with the new network interface you created earlier (i.e “GlusterNet”). to get the IP of your GlusterNet interface a quick ifconfig will show you an interface with an IP that matched the CIDR from earlier. In my case I now have 2 IPs of &

So now I add the following lines to my

  file gluster1 gluster2

Creating our volume

Now that the servers are prepared we can now play with the the tool

.This tool is a life saver in getting everything configured quickly and you can easily get a list of what its capable of by running 
gluster help
. Now Im not going to take you through every command and option and would recomend reading the gluster manual to learn more.

What this tool actually does is help generate and manipulate all the required config that is then stored at


Firstly we need to tell gluster is that we have a pool of servers that will communicate with each other. Gluster refers to these as peers. To do this you need to run 

gluster peer probe gluster2
 on each server for each server that will be used, replacing “gluster2″ with the name names you defined in your
  file. This will then create the appropriate files at 

Now that all our peers have been defined we can get to actually creating the new distributed volume. This however requires a little consideration as there are some decisions you need to make.

If we take a look at the help for creating a new volume we can see that we need to decide on what options to use

volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>

      – what are we going to name our volume
  2. [stripe <COUNT>] [replica <COUNT>]
    – are we going to crate a striped or replicated volume and how many “bricks” are we going to create this volume with
  3. [transport {tcp|rdma|tcp,rdma>]
     - What transport protocol do you want the peers to communicate with
  4. <NEW-BRICK>
     - which servers/bricks do you want to use.

for more information on how to create you volume and what all the options mean have a look at these links

for our purposes we are going to run

gluster volume create myvolume replica 2 transport tcp gluster1:/glusterfs/brick gluster2:/glusterfs/brick

This now creates a new volume that spans both of our servers. you can confirm that this is the case by running

gluster volume info
  and you should get something that looks like

Volume Name: myvolume
Type: Replicate
Volume ID: d3dd24fd-9482-44c3-9503-24291fad8193
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: gluster1:/glusterfs/brick
Brick2: gluster2:/glusterfs/brick

running this on both servers should give you the same results.

What you will now find is that the

  command has created a plethora of files at
. As you work with gluster more and more you will find yourself drawn to these files as they control all the different aspects of how the volume works and performs. Most importantly we will need some information from these files when we come to configure a client to mount the volume.

All that is left to do now is start the volume which can be easily done with a quick 

gluster volume start myvolume

At this point we have now completed setting up our volume but we need to add some security. I would strongly recommend setting up a firewall using

 to control access to the server. The easiest way to do this is to allow all traffic on your “GlusterNet” network interface as only the servers you attach to that network will have access. you can find a guide to using ufw at

Mounting a Client

Now that we have a working volume we need to add some clients. To do this you will need to create a new server as above that is attached to the “GlusterNet” network but without the block storage (unless you really want it that is).

Make sure to add your gluster dfinitions to your


Once you have your new client server ready we can install the gluster client

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-client

I’ve seen a number of different guides that tell you to install glusterfs-server as well but I have as yet had no need to as it all works without it.

Now there are a lot of way that you can mount your new Gluster volume. I have tried a few and have had varying results. What I have found is that the best way is to create a volume file. To do this we create a new file at


volume gluster1
  type protocol/client
  option transport-type tcp
  option remote-host gluster1
  option remote-subvolume /glusterfs/brick
  option username <username>
  option password <password>

volume gluster2
  type protocol/client
  option transport-type tcp
  option remote-host gluster2
  option remote-subvolume /glusterfs/brick
  option username <username>
  option password <password>

volume replicate
  type cluster/replicate
  subvolumes gluster1 gluster2

volume writebehind
  type performance/write-behind
  option cache-size 1MB
  subvolumes replicate

volume cache
  type performance/io-cache
  option cache-size 400MB
  subvolumes writebehind

What you will notice is that there is a

  required for this to work. You can find these details on one of your peer servers in the file 

This /etc/gluster.vol file is basically going to inform the gluster-client software about how to connect to the gluster volume and all the available nodes to connect to. This provides us with some level of fail-over so should one node become unavailable the gluster client will seamlessly switch to a different one. It also allows us to define additional “translators” such as the performance-io one that you can see here. I would strongly recommend reading through the available translators to see which may be useful to you.

Now one of the main issues you will find with Ubuntu is that it will fail on boot if you try to add this mount to your fstab. To get around this you can use Upstart. if you create the following file at

  making sure to change
  to the interface for your GlusterNet network (i.e. eth0 or eth1 or eth2, you get the idea)

author "Matt Cockayne"
description "Mount GlusterFS after networking available"

start on net-device-up IFACE=<interface>
stop on stopping network
stop on starting shutdown

    mount -t glusterfs /etc/glusterfs.vol /glusterfs
end script

As you can see we are using a straight mount command. The magic is that this will not be executed until the start clause validates which in this case is not until the network interface for “GlusterNet” is up and running properly. You will also see that we are mounting the

  file to
  (remember to create this folder to mount to) rather than mounting a network path as you might when mounting an NFS share.

If you wanted you could also add more to your upstart script to handle clean un-mounting of gluster thus allowing you to then use the service

gluster-mount (start|stop|restart)

A quick reboot of the client server should confirm that it boots successfully and you will now end up with your volume mounted at /gluster. You can now test this by creating a new file. I tend to create an empty file at

  just so I have a quick reference that the folder is mounted. Once that’s created if you now go and take a look at the
  on your “peers” you should see that there is now a file called “mounted” sat there looking all smug that it worked.


Some important things for you to be made aware of

by on October 11, 2012

A Buzzword-Packed Return to Gluster UFO

A little while back, I tested out the Unified File and Object feature in Gluster 3.3, which taps OpenStack’s Swift component to handle the object half of the file and object combo. It took me kind of a long time to get it all running, so I was pleased to find this blog post promising a Quick and Dirty guide to UFO setup, and made a mental note to return to UFO.

When my colleague John Mark asked me about this iOS Swift client from Rackspace, I figured that now would be a good time to revisit UFO, and do it on one of the Google Compute Engine instances available to me while I’m in my free trial period with the newest member of Google’s cloud computing family. (OpenStack, iOS & Cloud: Feel the Search Engine Optimization!)

That Quick and Dirty Guide

The UFO guide, written by Kaleb Keithley, worked just as quickly as advertised: start with Fedora 16, 17 or RHEL 6 (or one of the RHEL 6 rebuilds) and end with a simple Gluster install that abides by the OpenStack Swift API. I installed on CentOS 6 because this, along with Ubuntu, is what’s supported right now in Google Compute engine.

Kaleb notes at the bottom of his post that you might experience authentication issues with RHEL 6–I didn’t have this problem, but I did have to add in the extra step of starting the memcache service manually (service memcached start) before starting up the swift service (swift-init main start).

The guide directs you to configure a repository that contains the up-to-date Gluster packages needed. I’m familiar with this repository, as it’s the same one I use on my F17 and CentOS 6 oVirt test systems. I also had to configure the EPEL repository on my CentOS 6 instance, as UFO requires some packages not available in the regular CentOS repositories.

I diverged from the guide in one other place. Where the guide asks you to add this line to the  [filter:tempauth] section of /etc/swift/proxy-server.conf:

user_$myvolname_$username=$password .admin

I found that I had to tack on an extra URL to that line to make the iOS client work:

user_$myvolname_$username=$password .admin https://$myhostname:443/v1/AUTH_$myvolname

Without the extra URL, my UFO setup was pointing the iOS client to a address, which, not surprisingly, the iOS device wasn’t able to access.

The iOS Client (and the Android non-client)

Rackspace’s Cloud Mobile application enables users of the company’s Cloud Servers and Cloud Files offering to access these services from iOS and Android devices. I tried out both platforms, the former on my iPod Touch (recently upgraded to iOS 6) and on my Nexus S 4G smartphone (which runs a nightly build of Cyanogenmod 10).

My subhead above says Android non-client, because, as reviewers in the Google Play store and the developer in this github issue comment both indicate (but the app description and [non-existent] docs do not), the current version of the Android client doesn’t work with the recent, Swift-based incarnation of Rackspace’s cloud Files service.

What’s more, the Android version of the client does not allow any modification of one’s account settings. When I was trial-and-erroring my way toward figuring out the right account syntax, this got pretty annoying. Also annoying was the absence of any detailed error messages.

Things were better (albeit still undocumented) with the iOS version of the client, which allowed for account details editing, for ignoring invalid ssl certs, and for viewing the error message returned by any failed API operations.

In the parlance of the above Gluster UFO setup guide, here are the correct values for the account creation screen (the one you reach in the iOS client after selecting “Other” on the Provider screen:

  • Username:    $myvolname:$username
  • API Key:    $password
  • Name:   $whateveryouwant
  • API Url:    https://$myhostname:443/auth/v1.0
  • Validate SSL Certificate:   OFF
After getting those account details in place, you’ll be able to view the Swift/Gluster containers accessible to your account, create new containers, and upload/download files to and from those containers. There were no options for managing permisisons through the iOS client, so when I wanted to make a container world-readable, I did it from a terminal, using the API.

Google Compute Engine

As I mentioned above, I tested this on Google Compute Engine, the Infrastructure-as-a-Service offering that the search giant announced at its last Google I/O conference. I excitedly signed up for the GCE limited preview as soon as it was announced, but for various reasons, I haven’t done as much testing with it as I’d planned.

Here are my bullet-point impressions of GCE:

  • CentOS or Ubuntu — On GCE, for now, you run the instance types they give you, and that’s either CentOS 6 or Ubuntu 10.04. You can create your own images, by modifying one of the stock images and going through a little process to export and save it. This comes in handy, because, for now, on GCE, there are…
  • No persistent instances — It’s like the earlier days of Amazon EC2. Your VMs lose all their changes when they terminate. There is, however…
  • Persistent storage available — You can’t store VMs in persistent images, but you can hook up your VMs to virtual disks that persist, for storing data.
  • No SELinux — The CentOS images come with SELinux disabled. This turned out to be annoying for me, as OpenShift Origin and oVirt both expect to find SELinux enabled. This cut short a pair of my tests. I was able to modify the oVirt Engine startup script not to complain about SELinux, but was then foiled due to…
  • Monolithic kernel (no module loading) — oVirt engine, which I’d planned to test with a Gluster-only cluster (real virt wouldn’t have worked atop the already-virtualized GCE), wanted to load modules, and there’s no module-loading allowed (for now) on GCE. All told, though…
  • GCE is a lot like EC2 — With a bit of familiarity with the ways of EC2, you should feel right at home on GCE. I opened firewall ports for access to port 443 and port 22 using security groups functionality that’s much like what you have on EC2. You launch instances in a similar way, with Web or command line options, and so on.