by on May 6, 2013

GlusterFest Test Day and the Gluster Test Framework

The first beta of glusterfs 3.4 is scheduled for release tomorrow, and the project plans to greet this new beta with GlusterFest: a 24-hour test day, starting at 8pm PDT May 7/03:00 UTC May 8.

Since I plan on participating in the testing, I thought it’d be a good idea to study up on Gluster’s new test framework. You can learn all about the test framework in the video below, but I’ll also walk you through a brief testing experience of my own using the pre-beta code running on my Fedora 18 test machine.

The Gluster framework makes use of the app “prove” which is provided, for Fedora and related OSes, by the package “perl-Test-Harness.” (note: to figure out which package to install, I used the command “yum provides prove”)

sudo yum install -y perl-Test-Harness

To get the test framework itself, I headed over to the shiny new Gluster Forge to grab the source code, pull it down, to my test machine, and check out the 3.4 release branch:

git clone git://forge.gluster.org/glusterfs-core/glusterfs.git
git checkout origin/release-3.4

When Gluster Fest commences, there will be new glusterfs 3.4 beta 1 packages to install and test out, but for my pre ‘Fest test, I built new packages from the source I pulled down from git.

My colleague Justin Clift has written up a very nice howto for building Gluster RPMs from source, which you can also find at the Gluster Forge.

With my freshly-built glusterfs, glusterfs-server & glusterfs-fuse packages installed, I entered the glusterfs source directory and ran one of the “basic” tests. Note: before you run any of these tests, be aware that they delete “/var/lib/glusterd/*” in the course of creating test volumes and data and cleaning up after themselves. For more README-type information, see the README.

cd glusterfs
sudo prove ./tests/basic/bd.t

The test returned a passing result, which is great, but kind of boring. More interestingly, the second test I ran did not run successfully:

sudo prove ./tests/basic/mount.t

This test returned an error while attempting to mount a test volume over the NFS protocol, which is one of the ways you can consume Gluster storage.

mount.nfs: access denied by server while mounting MY_HOSTNAME:/patchy

Debugging time! Following guidance gleaned from the #gluster irc channel, I edited the mount.t test file to add a line containing “set -x” following the first instance of “cleanup;” in the file. I ran the test again, and it spit out a good deal more output, including the command being run to try and mount the volume:

mount -t nfs -o vers=3,nolock,soft,intr MY_HOSTNAME:/patchy /mnt/nfs/0

I could see that the test was, rightly, attempting to mount the volume with NFS version 3 (Gluster’s built-in NFS server doesn’t support version 4), so that wasn’t the problem. I also checked my SELinux log for denials (sudo tail -f /var/log/audit/audit.log | grep denied) and didn’t find any issues. I consulted my /var/log/messages as well, and found the following error message:

rpc.mountd[5465]: refused mount request from MY_IP_ADDRESS for /patchy (/patchy): unmatched host

After a little while Googling around, I remembered something I’d encountered while tangling with Gluster / oVirt integration last year: Gluster’s NFS server doesn’t play nice with another running NFS server. I checked to see if I had an NFS server running on my test machine (sudo service nfs-server status), and, sure enough, I did.

I shut the server down (sudo service nfs-server stop), re-ran my mount.t test, and — hooray — it completed successfully.

That’s perhaps not the most exciting example of Gluster’s test framework in action, but it should give you a sense of how to use it, and, when tests fail, how to go about figuring out why.

I hope to see you, virtually, at the GlusterFest this week. I’m jbrooks in the #gluster irc room — please drop in on Wednesday (or any time) and say hello.

1 Comment

  1. [...] time again – we want to test the GlusterFS 3.4 beta before we unleash it on the world. Like our last test fest, we want you to put the latest GlusterFS beta through real-world usage scenarios that will show you [...]