all posts tagged ceph


by on September 22, 2014

Experimenting with Ceph support for NFS-Ganesha

NFS-Ganesha is a user-space NFS-server that is available in Fedora. It contains several plugins (FSAL, File System Abstraction Layer) for supporting different storage backends. Some of the more interesting are:

Setting up a basic NFS-Ganesha server

Exporting a mounted filesystem is pretty simple. Unfortunately this failed for me when running with the standard nfs-ganesha packages on a minimal Fedora 20 installation. The following changes were needed to make NFS-Ganesha work for a basic export:

  • install rpcbind and make the nfs-ganesha.service depend on it
  • copy /etc/dbus-1/system.d/org.ganesha.nfsd.conf from the sources
  • createa /etc/sysconfig/nfs-ganesha environment file

When these initial things have been taken care of, a configuration file needs to be created. The default configuration file mentioned in the environment file is /etc/ganesha.nfsd.conf. The sources of nfs-ganesha contain some examples, the vfs.confis quite usable as a starting point. After copying the example and modifying the paths to something more suitable, starting the NFS-server should work:

# systemctl start nfs-ganesha

In case something failed, there should be a note about it in /var/log/ganesha.log.

Exporting the Ceph filesystem with NFS-Ganesha

This assumes you have a working Ceph Cluster which includes several MON, OSD and one or more MDS daemons. The FSAL_CEPH from NFS-Ganesha uses libcephfs which seems to be the same as the ceph-fuse package for Fedora. the easiest way to make sure that the Ceph filesystem is functional, is to try and mount it with ceph-fuse.

The minimal requirements to get a Ceph client system to access the Ceph Cluster, seem to be a /etc/ceph/ceph.conf with a [global]section and a suitable keyring. Creating the ceph.conf on the Fedora system that was done the ceph-deploy:

$ ceph-deploy config push $NFS_SERVER

In my setup I scp'd the /etc/ceph/ceph.client.admin.keyring from one of my Ceph servers to the $NFS_SERVER. There probably are better ways to create/distribute a keyring, but I'm new to Ceph and this worked sufficiently for my testing.

When the above configuration was done, it was possible to mount the Ceph filesystem on the Ceph client that is becoming the NFS-server. This command worked without issues:

# ceph-fuse /mnt
# echo 'Hello Ceph!' > /mnt/README
# umount /mnt

The first write to the Ceph filesystem took a while. This is likely due to the initial work the MDS and OSD daemons need to do (like creating pools for the Ceph filesystem).

After confirming that the Ceph Cluster and Filesystem work, the configuration for NFS-Ganesha can just be taken from the sources and saved as /etc/ganesha.nfsd.conf. With this configuration, and restarting the nfs-ganesha.service, the NFS-export becomes available:

# showmount -e $NFS_SERVER
Export list for $NFS_SERVER:
/ (everyone)

NFSv4 uses a 'pseudo root' as mentioned in the configuration file. This means that mounting the export over NFSv4 results in a virtual directory structure:

# mount -t nfs $NFS_SERVER:/ /mnt
# find /mnt
/mnt
/mnt/nfsv4
/mnt/nfsv4/pseudofs
/mnt/nfsv4/pseudofs/ceph
/mnt/nfsv4/pseudofs/ceph/README

Reading and writing to the mountpoint under /mnt/nfsv4/pseudofs/ceph works fine, as long as the usual permissions allow that. By default NFS-Ganesha enabled 'root squashing', so the 'root' user may not do a lot on the export. Disabling this security measure can be done by placing this option in the export section:

Squash = no_root_squash;

Restart the nfs-ganesha.service after modifying /etc/ganesha.nfsd.conf and writing files as 'root' should work too now.

Future Work

For me, this was a short "let's try it out" while learning about Ceph. At the moment, I have no intention on working on the FSAL_CEPH for NFS-Ganesha. My main interest of this experiment with exporting a Ceph filesystem though NFS-Ganesha on a plain Fedora 20 installation, was to learn about usability of a new NFS-Ganesha configuration/deployment. In order to improve the user experience with NFS-Ganesha, I'll try and fix some of the issues I run into. Progress can be followed in Bug 1144799.

In future, I will mainly use NFS-Ganesha for accessing Gluster Volumes. My colleague Soumya posted a nice explanation on how to download, build and run NFS-Ganesha with support for Gluster. We will be working on improving the out-of-the-box support in Fedora while stabilizing the FSAL_GLUSTER in the upstream NFS-Ganeasha project.

by on May 5, 2014

An OpenStack Storage Hackathon

With technologies around Open Software-defined Storage emerging as the way to get things done in the cloud, we’ve noticed strong interest in how to take advantage of this emerging software space. Storage is changing from the proprietary, expensive box in the corner to a set of APIs and open source software deployed in a scale-out way on your infrastructure. Storage services are now an integrated part of your scale-out applications.

To accelerate this momentum, we thought it would be fun to have a storage hackathon at the OpenStack Summit to encourage developers to dive into this brave new world.

We’re starting at 1pm on May 11, and we’ll be hacking into the night until 8 or whenever folks get tired. After that, Red Hat will sponsor drinks at a local watering hole.

Experts will be on hand to help new hackers find their way. Come by, learn, collaborate, and write some apps.

RSVP at osstorage-hack.eventbrite.com

by on November 12, 2013

On the Gluster vs Ceph Benchmarks

If you’ve been following the Gluster and Ceph communities for any length of time, you know that we have similar visions for open software-defined storage and are becoming more competitive with each passing day. We have been rivals in a similar space for some time, but on friendly terms – and for a couple of simple reasons: 1.) we each have common enemies in the form of proprietary big storage and 2.) we genuinely like each other. I’m a Sage Weil fan and have long been an admirer of his work. Ross Turk and Neil Levine are also members of the Inktank clan whom I respect and vouch for on a regular basis. There are others I’m forgetting, and I hope they don’t take it personally!

So you can imagine the internal debate I had when presented with the first results of a Red Hat Storage comparison with Ceph in a set of benchmarks commissioned by the Red Hat Storage product marketing group (for reference, they’re located here). If you saw my presentations at the OpenStack Summit in Hong Kong, then you know I went with it, and I’m glad I did. While the Ceph guys have been very good about not spouting FUD and focusing instead on the bigger picture – taking down EvilMaChines, for example – others in the clan of OpenStack hangers-on have not been so exemplary.

I don’t know who, exactly, the Red Hat Storage marketing group was targeting with the benchmarks, but I am targeting a very specific audience, and it isn’t anyone associated with Inktank or the Ceph project. I am targeting all the people in the OpenStack universe who wrote us off and wanted to declare the storage wars over. I’m also a bit tired of the inexplicable phrase that “Ceph is faster than Gluster”, often said with no qualification, which I’ve known for sometime was not true. It’s that truism, spouted by some moustachioed cloudy hipsters at an OpenStack meetup, that rankles me – almost as much as someone asking me in a public forum why we shouldn’t all ditch Gluster for Ceph. The idea that one is unequivocally faster or better than the other is completely ridiculous – almost as ridiculous as the thought that hipsters in early 20th century drag are trusted experts at evaluating technology. The benchmarks in question do not end any debates. On the contrary, they are just the beginning.

I felt uneasy when I saw Sage show up at our Gluster Cloud Night in Hong Kong, because I really didn’t intend for this to be an “In yo’ face!” type of event. I did not know beforehand that he would be there, but even if I had, I wouldn’t have changed my decision to show the results. The “Ceph is faster” truism had become one of those things that everyone “knows” without the evidence to support it, and the longer we let it go unopposed, the more likely it was to become a self-fulfilling prophecy. Also, while we may have common enemies, it has become increasingly clear that the OpenStack universe would really prefer to converge around a single storage technology, and I will not let that happen without a fight.

We’ve radically improved GlusterFS and the Gluster Community over the last couple of years, and we are very proud of our work. We don’t have to take a back seat to anyone; we don’t have to accept second place to anyone; and we’re not going to. In the end, it’s very clear who the winners of this rivalry will be. It won’t be Ceph, and it won’t be Gluster. It will be you, the users and developers, who will benefit from the two open source heavyweights scratching and clawing their way to the top of the heap. Rejoice and revel in your victory, because we work for you.

To see the benchmark results for yourself, see the Red Hat Storage blog post on the subject.

To see the VAR Guy’s take, see this article.