by on September 17, 2012

Howto: Using UFO (swift) — A Quick Setup Guide

This sets up a GlusterFS Unified File and Object (UFO) server on a single node (single brick) Gluster server using the RPMs contained in my YUM repo at http://repos.fedorapeople.org/repos/kkeithle/glusterfs/. This repo contains RPMs for Fedora 16, Fedora 17, and RHEL 6. Alternatively you may use the glusterfs-3.4.0beta1 RPMs from the GlusterFS YUM repo at http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.0beta1/

1. Add the repo to your system. See the README file there for instructions. N.B. If you’re using CentOS or some other RHEL clone you’ll want (need) to add the Fedora EPEL repo — see http://fedoraproject.org/wiki/EPEL.

2. Install glusterfs and UFO (remember to enable the new repo first):

  • glusterfs-3.3.1 or glusterfs-3.4.0beta1 on Fedora 17 and Fedora 18: `yum install glusterfs glusterfs-server glusterfs-fuse glusterfs-swift glusterfs-swift-account glusterfs-swift-container glusterfs-swift-object glusterfs-swift-proxy glusterfs-ufo`
  • glusterfs-3.4.0beta1 on Fedora 19, RHEL 6, and CentOS 6: `yum install glusterfs glusterfs-server glusterfs-fuse openstack-swift openstack-swift-account openstack-swift-container openstack-swift-object openstack-swift-proxy glusterfs-ufo`

3. Start glusterfs:

  • On Fedora 17, Fedora 18: `systemctl start glusterd.service`
  • On Fedora 16 or  RHEL 6 `service start glusterd`
  • On CentOS6.x `/etc/init.d/glusterd start`

4. Create a glusterfs volume:
`gluster volume create $myvolname $myhostname:$pathtobrick`

5. Start the glusterfs volume:
`gluster volume start $myvolname`

6. Create a self-signed cert for UFO:
`cd /etc/swift; openssl req -new -x509 -nodes -out cert.crt -keyout cert.key`

7. fixup some files in /etc/swift:

  • `mv swift.conf-gluster swift.conf`
  • `mv fs.conf-gluster fs.conf`
  • `mv proxy-server.conf-gluster proxy-server.conf`
  • `mv account-server/1.conf-gluster account-server/1.conf`
  • `mv container-server/1.conf-gluster container-server/1.conf`
  • `mv object-server/1.conf-gluster object-server/1.conf`
  • `rm {account,container,object}-server.conf

8. Configure UFO (edit /etc/swift/proxy-server.conf):
+ add your cert and key to the [DEFAULT] section:
bind_port = 443
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
+ add one or more users of the gluster volume to the [filter:tempauth] section:
user_$myvolname_$username=$password .admin
+ add the memcache address to the [filter:cache] section:
memcache_servers = 127.0.0.1:11211

9. Generate builders:
`/usr/bin/gluster-swift-gen-builders $myvolname`

10. Start memcached:

  • On Fedora 17: `systemctl start memcached.service`
  • On Fedora 16 or  RHEL 6 `service start memcached`
  • On CentOS6.x `/etc/init.d/memcached start`

11. Start UFO:

`swift-init main start`

» This has bitten me more than once. If you ssh -X into the machine running swift, it’s likely that sshd will already be using ports 6010, 6011, and 6012, and will collide with the swift processes trying to use those ports «

12. Get authentication token from UFO:
`curl -v -H 'X-Storage-User: $myvolname:$username' -H 'X-Storage-Pass: $password' -k https://$myhostname:443/auth/v1.0`
(authtoken similar to AUTH_tk2c69b572dd544383b352d0f0d61c2e6d)

13. Create a container:
`curl -v -X PUT -H 'X-Auth-Token: $authtoken' https://$myhostname:443/v1/AUTH_$myvolname/$mycontainername -k`

14. List containers:
`curl -v -X GET -H 'X-Auth-Token: $authtoken' https://$myhostname:443/v1/AUTH_$myvolname -k`

15. Upload a file to a container:

`curl -v -X PUT -T $filename -H 'X-Auth-Token: $authtoken' -H 'Content-Length: $filelen' https://$myhostname:443/v1/AUTH_$myvolname/$mycontainername/$filename -k`

16. Download the file:

`curl -v -X GET -H 'X-Auth-Token: $authtoken' https://$myhostname:443/v1/AUTH_$myvolname/$mycontainername/$filename -k > $filename`

More information and examples are available from

=======================================================================

N.B. We (Red Hat, Gluster) generally recommend using xfs for brick volumes; or if you’re feeling brave, btrfs. If you’re using ext4 be aware of the ext4 issue* and if you’re using ext3 make sure you mount it with -o user_xattr.

* http://joejulian.name/blog/glusterfs-bit-by-ext4-structure-change/

20 Comments

  1. […] little while back, I tested out the Unified File and Object feature in Gluster 3.3, which taps OpenStack’s Swift component to handle the object half of the file and object combo. It took me kind of a long time to […]

  2. Rob says:

    Thanks for the howto document! I’m looking to test this out for a project I’m working on, so this was very helpful. One thing though… when I try to perform step 7, I see that /etc/swift just contains a few empty directories, and not the files you have indicated. Does this imply that I forgot to install a package? Thanks!

  3. kkeithley says:

    I missed an edit when I updated the post. Install the glusterfs-swift-ufo rpm — instead of glusterfs-swift-plugin rpm — and you’ll get the files you need.

  4. Rob says:

    That fixed it… thanks! Once again, great howto. Thanks for putting it together.

  5. Rob says:

    Hmm… new issue. Everything seems to work great, except for downloading files. No matter how I try to do it (curl, swift, etc) I wind up with a 503 Internal Server Error. The errors printed in /var/log/messages are too long to list here, but here’s a small sample:

    Dec 17 10:00:37 37 object-server ERROR __call__ error with GET /bank0/0/AUTH_bank0/testcontainer/test_file : #012Traceback (most recent call last):#012 File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 892, in __call__#012 res = method(req)#012 File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 1350, in wrapped#012 return func(*a, **kw)#012 File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 677, in GET#012 iter_hook=sleep)#012TypeError: __init__() got an unexpected keyword argument ‘iter_hook’ (txn: tx47b9a96c72424f1889bbeb976d185498)
    Dec 17 10:00:37 37 object-server 127.0.0.1 – – [17/Dec/2012:18:00:37 +0000] “GET /bank0/0/AUTH_bank0/testcontainer/test_file” 500 425 “-” “tx47b9a96c72424f1889bbeb976d185498” “-” 0.0013
    Dec 17 10:00:37 37 proxy-server ERROR 500 Traceback (most recent call last):#012 File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 892, in __call__#012 res = method(req)#012 File “/usr/lib/python2.6/site-packages/swift/common/utils.py”, line 1350, in wrapped#012 return func(*a, **kw)#012 File “/usr/lib/python2.6/site-packages/swift/obj/server.py”, line 677, in GET#012 iter_hook=sleep)#012TypeError: __init__() got an unexpected keyword argument ‘iter_hook’#012 From Object Server 127.0.0.1:6010 (txn: tx47b9a96c72424f1889bbeb976d185498) (client_ip: 172.16.1.9)
    Dec 17 10:00:37 37 proxy-server Object GET returning 503 for [500] (txn: tx47b9a96c72424f1889bbeb976d185498) (client_ip: 172.16.1.9)

    Any ideas? Everything else works so perfectly. And by the way, the swift client and the glusterfs server are on the same private vlan, no firewall between them.

    And finally, the command I executed to download:

    swift -A https://$hostname:443/auth/v1.0 -U $volume:$user -K $password download $container $file

  6. kkeithley says:

    New RPMs are available in my fedorapeople.org yum repo to fix the bug.

    Note that one of the RPMs has been renamed: glusterfs-swift-plugin -> glusterfs-swift-ufo -> glusterfs-ufo. I found that an orderinary update didn’t work and I had to first remove, then add. All your volume files and swift configuration will be retained.

  7. Rob says:

    I got the correct packages installed, and that did fix the errors I listed above. But now, all downloads stall after downloading the first 65536 (2^16) bytes. I’ve only tested with curl and the swift client. But I don’t see anything in any configurations that would impose this limit. Any ideas?

  8. Rob says:

    Is anyone else using this and noticing that downloads stall after the first 64K bytes (65536 to be exact)?

    I’ve tried tweaking the settings to avoid this. For example, setting ‘object_chunk_size’ and ‘client_chunk_size’ in proxy-server.conf to larger sizes, but that doesn’t help. I’ve set them as large as 1GB.

    The only way I can successfully download anything larger than 64k is if I first upload it via the swift client and use ‘-S 65536’ to make sure it breaks the file up into 64K segments or smaller.

  9. kkeithley says:

    We’re working on a fix for the download stall.

  10. Rob says:

    That’s great news! I’ll look forward to that.

  11. kkeithley says:

    I’ve just put up new RPMs with the fix in my fedorapeople.org yum repository.

    3.3.1-8, for fedora 16, fedora 17, epel 5, epel 6, and rhel 7 beta.

    RPMs for fedora 18 will appear in the updates yum repo after the requisite testing period, after f18 ships in a few days.

  12. Rob says:

    I installed the updates, and so far, so good! I’m only playing around with file sizes less than 2GB. But when I get some time, I’ll be pushing larger stuff (~200GB ProRes video files). I imagine that will require some config tweaks to allow to work, and to deal with Swift’s 5GB limits.

    Thanks for the help once again!

  13. Kaleb says:

    UFO ships with the 5GB limit already disabled.

  14. Rob says:

    I can confirm uploads/downloads of 200GB+ with no issues. It’s stressing the NICs on my servers, but that isn’t Swift’s fault! Thanks for the guide, and especially thanks for rapidly responding with updates to the packages, etc.

  15. Sachin says:

    Hi there,

    Thanks for the guide. Before I start, I wanted to learn of there is any performance/benchmark study on this way of doing the Object store. Reason to ask is that Openstack Swift is really sub-optimal in RAID back-ends (since it does may small writes) as noted here

    http://docs.openstack.org/trunk/openstack-object-storage/admin/content/raid.html

    On the other hand, any sizable Gluster deployment probably uses RAID in a big way – irc has several conversations around how to setup a big Gluster deployment (and this uses RAID). See for example

    http://irclog.perlgeek.de/gluster/2012-12-01

    So, any benchmarks anyone has with UFO?

    Thanks in advance!

    -Sachin

  16. Rob says:

    @sachin – I haven’t done any really scientific benchmarking, but I did some throughput tests to ensure the content I am storing on GlusterFS volumes is accessible with a reasonable degree of performance. In my tests, I was moving files ranging in size between 30GB and 200GB, and in all cases, with UFO, I was seeing ~120-130Mbps tranfser rates… and this is using 36 x 3TB SATA drives in RAID6 under my GlusterFS volumes, the volumes are replica 2 in my case. I haven’t done rebuild tests under load on the RAID6 arrays, but I will be doing that later this week. If it takes weeks, as the Rackspace documentation you linked to indicates, then yeah that will be terrible. I have slightly smaller RAID6 arrays that can rebuild in a matter of hours, so I’m hopeful. Hope this helps…

  17. Patrick McShane says:

    Hi there,

    The gist of the problem we’re seeing occurs when we’re doing test uploads of file parts using curl (i.e. foo/test_file.dat/01, foo/test_file.dat/02, …) and using the “X-Object-Manifest” header to allow for downloading the parts as a single file. Uploads/downloads (from 1GB to 20GB in size) of individual files always works just fine.

    It’s only when we try to bind the parts using X-Object-Manifest that we’re unable to download them back into to a single large file.

    Curl reports a 503 error immediately when the multipart download is attempted.

    We’re using openstack folsom build running on CentOS 6.3 x86_64 (using KVM – from the EPEL repo).

    See curl commands and resulting tracebacks in /var/log/messages below.

    Here is our package recipe on CentOS 6.3 x86_64.
    [root@sd-pigshead ~]# rpm -qa|grep glust
    glusterfs-fuse-3.3.1-10.el6.x86_64
    glusterfs-swift-proxy-3.3.1-10.el6.noarch
    glusterfs-3.3.1-10.el6.x86_64
    glusterfs-swift-container-3.3.1-10.el6.noarch
    glusterfs-ufo-3.3.1-10.el6.noarch
    glusterfs-swift-3.3.1-10.el6.noarch
    glusterfs-swift-account-3.3.1-10.el6.noarch
    glusterfs-swift-object-3.3.1-10.el6.noarch
    glusterfs-server-3.3.1-10.el6.x86_64
    [root@sd-pigshead ~]# rpm -qa|egrep “openstack|glust”
    glusterfs-fuse-3.3.1-10.el6.x86_64
    openstack-glance-2012.2.3-1.el6.noarch
    openstack-nova-objectstore-2012.2.2-1.el6.noarch
    openstack-utils-2013.1-1.el6.noarch
    glusterfs-swift-proxy-3.3.1-10.el6.noarch
    openstack-nova-volume-2012.2.2-1.el6.noarch
    openstack-nova-2012.2.2-1.el6.noarch
    openstack-keystone-2012.2.1-1.el6.noarch
    glusterfs-3.3.1-10.el6.x86_64
    glusterfs-swift-container-3.3.1-10.el6.noarch
    glusterfs-ufo-3.3.1-10.el6.noarch
    openstack-nova-common-2012.2.2-1.el6.noarch
    openstack-nova-api-2012.2.2-1.el6.noarch
    python-django-openstack-auth-1.0.2-3.el6.noarch
    openstack-dashboard-2012.2.1-1.el6.noarch
    glusterfs-swift-3.3.1-10.el6.noarch
    glusterfs-swift-account-3.3.1-10.el6.noarch
    openstack-nova-console-2012.2.2-1.el6.noarch
    openstack-nova-cert-2012.2.2-1.el6.noarch
    openstack-quantum-2012.2.1-1.el6.noarch
    glusterfs-swift-object-3.3.1-10.el6.noarch
    openstack-nova-scheduler-2012.2.2-1.el6.noarch
    openstack-cinder-2012.2.1-1.el6.noarch
    glusterfs-server-3.3.1-10.el6.x86_64
    openstack-nova-network-2012.2.2-1.el6.noarch
    openstack-nova-compute-2012.2.2-1.el6.noarch

    curl -v -X PUT -H “X-Auth-Token: ${AUTHTOKEN}” -H “Content-Length: ${FILELEN}” $URLBASE/${DIRNAME}/`basename ${FILENAME}`/00 –data-binary @${FILENAME}_00
    curl -v -X PUT -H “X-Auth-Token: ${AUTHTOKEN}” -H “Content-Length: ${FILELEN}” $URLBASE/${DIRNAME}/`basename ${FILENAME}`/01 –data-binary @${FILENAME}_01

    curl -v -X PUT -H “X-Auth-Token: ${AUTHTOKEN}” -H ‘Content-Length: 0’ -H “X-Object-Manifest: $DIRNAME/`basename $FILENAME`/” $URLBASE/$DIRNAME/`basename $FILENAME` –data-binary ”

    curl -v -H “X-Auth-Token: ${AUTHTOKEN}” $URLBASE/${DIRNAME}/`basename ${FILENAME}` >`basename $FILENAME`_downloaded

    ######### /var/log/messages ##########

    Mar 4 10:24:01 sd-pigshead container-server 127.0.0.1 – – [04/Mar/2013:10:24:01 +0000] “PUT /data-volume/0/AUTH_data-volume/create_dir_test/test_file.dat” 201 – “tx9766153aca824b77954cd96b996645c2” “-” “-” 0.0010
    Mar 4 10:24:01 sd-pigshead object-server 127.0.0.1 – – [04/Mar/2013:10:24:01 +0000] “PUT /data-volume/0/AUTH_data-volume/create_dir_test/test_file.dat” 201 – “-” “tx9766153aca824b77954cd96b996645c2” “curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2” 0.0450
    Mar 4 10:24:01 sd-pigshead container-server 127.0.0.1 – – [04/Mar/2013:10:24:01 +0000] “GET /data-volume/0/AUTH_data-volume/create_dir_test” 200 – “txa396989286234612969f82de55ef9862” “-” “curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2” 0.0224
    Mar 4 10:24:01 sd-pigshead object-server 127.0.0.1 – – [04/Mar/2013:10:24:01 +0000] “GET /data-volume/0/AUTH_data-volume/create_dir_test/test_file.dat” 200 – “-” “tx029ac6fcfb464d199bd92e1b1393a508” “curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2” 0.0009
    Mar 4 10:24:01 sd-pigshead proxy-server ERROR with Object server 127.0.0.1:6010/data-volume re: Trying to GET /v1/AUTH_data-volume/create_dir_test/test_file.dat: #012Traceback (most recent call last):#012 File “/usr/lib/python2.6/site-packages/swift/proxy/controllers/base.py”, line 595, in GETorHEAD_base#012 possible_source = conn.getresponse()#012 File “/usr/lib/python2.6/site-packages/swift/common/bufferedhttp.py”, line 102, in getresponse#012 response = HTTPConnection.getresponse(self)#012 File “/usr/lib64/python2.6/httplib.py”, line 990, in getresponse#012 response.begin()#012 File “/usr/lib64/python2.6/httplib.py”, line 391, in begin#012 version, status, reason = self._read_status()#012 File “/usr/lib64/python2.6/httplib.py”, line 355, in _read_status#012 raise BadStatusLine(line)#012BadStatusLine (txn: tx029ac6fcfb464d199bd92e1b1393a508) (client_ip: 10.12.33.224)
    Mar 4 10:24:01 sd-pigshead proxy-server Object GET returning 503 for [] (txn: tx029ac6fcfb464d199bd92e1b1393a508) (client_ip: 10.12.33.224)

  18. johnmark says:

    Patrick – I would recommend you bring this up in #gluster on IRC or on the gluster-users mailing list.

  19. Andreas Calvo says:

    When uploading large files, it seems that swift is failing:
    Apr 11 15:56:16 DFS1 proxy-server ERROR with Object server 127.0.0.1:6010/testvol re: Trying to write to /v1/AUTH_testvol/vol2/HP_Service_Pack_for_Proliant_2012.10.0-0_713293-001_spp_2012.10.0-SPP2012100.2012_1005.37.iso: ChunkWriteTimeout (10s)
    Apr 11 15:56:16 DFS1 proxy-server Object PUT exceptions during send, 0/1 required connections (txn: tx3552094ea43b467ab88a53075e32cb3a) (client_ip: 10.0.96.43)

    Using 2 nodes (2 bricks replica) with 1 proxy server.

    Is it necessary to tweak swift to allow uploading large files?

  20. kkeithley says:

    As indicated above (14 Jan 2013), Gluster-Swift comes out-of-the-box with the file size limit disabled.

    If you’re using 3.4.0alpha or 3.4.0alpha2, Gluster-Swift is broken in those releases.

Leave a Reply

Your email address will not be published.