by on April 10, 2014

Play with libgfapi and its python bindings..

What is libgfapi ?


User-space library for accessing data in GlusterFS
Filesystem-like API
Runs in application process
no FUSE, no copies, no context switches
…but same volfiles, translators, etc.
Could be used for Apache/nginx modules, MPI I/O
(maybe), Ganesha, etc. ad infinitum
BTW it’s usable from Python too :)

Yes, I copied it from rhsummit.files.wordpress.com/2013/06/darcy_th_1040_glusterfs.pdf

libgfapi improves gluster performance by avoiding “fuse” layer. Its a different route to access glusterfs data. Imagine that, libgfapi sits in the application layer. Different bindings are available to access libgfapi.

www.gluster.org/community/documentation/index.php/Language_Bindings

In this article, I will introduce the python bindings of libgfapi.

The libgfapi python binding is available on GitHub, with a mirror on the Gluster Forge:

github.com/gluster/libgfapi-python

1) First of all, clone the git repo:

    $ git clone github.com/gluster/libgfapi-python.git
    $ cd libgfapi-python
    

2) Then run the setup script:

    $ sudo python setup.py install
    

Once its done, you are almost done with the dev environment :)

Now its really easy to use the functions provided by libgfapi. Let me mention some of the gluster functions available through the python bindings:


glfs_discard(self.fd, offset, len)
glfs_dup(self.fd)
glfs_fallocate(self.fd, mode, offset, len)
glfs_fchown(self.fd, uid, gid)
glfs_fdatasync(self.fd)
glfs_fstat(self.fd, ctypes.byref(s))
glfs_fsync(self.fd)
glfs_read(self.fd, rbuf, buflen, flags)
glfs_write(self.fd, buf, len(buf))
glfs_closedir(self.fd)
glfs_close(self.fd)
glfs_readdir_r(self.fd, ctypes.byref(entry),
glfs_new(volid)
gfs_set_volfile_server(self.fs, proto, host, port)
glfs_fini(self.fs)
glfs_set_logging(self.fs, path, level)
glfs_chown(self.fs, path, uid, gid)
glfs_getxattr(self.fs, path, key, buf, maxlen)
glfs_listxattr(self.fs, path, buf, 512)
glfs_lstat(self.fs, path, ctypes.byref(s))
glfs_mkdir(self.fs, path, mode)
glfs_creat(self.fs, path, flags, mode)
glfs_open(self.fs, path, flags)
glfs_opendir(self.fs, path)
glfs_removexattr(self.fs, path, key)
glfs_rename(self.fs, opath, npath)
glfs_rmdir(self.fs, path)
glfs_setxattr(self.fs, path, key, value, vlen, 0)
glfs_stat(self.fs, path, ctypes.byref(s))
glfs_statvfs(self.fs, path, ctypes.byref(s))  --------------> [1]
glfs_symlink(self.fs, source, link_name)
glfs_unlink(self.fs, path)

The Libgfapi functions available through the bindings are:

    def close(self):
    def discard(self, offset, len):
    def dup(self):
    def fallocate(self, mode, offset, len):
    def fchown(self, uid, gid):
    def fdatasync(self):
    def fstat(self):
    def fsync(self):
    def lseek(self, pos, how):
    def read(self, buflen, flags=0):
    def write(self, data):
    def next(self):
    def set_logging(self, path, level):
    def mount(self):
    def chown(self, path, uid, gid):
    def exists(self, path):
    def getatime(self, path):
    def getctime(self, path):
    def getmtime(self, path):
    def getsize(self, filename):
    def getxattr(self, path, key, maxlen):
    def isdir(self, path):
    def isfile(self, path):
    def islink(self, path):
    def listdir(self, path):
    def listxattr(self, path):
    def lstat(self, path):
    def makedirs(self, name, mode):
    def mkdir(self, path, mode):
    def open(self, path, flags, mode=0777):
    def opendir(self, path):
    def removexattr(self, path, key):
    def rename(self, opath, npath):
    def rmdir(self, path):
    def rmtree(self, path, ignore_errors=False, onerror=None):
    def setxattr(self, path, key, value, vlen):
    def stat(self, path):
    def statvfs(self, path): -------------------------------->[1]
    def symlink(self, source, link_name):
    def unlink(self, path):
    def walk(self, top, topdown=True, onerror=None, followlinks=False):

[1] The patch (review.gluster.org/#/c/7430/ ) for “statvfs” is not merged as of now , :)

I have a gluster setup where I created a distributed volume called “vol2″ :


Volume Name: vol2
Type: Distribute
Volume ID: d355c575-d345-4e54-a7f1-d77b1bfebaf9
Status: Stopped
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.70.43.152:/home/rhs2-1
Options Reconfigured:
server.allow-insecure: on

Lets start accessing this volume using Python. To use the gfapi binding, you need to import gfapi as shown below:

>>> from glusterfs import gfapi

Once it’s done, you access the volume with the mount() method like this:


>>> myVol = gfapi.Volume("10.X.X.152","vol2")
>>> myVol_init = myVol.mount()
>>> myVol_init
0
>>> 

“10.X.X.152″ is my gluster server and “vol2″ is the volume name:

The mount() method basically initialises the connection to the volume.

These are the methods available for the Volume object:

>>> dir(myVol)
['__class__', '__del__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_api', 'chown', 'exists', 'fs', 'getsize', 'getxattr', 'isdir', 'isfile', 'islink', 'listdir', 'listxattr', 'lstat', 'makedirs', 'mkdir', 'mount', 'open', 'opendir', 'removexattr', 'rename', 'rmdir', 'rmtree', 'set_logging', 'setxattr', 'stat', 'statvfs', 'symlink', 'unlink', 'walk']
>>> 

Lets create some entries in this volume and check further:


[root@ ~]# mount -t glusterfs XX.XX.humblec.com:/vol2 /hum
[root@ ~]# cd /hum/
[root@ hum]# dd if=/dev/random of=file1 bs=1M count=5
0+5 records in
0+5 records out
54 bytes (54 B) copied, 11.6826 s, 0.0 kB/s
[root@n hum]# 

This created a 5M file called “file1″ inside “vol2″ volume.

To list the files and directories inside the volume, we use the listdir() method:

>>> myVol.listdir("/")
['file1']

Running a “stat” on the file, from the server, shows this information:


[root@ hum]# stat file1
  File: `file1'
  Size: 54        	Blocks: 1          IO Block: 131072 regular file
Device: 1fh/31d	Inode: 9316719741945628140  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-04-09 19:16:10.209995591 +0530
Modify: 2014-04-09 18:05:00.719006150 +0530
Change: 2014-04-09 18:05:00.719006150 +0530
[root@ hum]# 

Lets prove the python binding works :)

>>> myVol.stat("file1").st_ino
9316719741945628140L
>>> myVol.stat("file1").st_size
54L

If I want to list the extended attributes, I can try something like:

>>> myVol.listxattr("/")
['trusted.glusterfs.volume-id']
>>> 

Lets create a directory called “humble”:

>>> myVol.mkdir("humble/", 0775)
0
>>> 

Checking on the server using ls, it should be there:

[root@ hum]# ls
file1  humble
[root@ hum]# 

Success! For fun, some ‘stat’ information can be displayed using these methods:

>>> myVol.statvfs("/").f_bavail
1060674L

>>> myVol.statvfs("/").f_bfree
1133685L
>>> myVol.statvfs("/").f_files
365760L
>>> 

If you want to mount a gluster volume as a non-root user, you need to follow the steps below.
By default, gluster allows client connections only from privileged ports. To enable connections from unprivileged ports you have to follow below steps.

1. Turn on the allow-insecure option for the volume:

       gluster volume set <volume_name> allow-insecure on

2. Edit /etc/glusterfs/glusterd.vol, adding the line:

       option rpc-auth-allow-insecure on

3. Obviously you should have the permissions on the directory you are accessing for non-root access

4. Start and stop volume

1 Comment

  1. Ram says:

    Very nice! Using these bindings through say a wsgi interface to a web server would allow for many cool things to happen but unfortunately it does not seem to work. The mount, when done in the context of an app server like django’s dev server or wsgi results in a seg fault. Would someone please fix this and open out this avenue of possibilities?!

Leave a Reply

Your email address will not be published.