all posts by


by on October 7, 2016

Compiling GlusterFS with a gcc plug-in – an experiment

Back in February of this year Martin Kletzander gave a talk at devconf.cz on GCC plug-ins. It would seem that gcc plug-ins are a feature that has gone largely overlooked for many years.

I came back from DevConf inspired to try it out. A quick search showed me I was not alone – a colleague here at Red Hat had also seen Martin’s talk, and he wrote about his experiment here source

I had had something similar in mind to what Richard had done. I wanted to check all the struct definitions, i.e. all instances of all the variables of any particular type, and make sure the defined sizes were consistent throughout the GlusterFS sources.

Using Richard’s plug-in I found that while things were generally good, there were a couple structs that appeared to have mismatched sizes. The only thing was, Richard’s plug-in didn’t tell me where those structs were defined. And unfortunately GlusterFS has a lot of cut-and-pasted code, so it wasn’t a matter of a simple grep to find them.

As both Martin and Richard note, the GCC plug-in framework is not well documented. It was not obvious how to do something that seems – on the surface – like it should be trivial. But, with a bit of detective work, I was able to solve it. (And after the fact the change was, in fact, quite simple; finding it however took some time.)

/* structsizes.cc plugin: public domain example code written by
 * Richard W.M. Jones, with modifications by Kaleb S. KEITHLEY
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <gcc-plugin.h>
#include <tree.h>
#include <print-tree.h>

int plugin_is_GPL_compatible;

static FILE *log_fp;

static void
plugin_finish_type (void *event_data, void *user_data)
{
  tree type = (tree) event_data;

  if (user_data) {
    char *c = (char *) user_data;
    fprintf (log_fp, "user_data %x %x %x %x\n", c[0], c[1], c[2], c[3]);
  }

  /* We only care about structs, not any other type definition. */
  if (TREE_CODE (type) == RECORD_TYPE) {
    /* This is useful for working out how to navigate the tree below. */
    /* debug_tree (type); */

    /* If the type is not complete, we can't do anything. */
    if (!COMPLETE_TYPE_P (type)) {
      /* fprintf (log_fp, "struct '%s' has incomplete type\n", name); */
      return;
    }

    /* Struct name? */
    tree name_tree = TYPE_NAME (type);

    /* Ignore unnamed structs. */
    if (!name_tree) {
      /* fprintf (log_fp, "ignoring unnamed struct\n"); */
      return;
    }

    const char *name;
    if (TREE_CODE (name_tree) == IDENTIFIER_NODE)
      name = IDENTIFIER_POINTER (name_tree);
    else if (TREE_CODE (name_tree) == TYPE_DECL && DECL_NAME (name_tree))
      name = IDENTIFIER_POINTER (DECL_NAME (name_tree));
    else
      name = "unknown struct name"; /* should never happen? */

    tree decl = type;
    for (; decl; decl = TREE_CHAIN (decl)) {
      if (DECL_P (decl))
        break;
    }

    const char *filename = DECL_SOURCE_FILE (decl);
    int lineno = DECL_SOURCE_LINE (decl);

    /* Get the size of the struct that has been defined. */
    tree size_tree = TYPE_SIZE (type);
    if (TREE_CODE (size_tree) == INTEGER_CST &&
        !TYPE_P (size_tree) && TREE_CONSTANT (size_tree)) {
      size_t size = TREE_INT_CST_LOW (size_tree);
      fprintf (log_fp, "struct '%s' has size %zu [bits] in %s, line %d\n", name, size, filename, lineno);
    }
    else
      fprintf (log_fp, "struct '%s' has non-constant size\n", name);
  }

  fflush (log_fp);
}

int
plugin_init (struct plugin_name_args *plugin_info,
             struct plugin_gcc_version *version)
{
  const char *logfile = NULL;
  size_t i;

  /* Open the log file. */
  for (i = 0; i argc; ++i) {
    if (strcmp (plugin_info->argv[i].key, "log") == 0) {
      logfile = plugin_info->argv[i].value;
    }
  }

  if (!logfile) {
    fprintf (stderr, "structsizes plugin: missing parameter: -fplugin-arg-structsizes-log=\n");
    exit (EXIT_FAILURE);
  }

  log_fp = fopen (logfile, "a");
  if (log_fp == NULL) {
    perror (logfile);
    exit (EXIT_FAILURE);
  }

  fprintf (log_fp, "Loaded structsizes plugin (GCC %s.%s.%s)\n",
           version->basever, version->devphase, version->revision);

  register_callback (plugin_info->base_name, PLUGIN_FINISH_TYPE,
                     plugin_finish_type, NULL);

  return 0;
}

Compile the plug-in using

    gcc -g -I`gcc -print-file-name=plugin`/include \
        -fpic -shared -o structsizes.so structsizes.cc

and when you compile your source, to use the plug-in you must add

    ... -fplugin=./structsizes.so \
        -fplugin-arg-structsizes-log=<logfile> ...

to the compiler command line options.

For the purposes of compiling GlusterFS with this plug-in, I changed GlusterFS’s configure.ac file like this:

    ...
    CFLAGS="${CFLAGS} -g -O2 -fplugin=/path/to/structsizes.so \
        -fplugin-arg-structsizes-log=/tmp/ss2.out"
    ...

and then ran

    autogen.sh && ./configure && make

to build GlusterFS.

Afterwards you can

    sort -u < /tmp/ss2.out > outfile

to reduce the output to something digestible.

And in the end, the seemingly mismatched struct sizes weren’t really a problem. They were two different types – that happened to have the same type name – in two different translators.

by on October 12, 2015

Linux scale out NFSv4 using NFS-Ganesha and GlusterFS — one step at a time

NFS-Ganesha 2.3 is rapidly winding down to release and it has a bunch of new things in it that make it fairly compelling. A lot of people are also starting to use Red Hat Gluster Storage with the NFS-Ganesha NFS server that is part of that package. Setting up a highly available NFS-Ganesha system using GlusterFS is not exactly trivial. This blog post will “eat the elephant” one bite at a time.

Some people might wonder why use NFS-Ganesha — a user space NFS server — when kernel NFS (knfs) already supports NFSv4? The answer is simple really. NFSv4 in the kernel doesn’t scale. It doesn’t scale out, and it’s a single point of failure. This blog post will show how to set up a resilient, highly available system with no single point of failure.

Crawl

Let’s start small and simple. We’ll set up a single NFS-Ganesha server on CentOS 7, serving a single disk volume.

Start by setting up a CentOS 7 machine. You may want to create a separate volume for the NFS export. We’ll leave this as an exercise for the reader. do not install any NFS.

1. Install EPEL, NFS-Ganesha and GlusterFS. Use the yum repos on download.gluster.org. Repo files are at
nfs-ganesha.repo and glusterfs-epel.repo. Copy them to /etc/yum.repos.d.

    % yum -y install epel-release
    % yum -y install glusterfs-server glusterfs-fuse glusterfs-cli glusterfs-ganesha
    % yum -y install nfs-ganesha-xfs

2. Create a directory to mount the export volume, make a file system on the export volume, and finally mount it:

    % mkdir -p /bricks/demo
    % mkfs.xfs /dev/sdb
    % mount /dev/sdb /bricks/demo

3. Gluster recommends not creating volumes on the root directory of the brick. If something goes wrong it’s easier rm -rf the directory than it is to try and clean it or remake the file system. Create a couple subdirs on the brick:

    % mkdir /bricks/demo/vol
    % mkdir /bricks/demo/scratch

4. Edit the Ganesha config file at /etc/ganesha/ganesha.conf. Here’s what mine looks like:

EXPORT
{
	# Export Id (mandatory, each EXPORT must have a unique Export_Id)
	Export_Id = 1;

	# Exported path (mandatory)
	Path = /bricks/demo/scratch;

	# Pseudo Path (required for NFS v4)
	Pseudo = /bricks/demo/scratch;

	# Required for access (default is None)
	# Could use CLIENT blocks instead
	Access_Type = RW;

	# Exporting FSAL
	FSAL {
		Name = XFS;
	}
}

5. Start ganesha:

    % systemctl start nfs-ganesha

6. Wait one minute for NFS grace to end, then mount the volume:


    % mount localhost:/scratch /mnt

Walk

7. Now we’ll create a simple gluster volume and use NFS_Ganesha to serve it. We also need to disable gluster’s nfs (gnfs).


    % gluster volume create simple $hostname:/bricks/demo/simple
    % gluster volume set simple nfs.disable on
    % gluster volume start simple

8. Edit the Ganesha config file at /etc/ganesha/ganesha.conf. Here’s what mine looks like:

EXPORT
{
	# Export Id (mandatory, each EXPORT must have a unique Export_Id)
	Export_Id = 1;

	# Exported path (mandatory)
	Path = /simple;

	# Pseudo Path (required for NFS v4)
	Pseudo = /simple;

	# Required for access (default is None)
	# Could use CLIENT blocks instead
	Access_Type = RW;

	# Exporting FSAL
	FSAL {
		Name = GLUSTER;
		Hostname = localhost;
		Volume = simple;
	}
}

9. Restart ganesha:


    % systemctl stop nfs-ganesha
    % systemctl start nfs-ganesha

10. Wait one minute for NFS grace to end, then mount the volume:


    % mount localhost:/simple /mnt

Copy a file to the NFS volume. You’ll see it on the gluster brick in /bricks/demo/simple.

Run

Now for the part you’ve been waiting for. For this we’ll start from scratch. This will be a four node cluster: node0, node1, node2, and node3.

1. Tear down anything left over from the above.

2. Ensure that all nodes are resolvable either in DNS or /etc/hosts:


    node0% cat /etc/hosts
    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

    172.16.3.130 node0
    172.16.3.131 node1
    172.16.3.132 node2
    172.16.3.133 node3

    172.16.3.140 node0v
    172.16.3.141 node1v
    172.16.3.142 node2v
    172.16.3.143 node3v

3. Set up passwordless ssh among the four nodes. On node1 create a keypair and deploy it to all the nodes:


    node0% ssh-keygen -f /var/lib/glusterd/nfs/secret.pem
    node0% ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root@node0
    node0% ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root@node1
    node0% ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root@node2
    node0% ssh-copy-id -i /var/lib/glusterd/nfs/secret.pem.pub root@node3
    node0% scp /var/lib/glusterd/nfs/secret.* node1:/var/lib/glusterd/nfs/
    node0% scp /var/lib/glusterd/nfs/secret.* node2:/var/lib/glusterd/nfs/
    node0% scp /var/lib/glusterd/nfs/secret.* node3:/var/lib/glusterd/nfs/

You can confirm that it works with:

    node0% ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/nfs/secret.pem root@node1

4. Start glusterd on all nodes:

    node0% systemctl enable glusterd && systemctl start glusterd
    node1% systemctl enable glusterd && systemctl start glusterd
    node2% systemctl enable glusterd && systemctl start glusterd
    node3% systemctl enable glusterd && systemctl start glusterd

5. From node0, peer probe the other nodes:

    node0% gluster peer probe node1
    peer probe: success
    node0% gluster peer probe node2
    peer probe: success
    node0% gluster peer probe node3
    peer probe: success

You can confirm their status with:

    node0% gluster peer status
    Number of Peers: 3

    Hostname: node1
    Uuid: ca8e1489-0f1b-4814-964d-563e67eded24
    State: Peer in Cluster (Connected)

    Hostname: node2
    Uuid: 37ea06ff-53c2-42eb-aff5-a1afb7a6bb59
    State: Peer in Cluster (Connected)

    Hostname: node3
    Uuid: e1fb733f-8e4e-40e4-8933-e215a183866f
    State: Peer in Cluster (Connected)

6. Create the /etc/ganesha/ganesha-ha.conf file on node0. Here’s what mine looks like:

# Name of the HA cluster created.
# must be unique within the subnet
HA_NAME="demo-cluster"
#
# The gluster server from which to mount the shared data volume.
HA_VOL_SERVER="node0"
#
# You may use short names or long names; you may not use IP addresses.
# Once you select one, stay with it as it will be mildly unpleasant to clean up if you switch later on. Ensure that all names - short and/or long - are in DNS or /etc/hosts on all machines in the cluster.
#
# The subset of nodes of the Gluster Trusted Pool that form the ganesha HA cluster. Hostname is specified.
HA_CLUSTER_NODES="node0,node1,node2,node3"
#
# Virtual IPs for each of the nodes specified above.
VIP_node0="172.16.3.140"
VIP_node1="172.16.3.141"
VIP_node2="172.16.3.142"
VIP_node3="172.16.3.143"

7. Enable the Gluster shared state volume:

    node0% gluster volume set all cluster.enable-shared-storage enable

Wait a few moments for it to be mounted everywhere. You can check that it’s mounted at /run/gluster/shared_storage (or /var/run/gluster/shared_storage) on all the nodes.

8. Enable and start the Pacemaker pcsd on all nodes:

    node0% systemctl enable pcsd && systemctl start pcsd
    node1% systemctl enable pcsd && systemctl start pcsd
    node2% systemctl enable pcsd && systemctl start pcsd
    node3% systemctl enable pcsd && systemctl start pcsd

9. Set a password for the user ‘hacluster’ on all nodes. Use the same password for all nodes:

    node0% echo demopass | passwd --stdin hacluster
    node1% echo demopass | passwd --stdin hacluster
    node2% echo demopass | passwd --stdin hacluster
    node3% echo demopass | passwd --stdin hacluster

10. Perform cluster auth between the nodes. Username is ‘hacluster’, Password is the one you used in step 9:

    node0% pcs cluster auth node0
    node0% pcs cluster auth node1
    node0% pcs cluster auth node2
    node0% pcs cluster auth node3

11. Create the Gluster volume to export. We’ll create a 2×2 distribute-replicate volume. Start the volume:

    node0% gluster volume create cluster-demo replica 2 node0:/home/bricks/demo node1:/home/bricks/demo node2:/home/bricks/demo node3:/home/bricks/demo
    node0% gluster volume start cluster-demo

12. Enable ganesha, i.e. start the ganesha.nfsd:

    node0% gluster nfs-ganesha enable

13. Export the volume:

    node0% gluster vol set cluster-demo ganesha.enable on

14. And finally mount the NFS volume from a client using one of the virtual IP addresses:

    nfs-client% mount node0v:/cluster-demo /mnt

by on October 7, 2015

Automating closing bugs on end-of-life versions of glusterfs

closing a lot of bugs in bugzilla through the (Red Hat) bugzilla web site is a royal pain in the a**.

So—

1) get the bugzilla CLI utility:
  `dnf install -y python-bugzilla`
(or `yum install python-bugzilla`)

2) sign in to bugzilla using your bugzilla account and password:
  `bugzilla login`

3) get a list of bugs to close:
  `bugzilla query --from-url='https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&classification=Community&list_id=3937132&product=GlusterFS&query_format=advanced&version=3.4.2'`
or:
  bugzilla query --from-url='https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&classification=Community&list_id=3937132&product=GlusterFS&query_format=advanced&version=3.4.2' | cut -f 1 -d ' ' | sed -e s/#//`
`
4) and close them:
  `bugzilla modify <bug> [<bug> ...] --close=EOL --comment="GlusterFS 3.4.x has reached end-of-life.\
\
If this bug still exists in a later release please reopen this and change the version or open a new bug."`

Additional scripting is left as an exercise to the reader.

by on August 4, 2015

Gluster Community Packages

The Gluster Community currently provides GlusterFS packages for the following distributions:

                            3.5 3.6 3.7
Fedora 21                    ¹   ×   ×
Fedora 22                    ×   ¹   ×
Fedora 23                    ×   ×   ¹
Fedora 24                    ×   ×   ¹
RHEL/CentOS 5                ×   ×
RHEL/CentOS 6                ×   ×   ×
RHEL/CentOS 7                ×   ×   ×
Ubuntu 12.04 LTS (precise)   ×   ×
Ubuntu 14.04 LTS (trusty)    ×   ×   ×
Ubuntu 15.04 (vivid)             ×   ×
Ubuntu 15.10 (wily)
Debian 7 (wheezy)            ×   ×
Debian 8 (jessie)            ×   ×   ×
Debian 9 (squeeze)           ×   ×   ×
SLES 11                      ×   ×
SLES 12                          ×   ×
OpenSuSE 13                  ×   ×   ×
RHELSA 7                             ×

(Packages are also available in NetBSD and maybe FreeBSD.)

Most packages are available from download.gluster.org

Ubuntu packages are available from Launchpad

As can be seen, the old distributions don’t have pkgs of the latest GlusterFS, usually due to dependencies that are too old or missing. Similarly, new distributions don’t have pkgs of the older versions, for the same reason.

[1] In Fedora, Fedora Updates, or Fedora Updates-Testing for Primary architectures. Secondary architectures seem to be slow to sync with Primary; RPMs for aarch64 are often available from download.gluster.org

by on April 8, 2015

GlusterFS-3.4.7 Released

The Gluster community is please to announce the release of GlusterFS-3.4.7.

The GlusterFS 3.4.7 release is focused on bug fixes:

  • 33608f5 cluster/dht: Changed log level to DEBUG
  • 076143f protocol: Log ENODATA & ENOATTR at DEBUG in removexattr_cbk
  • a0aa6fb build: argp-standalone, conditional build and build with gcc-5
  • 35fdb73 api: versioned symbols in libgfapi.so for compatibility
  • 8bc612d cluster/dht: set op_errno correctly during migration.
  • 8635805 cluster/dht: Fixed double UNWIND in lookup everywhere code

GlusterFS 3.4.7 packages for a variety of popular distributions are available from http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.7/

by on March 27, 2015

GlusterFS 3.4.7beta4 is now available for testing

The 4th beta for GlusterFS 3.4.7 is now available for testing. A handful of bugs have been fixed since the 3.4.6 release, check the references below for details.

Bug reporters are encouraged to verify the fixes, and we invite others to test this beta to check for regressions. The ETA for 3.4.7 GA is tentatively set for April 6 so time is short for testing. Please note that the 3.4 release will EOL when 3.7 is released.

Packages for different distributions can be found on the main download site.

Release Notes for GlusterFS 3.4.7

GlusterFS 3.4.7 consists entirely of bug fixes. The Release Notes for 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.4.5, and 3.4.6 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.4 stable release.

The following changes are included in 3.4.7:

  • 33608f5 cluster/dht: Changed log level to DEBUG
  • 076143f protocol: Log ENODATA & ENOATTR at DEBUG in removexattr_cbk
  • a0aa6fb build: argp-standalone, conditional build and build with gcc-5
  • 35fdb73 api: versioned symbols in libgfapi.so for compatibility
  • 8bc612d cluster/dht: set op_errno correctly during migration.
  • 8635805 cluster/dht: Fixed double UNWIND in lookup everywhere code

Known Issues:

  • memory leak in glusterfs fuse bridge
  • File replicas differ in content even as heal info lists 0 entries in replica 2 setup
by on March 17, 2015

GlusterFS 3.4.7beta2 is now available for testing

The 2nd beta for GlusterFS 3.4.7 is now available for testing. A handful of bugs have been fixed since the 3.4.6 release, check the references below for details.

Bug reporters are encouraged to verify the fixes, and we invite others to test this beta to check for regressions. The ETA for 3.4.7 GA is not set, but will likely be before the end of March. Please note that the 3.4 release will EOL when 3.7 is released.

Packages for different distributions can be found on the main download site.

Release Notes for GlusterFS 3.4.7

GlusterFS 3.4.7 consists entirely of bug fixes. The Release Notes for 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.4.5, and 3.4.6 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.4 stable release.

The following changes are included in 3.4.7:

  • a0aa6fb build: argp-standalone, conditional build and build with gcc-5
  • 35fdb73 api: versioned symbols in libgfapi.so for compatibility
  • 8bc612d cluster/dht: set op_errno correctly during migration.
  • 8635805 cluster/dht: Fixed double UNWIND in lookup everywhere code

Known Issues:

  • memory leak in glusterfs fuse bridge
  • File replicas differ in content even as heal info lists 0 entries in replica 2 setup

And if you’re wondering what happened to the first beta, it was made, but did not build with gcc-5.

by on January 27, 2015

GlusterFS Office Hours at FOSDEM

Lalatendu Mohanty, Niels de Vos, and myself will be holding GlusterFS Office Hours at FOSDEM.

Look for us at the CentOS booth, from 16h00 to 17h00 on Saturday, 31 January.

FOSDEM is taking place this weekend, 31 Jan and 1 Feb, at ULB Solbosch Campus, Brussels. FOSDEM is a free event, no registration is necessary. For more info see https://fosdem.org/2015/

Hope to see you there.

by on

GlusterFS 3.6.2 GA released

The release source tar file and packages for Fedora {20,21,rawhide},
RHEL/CentOS {5,6,7}, Debian {wheezy,jessie}, Pidora2014, and Raspbian
wheezy are available at
http://download.gluster.org/pub/gluster/glusterfs/3.6/3.6.2/

(Ubuntu packages will be available soon.)

This release fixes the following bugs. Thanks to all who submitted bugs
and patches and reviewed the changes.

1184191 – Cluster/DHT : Fixed crash due to null deref
1180404 – nfs server restarts when a snapshot is deactivated
1180411 – CIFS:[USS]: glusterfsd OOM killed when 255 snapshots were
browsed at CIFS mount and Control+C is issued
1180070 – [AFR] getfattr on fuse mount gives error : Software caused
connection abort
1175753 – [readdir-ahead]: indicate EOF for readdirp
1175752 – [USS]: On a successful lookup, snapd logs are filled with
Warnings “dict OR key (entry-point) is NULL”
1175749 – glusterfs client crashed while migrating the fds
1179658 – Add brick fails if parent dir of new brick and existing
brick is same and volume was accessed using libgfapi and smb.
1146524 – glusterfs.spec.in – synch minor diffs with fedora dist-git
glusterfs.spec
1175744 – [USS]: Unable to access .snaps after snapshot restore after
directories were deleted and recreated
1175742 – [USS]: browsing .snaps directory with CIFS fails with
“Invalid argument”
1175739 – [USS]: Non root user who has no access to a directory, from
NFS mount, is able to access the files under .snaps under that directory
1175758 – [USS] : Rebalance process tries to connect to snapd and in
case when snapd crashes it might affect rebalance process
1175765 – USS]: When snapd is crashed gluster volume stop/delete
operation fails making the cluster in inconsistent state
1173528 – Change in volume heal info command output
1166515 – [Tracker] RDMA support in glusterfs
1166505 – mount fails for nfs protocol in rdma volumes
1138385 – [DHT:REBALANCE]: Rebalance failures are seen with error
message ” remote operation failed: File exists”
1177418 – entry self-heal in 3.5 and 3.6 are not compatible
1170954 – Fix mutex problems reported by coverity scan
1177899 – nfs: ls shows “Permission denied” with root-squash
1175738 – [USS]: data unavailability for a period of time when USS is
enabled/disabled
1175736 – [USS]:After deactivating a snapshot trying to access the
remaining activated snapshots from NFS mount gives ‘Invalid argument’ error
1175735 – [USS]: snapd process is not killed once the glusterd comes back
1175733 – [USS]: If the snap name is same as snap-directory than cd to
virtual snap directory fails
1175756 – [USS] : Snapd crashed while trying to access the snapshots
under .snaps directory
1175755 – SNAPSHOT[USS]:gluster volume set for uss doesnot check any
boundaries
1175732 – [SNAPSHOT]: nouuid is appended for every snapshoted brick
which causes duplication if the original brick has already nouuid
1175730 – [USS]: creating file/directories under .snaps shows wrong
error message
1175754 – [SNAPSHOT]: before the snap is marked to be deleted if the
node goes down than the snaps are propagated on other nodes and glusterd
hungs
1159484 – ls -alR can not heal the disperse volume
1138897 – NetBSD port
1175728 – [USS]: All uss related logs are reported under
/var/log/glusterfs, it makes sense to move it into subfolder
1170548 – [USS] : don’t display the snapshots which are not activated
1170921 – [SNAPSHOT]: snapshot should be deactivated by default when
created
1175694 – [SNAPSHOT]: snapshoted volume is read only but it shows rw
attributes in mount
1161885 – Possible file corruption on dispersed volumes
1170959 – EC_MAX_NODES is defined incorrectly
1175645 – [USS]: Typo error in the description for USS under “gluster
volume set help”
1171259 – mount.glusterfs does not understand -n option

Regards,
Kaleb, on behalf of Raghavendra Bhat, who did all the work.

by on November 7, 2014

Some notes on libgfapi.so symbol versions in GlusterFS 3.6.1

A little bit of background——

We started to track API/ABI changes to libgfapi.so by incrementing the SO_NAME, e.g. libgfapi.so.0(.0.0). In the master branch it was incremented to to ‘7’ or libgfapi.so.7(.0.0) for the eventual glusterfs-3.7.

I believe, but I’m not entirely certain¹, that we were supposed to reset this when we branched for release-3.6. Reset it to either ‘6’ or ‘0’, but we didn’t — apparently we forgot about it. In the 3.6.0 betas and if you build the GA release of 3.6.0 you’ll get a libgfapi.so.7(.0.0).

We didn’t hear much, if anything about this until a few days before 3.6.0 was scheduled to GA, when we were ‘reminded’ that older versions of applications like qemu, Samba, and more — linked against previous versions of libgfapi.so — no longer worked after upgrading to the new version of glusterfs.

We briefly experimented with adding a -compat package that installed a symlink: libgfapi.so.0 -> libgfapi.so.7; but some thought this was too much of a hack, and we abandoned that idea.

As a result we now have symbol versions in libgfapi.so. I’ve posted a public spreadsheet with a table of the symbols and the versions of glusterfs that they appear in at

https://docs.google.com/spreadsheets/d/1SKtzgEeVZbKFJgGGMdftf0p-AB5U7yyhVq1n2b6hBeQ/edit?usp=sharing

and also at

https://ethercalc.org/lrjvqrapzu

There are a few things to note about the symbol versions:

  1. so far all the function signatures, i.e. number of parameters and their types have not changed since libgfapi was introduced in 3.4.0. That’s a Good Thing.
  2. the symbol versions are taken from the (community) glusterfs release that they first appeared in.
  3. there are two functions declared in glfs.h that do not have an associated definition. So far it’s not clear why.
  4. there are two functions defined (in glfs-fops.c) that look like they ought to have declarations in glfs.h. Perhaps this was an oversight in the original implementation.
  5. there are several (private?) functions in libgfapi that are not declared in a public header that are used/referenced outside the library. That’s not a Bad Thing, per se, but it’s also not a Good Thing. It seems a bit strange for, e.g. glfs-heal and the nfs server xlator to have to be linked against libgfapi.so. These functions should either be made public or moved to another library, e.g. libglusterfs.so

N.B. that 3, 4, and 5 are also noted in comments in the spreadsheets.

In parallel to all this, RHEL 6.6, RHEL 7.0, and the related CentOS releases shipped updated RHS-GlusterFS client-side packages with version 3.6.0.X. This has resulted in confusion for many of the users of Community GlusterFS who are having issues with upgrading their systems. On top of that the libgfapi.so in these releases is libgfapi.so.0(.0.0), and there are applications included in those releases that are linked with it.

Earlier today (7 Nov, 2014) we released GlusterFS 3.6.1 to address and hopefully mitigate the 3.6.0 upgrade issue and fix the libgfapi.so compatibility issue by reverting to the original SO_NAME (libgfapi.so.0.0.0), now with symbol versions. The applications will not need to be recompiled or relinked.

Knowing that we were going to quickly release GlusterFS 3.6.1 to address these issues, to save our community packagers some work and to try to minimize confusion² we agreed in the community to not package GlusterFS 3.6.0 for any of the Linux distributions. We expect packages for 3.6.1 to be available soon on download.gluster.org.

And if anyone is looking for a nice Google Summer of Code project, linking libglusterfs and the xlators with link maps — with or without symbol versions — is an idea that I think has some merit.

HTH.

¹Not without slogging through a lot of old emails to reconstruct what we originally intended.

²Then we don’t have to explain why some Linux distributions have community 3.6.0 packages and others have (only) 3.6.1.