all posts by

by on March 29, 2017

Enhancing & Optimizing Gluster for Container Storage

Containers are designed to run applications and be stateless in nature. This necessitates containerized applications to store data externally on persistent storage. Since applications can be launched at any point in time in a container cloud, the persistent storage shares also need to be dynamically provisioned without any administrative intervention. Gluster has been taking big strides for this form of container storage by introducing new features and deepening integration with other projects in the container ecosystem.

We have introduced two deployment models for addressing persistent storage with Gluster:


  • Container Native Storage: Containerized Gluster runs hyperconverged with application containers and builds volumes from disks that are available on the container hosts.
  • Container Ready Storage: Non-containerized Gluster running as a traditional trusted storage pool. Volumes are carved out of this pool and shares are made available to containers.

A lot of our integration focus for persistent storage has been with Kubernetes. Kubernetes provides multiple access modes for persistent storage – Read Write Many (RWM), Read Write Once (RWO) and Read Only Many (ROM).  Gluster’s native file based access has been found to be an apt match for RWM & ROM workloads. Block devices in gluster volumes are suitable for RWO workloads.

For RWM scenarios with both CNS and CRS, we recommend mapping a Kubernetes persistent volume claim to a Gluster volume. This approach provides isolation, reduces the likelihood of noisy neighbors and enables data services like geo-replication, snapshotting to be applied separately for different persistent volumes.

To enable dynamic provisioning of Gluster volumes, ReST based volume management operations have been introduced via Heketi. Heketi can manage multiple trusted storage pools and has the intelligence to carve out a volume in a trusted storage pool with minimal inputs from users. The provisioner for glusterfs in Kubernetes leverages the capabilities exposed by Heketi and creates volumes on the fly for addressing persistent volume claims made by users. You can find our work to bring together all these projects in the gluster-kubernetes project on github.  With support for Storage Classes and Daemon Sets, we have eased the storage setup and dynamic provisioning even further.

Along with  dynamic provisioning, a key requirement in container storage environments is the ability to scale and address a large number of persistent volume claims. To get to this level of scale, Gluster has evolved significantly in the recent 3.10 release. Key features that enable scale include:

  • Brick Multiplexing

  Brick multiplexing introduces the capability of aggregating bricks belonging to several volumes in a single glusterfsd process. This vastly improves the memory footprint of gluster for serving multiple brick directories from the same node. In addition to being a lesser memory hog, a multiplexed brick also consumes far fewer network ports than the non-multiplexed model. In hyperconverged CNS deployments where resources need to be shared between compute and storage, brick multiplexing optimizes gluster to scale to more number of volumes.

 gluster-block provides a management framework for exposing block devices backed by files in a volume through iSCSI. Going forward, we intend using this block interface for scalable RWO persistent volumes. We already have an external provisioner to integrate Kubernetes, Heketi and gluster-block to dynamically provision RWO persistent volumes.

Along with file and block accesses, we have envisioned the need for an Amazon S3 compatible object store in containerized environments. Several applications that are containerized look for ReSTful access to persist data. To address that we recently announced the availability of a gluster-object container that enables accessing a gluster volume through S3 APIs.

We are excited about these innovations in file, block and object accesses of Gluster to address container storage needs. Do let us know if our vision matches your container storage requirements and look forward to more details about our onward journey in the container world here!


by on February 28, 2017

Further notes on Gluster 3.10 and the direction for Gluster

This release of Gluster ushers in improvements for container storage, hyperconverged storage and scale-out Network Attached Storage (NAS) use cases. These use cases have been the primary focus areas for previous releases over the last 12-18 months and will continue to be the primary focus for the next three planned releases.

One of the things we’re really focused on as a project is persistent storage for containerized microservices. Part of this effort has been working with heketi and gluster-kubernetes to enhance our integration with containers. Continuing in the same vein, 3.10 brings about the following key improvements for container storage:


  • Brick multiplexing: Provides the ability to scale the number of exports and volumes per node. This is useful in container storage where there is a need for a large number of shared storage (Read Write Many) volumes. Brick Multiplexing also provides the infrastructure needed to implement Quality of Service in Gluster for a multi-tenant container deployment.
  • gluster-block: Along with 3.10, we are also releasing gluster-block v0.1.  gluster-block provides a very intuitive lifecycle management interface for block devices in a Gluster volume. This release of gluster-block configures block devices to be accessed from initiators through iSCSI.  Work on integrating gluster-block with Heketi for supporting Read Write Once volumes in Kubernetes is in progress.
  • S3 access for Gluster: We are also releasing an Amazon S3 compatible object storage container based on Swift3 and gluster-swift in Gluster’s docker hub. S3 access for Gluster will be useful for application developers who leverage S3 API for storage.


Deployment of hyperconverged storage for containers and virtualization is also a focus area for 3.10. gdeploy provides an improved ansible playbook for deploying hyperconvergence with oVirt and cockpit-gluster provides a wizard to make deployment using this playbook easy with oVirt. gk-deploy makes it easy to deploy Heketi and Gluster in hyperconverged container deployments.


There have been various improvements for scale-out NAS deployments in 3.10. Some of them include:


  • Parallelized readdirp: Improves performance of operations that perform directory crawl. This in conjunction with meta-data caching introduced in 3.9 provides a nice performance boost for small file operations.
  • Statedump support for libgfapi:  Debuggability and supportability improvement for projects like NFS Ganesha and Samba that have been integrated with libgfapi.
  • Estimate for rebalance completion: Helps administrators understand when rebalancing would complete.


Needless to say, none of this would have been possible without the support of our contributors and maintainers. Thank you to all those who made it happen! We  are excited at this juncture to deliver features that enhance the user experience on our key focus areas of container storage, hyperconvergence & scale-out NAS. We intend building on this momentum for further improvements on our key focus areas. Stay tuned and get involved as we progress along this journey with further Gluster releases!


by on October 2, 2015

Gluster News: September 2015

Since we did not have any weekly Gluster news go out in September, this post tries to capture and summarize action from the entire month of September 2015.

==  General News ==

GlusterFS won yet another Bossie in the open source platforms, infrastructure, management, and orchestration software category. Long time users of the project might remember the first Bossie win in 2011.

GlusterFS had three bug fix releases happen in September. 3.7.4, 3.6.6 and 3.5.6 releases happened over the course of this month.

GlusterFS 3.7.5 is expected to be released in the first week of October.

GlusterFS 3.7.x  – Samba libgfapi support is affected currently. Stay on GlusterFS 3.6.x if you use it.

gdeploy, an ansible based deployment tool for Gluster, was released this month. More details about gdeploy can be found in the announcement email.

Introducing georepsetup – Gluster Geo-replication  Setup Tool.

If you are interested in Containers & Gluster, learn more about Running GlusterFS inside Kubernetes.

== Technical News == design discussions happened virtually in the week of 09/28. Discussions happened around these topics:

Several new features have been proposed for Gluster 3.8 and can be found here. A release planning page on for 3.8 is expected in the next few weeks.

Getting Started with Code contributions to GlusterFS –

== Community News ==

Introduction of Amye Scavarda – new community lead for Gluster.

GlusterFS Silicon Valley Meetup group had a meetup at Facebook Campus in Menlo Park, CA. More details about the meetup can be found here.

GlusterFS India Community had a meetup in Bangalore on September 12.

Soumya Koduri & Poornima Gurusiddaiah presented about the following topics at SNIA Software Developers Conference event in Santa Clara, CA.

Achieving Coherent and Aggressive Client Caching in Gluster, a Distributed System
Introduction to Highly Available NFS Server on Scale-Out Storage Systems Based on GlusterFS

Niels de Vos gave a presentation and demo about writing backups with Bareos to Gluster at the Open Source Backup Conference.

Several talks related to Gluster are planned at the upcoming LinuxCon EU Conference in Dublin, Ireland: NFS-Ganesha and Clustered NAS on Distributed Storage Systems – Soumya Koduri, Meghana Madhusudhan Advancements in Automatic File Replication in Gluster – Ravishankar N BitRot Detection in GlusterFS – Gaurav Garg, Venky Shankar Open Storage in the Enterprise with Gluster and Ceph – Dustin Black

We plan to have bi-weekly updates from the next edition. Do stay tuned in to learn about happenings in the Gluster World!






by on January 30, 2015

GlusterFS events over the next 10 days

There is plenty of action planned out for GlusterFS in the period between Jan 30th to Feb 8th 2015. Our community enthusiasts will be attending and presenting at various conferences and meetups across the globe. Here is a breakdown of the planned action:

  1. GlusterFS QuickStart Tutorial by Lalatendu Mohanty at the CentOS Dojo in Brussels, Belgium on Jan 30th.
  2. Talk on GlusterFS Overview and Future Directions by Niels de Vos as part of FOSDEM’s IaaS Devroom in Brussels, Belgium on Jan 31st.
  3. Talk on oVirt and Gluster Hyperconvergence by Federico Simoncelli as part of FOSDEM’s IaaS Devroom on Jan 31st.
  4. Lalatendu Mohanty, Niels de Vos, and Kaleb Keithley will be holding GlusterFS Office Hours at FOSDEM in the CentOS Booth on Jan 31st.
  5. Kaleb Keithley and Lalatendu Mohanty intend talking about The CentOS Storage SIG and GlusterFS as part of FOSDEM’s Distributions Devroom on Feb 1st.
  6. Vijay Bellur will be talking about GlusterFS – Architecture & Roadmap at in Brno, Czech Republic on Feb 6th.
  7. James Shubin will be showcasing Automatically deploying GlusterFS with Puppet-Gluster + Vagrant at on Feb 7th.
  8. Following talks are scheduled as part of GlusterFS Meetup in Pune, India on Feb 7th:
    • GlusterFS & Roadmap by Atin Mukherjee
    • GlusterFS and Ovirt by Ramesh Nachimuthu
    • GlusterFS in Openstack by Prasanth Pai
    • GlusterFS for storage admins by Vikhyat Umrao
  9. Dan Lambright and Joseph Fernandes will talk about Efficient Data Maintenance in GlusterFS using databases in on Feb 8th.

As you can see, it is a busy few days ahead for the Gluster community. Do drop by to one of these events and connect with other Gluster enthusiasts from all over. If you are not attending any of these events, do watch this space for event reports from everywhere!

by on November 4, 2014

GlusterFS 3.6.0 – Looking Back and Looking Ahead..

GlusterFS 3.6.0 was released over the last weekend. As I mulled over the release cycle during the weekend, several thoughts on both on the technical and project fronts crossed my mind. This post is an attempt to capture some of those thoughts.

On the technical front, several new features and significant number of bug fixes are part of GlusterFS 3.6.0. The release notes contains a comprehensive list of what’s new in this release. My noteworthy list includes:

  •  Volume Snapshots – Distributed lvm thin-pool based snapshots for backing up volumes in a Gluster Trusted Storage Pool.  Apart from providing cluster wide co-ordination to trigger a consistent snapshot, several improvements have been  performed through the GlusterFS stack to make translators more crash consistent. Snapshotting of volumes is tightly coupled with lvm today but one could also enhance the same framework to integrate with a backend storage technology like btrfs that can perform snapshots.
  •  Erasure Coding – Xavier Hernandez from Datalab added support to perform erasure coding of data in a GlusterFS volume across nodes in a Trusted Storage Pool. Erasure Coding requires fewer nodes to provide better redundancy than a n-way replicated volume and can help in reducing the overall deployment cost. We look forward to build on this foundation and deliver more enhancememnts in upcoming releases.
  • Meta translator – This translator provides a /proc like view for examining internal state of translators on the client stack of a GlusterFS volume and certainly looks like an interface that I would be heavily consuming for introspection of  GlusterFS.
  • Automatic File Replication (AFR) v2 – A significant re-factor of the synchronous replication translator, provides granular entry self-healing and reduced resource consumption with entry self-heals.
  • NetBSD, OSX and FreeBSD ports – Lot of fixes on the portability front. The NetBSD port passes most regression tests as of 3.6.0. At this point, none of these ports are ready to be deployed in production. However, with dedicated maintainers for each of these ports, we expect to reach production readiness on these platforms in a future release.

On the project front,  there have been several things of note happening as far as this release cycle goes. Here are my personal highlights from 3.6.0:

  • 3.6.0 marks the first occurrence of two major GlusterFS releases happening in a calendar year.
  • More patches merged in the 6 month cycle leading to 3.6.0 than any other major release. Addition of more maintainers for GlusterFS and revamp of our Jenkins based CI infrastructure seems to be helping here.

Looking ahead, there are already signs of what is coming up in GlusterFS 3.7. Projects like bitrot detection, volume tiering, Trash translator, Export/Netgroup style authentication for gluster NFS are already in progress and in all probability will be part of 3.7.0 along with other features. We have also started early discussions on what we would want to build for releases and more details will be available in the community soon. If you have thoughts on how addition of a new feature can address your problem(s), do let us know and that can help us in planning out the future of GlusterFS.

Overall, it has been a gratifying time for me to be in the Gluster Community. Personally I look forward to build on this feeling of gratification as we build the next versions of GlusterFS to play a significant role in the distributed storage segment.

by on July 28, 2013

Feedback requested: Governance of GlusterFS project

We are in the process of formalizing the governance model of the GlusterFS project. Historically, the governance of the project has been loosely structured. This is an invitation to all of you to participate in this discussion and provide your feedback and suggestions on how we should evolve a formal model. Feedback from this thread will […]

by on July 15, 2013

Upgrading to GlusterFS 3.4.0

Now that GlusterFS 3.4.0 is out, here are some mechanisms to upgrade from earlier installed versions of GlusterFS.

Upgrade from GlusterFS 3.3.x:

GlusterFS 3.4.0 is compatible with 3.3.x (yes, you read it right!). You can upgrade your deployment by following one of the two procedures below.

a) Scheduling a downtime (Recommended)

For this approach, schedule a downtime and prevent all your clients from accessing the servers.

  • Stop all glusterd, glusterfsd and glusterfs processes on your server.
  • Install  GlusterFS 3.4.0
  • Start glusterd.
  • Ensure that all started volumes have processes online in “gluster volume status”.

You would need to repeat these 4 steps on all servers that form your trusted storage pool.

After upgrading the servers, it is recommended to upgrade all client installations to 3.4.0.

b) Rolling upgrades with no downtime

If you have replicated or distributed replicated volumes with bricks placed in the right fashion for redundancy, have no data to be self-healed and feel adventurous, you can perform a rolling upgrade through the following procedure:

  • Stop all glusterd, glusterfs and glusterfsd processes on your server.
  • Install GlusterFS 3.4.0.
  • Start glusterd.
  • Run “gluster volume heal <volname> info” on all volumes and ensure that there is nothing left to be self-healed on every volume. If you have pending data for self-heal, run “gluster volume heal <volname>” and wait for self-heal to complete.
  • Ensure that all started volumes have processes online in “gluster volume status”.

Repeat the above steps on all servers that are part of your trusted storage pool.

Again after upgrading the servers, it is recommended to upgrade all client installations to 3.4.0.

Upgrade from GlusterFS 3.1.x and 3.2.x:

You can follow the procedure listed here and replace 3.3.0 with 3.4.0 in the procedure.

Do report your findings on 3.4.0 in gluster-users, #gluster on Freenode and bugzilla.

Please note that this may not work for all installations & upgrades. If you notice anything amiss and would like to see it covered here, please leave a comment and I will try to incorporate your findings.

by on May 31, 2012

Upgrading to GlusterFS 3.3.0

Now that GlusterFS 3.3.0 is out, here is a quick primer on upgrading from earlier installed versions of GlusterFS. This howto covers upgrades from 3.1.x and 3.2.x versions of GlusterFS.

1) GlusterFS 3.3.0 is not compatible with any earlier released versions. Please make sure that you schedule a downtime before you upgrade.

2) Stop all glusterd, glusterfs and glusterfsd processes running in all your servers and clients.

3) Take a backup of glusterd’s  working directory on all servers – usually glusterd’s working directory is /etc/glusterd.

4) Install GlusterFS 3.3.0 on all your servers and clients. With 3.3.0, the default working directory has been changed to  /var/lib/glusterd. RPM and source installations move all content under /etc/glusterd to /var/lib/glusterd. On all servers, ensure that the directory structure remains consistent with the backup obtained in 3).  New files are created as part of the upgrade process and hence you may see some additional files. However, ensure all files backed up in 3) are present in /var/lib/glusterd.

5) If you have installed from RPM, goto 6).  Else, start glusterd in upgrade mode. glusterd terminates after it performs the necessary steps for upgrade. Re-start glusterd normally after this termination. Essentially this process boils down to:

a) killall glusterd

b) glusterd --xlator-option *.upgrade=on -N

This will re-generate volume files with the new ‘index’ translator which is needed for features like pro-active self heal in 3.3.0.

c) Start glusterd

Ensure that you repeat a), b) and c) on all servers.

6) All your gluster services on the servers should be back online now. Mount your native clients with GlusterFS 3.3.0 to access your volumes.

Enjoy your shiny new 3.3.0 and report your findings in #gluster on Freenode, bugzilla and c.g.o

Please note that this may not work for all installations & upgrades. If you notice anything amiss and would like to see it covered here, please leave a comment and I will try to incorporate your findings.