all posts tagged Syndicated


by on February 16, 2017

GlusterFS 3.8.9 is an other Long-Term-Maintenance update

We are proud to announce the General Availability of yet the next update to the Long-Term-Stable releases for GlusterFS 3.8. Packages are being prepared to hit the mirrors expected to hit the repositories of distributions and the Gluster download server over the next few days. Details on which versions are part of which distributions can be found on the Community Packages in the documentation. The release notes are part of the git repository, the downloadable tarball and are included in this post for easy access.

Release notes for Gluster 3.8.9

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1, 3.8.2, 3.8.3, 3.8.4, 3.8.5, 3.8.6, 3.8.7 and 3.8.8contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 16 patches have been merged, addressing 14 bugs:
  • #1410852: glusterfs-server should depend on firewalld-filesystem
  • #1411899: DHT doesn't evenly balance files on FreeBSD with ZFS
  • #1412119: ganesha service crashed on all nodes of ganesha cluster on disperse volume when doing lookup while copying files remotely using scp
  • #1412888: Extra lookup/fstats are sent over the network when a brick is down.
  • #1412913: [ganesha + EC]posix compliance rename tests failed on EC volume with nfs-ganesha mount.
  • #1412915: Spurious split-brain error messages are seen in rebalance logs
  • #1412916: [ganesha+ec]: Contents of original file are not seen when hardlink is created
  • #1412922: ls and move hung on disperse volume
  • #1412941: Regression caused by enabling client-io-threads by default
  • #1414655: Upcall: Possible memleak if inode_ctx_set fails
  • #1415053: geo-rep session faulty with ChangelogException "No such file or directory"
  • #1415132: Improve output of "gluster volume status detail"
  • #1417802: debug/trace: Print iatts of individual entries in readdirp callback for better debugging experience
  • #1420184: [Remove-brick] Hardlink migration fails with "lookup failed (No such file or directory)" error messages in rebalance logs
by on January 15, 2017

An other Gluster 3.8 Long-Term-Maintenance update with the 3.8.8 release

The Gluster team has been busy over the end-of-year holidays and this latest update to the 3.8 Long-Term-Maintenance release intends to fix quite a number of bugs. Packages have been built for many different distributions and are available from the download server. The release-notes for 3.8.8 have been included below for the ease of reference. All users on the 3.8 version are recommended to update to this current release.

Release notes for Gluster 3.8.8

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1, 3.8.2, 3.8.3, 3.8.4, 3.8.5, 3.8.6 and 3.8.7 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 38 patches have been merged, addressing 35 bugs:
  • #1375849: [RFE] enable sharding with virt profile - /var/lib/glusterd/groups/virt
  • #1378384: log level set in glfs_set_logging() does not work
  • #1378547: Asynchronous Unsplit-brain still causes Input/Output Error on system calls
  • #1389781: build: python on Debian-based dists use .../lib/python2.7/dist-packages instead of .../site-packages
  • #1394635: errors appear in brick and nfs logs and getting stale files on NFS clients
  • #1395510: Seeing error messages [snapview-client.c:283:gf_svc_lookup_cbk] and [dht-helper.c:1666ht_inode_ctx_time_update] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/replicate.so(+0x5d75c)
  • #1399423: GlusterFS client crashes during remove-brick operation
  • #1399432: A hard link is lost during rebalance+lookup
  • #1399468: Wrong value in Last Synced column during Hybrid Crawl
  • #1399915: [SAMBA-CIFS] : IO hungs in cifs mount while graph switch on & off
  • #1401029: OOM kill of nfs-ganesha on one node while fs-sanity test suite is executed.
  • #1401534: fuse mount point not accessible
  • #1402697: glusterfsd crashed while taking snapshot using scheduler
  • #1402728: Worker restarts on log-rsync-performance config update
  • #1403109: Crash of glusterd when using long username with geo-replication
  • #1404105: Incorrect incrementation of volinfo refcnt during volume start
  • #1404583: Upcall: Possible use after free when log level set to TRACE
  • #1405004: [Perf] : pcs cluster resources went into stopped state during Multithreaded perf tests on RHGS layered over RHEL 6
  • #1405130: `gluster volume heal split-brain' does not heal if data/metadata/entry self-heal options are turned off
  • #1405450: tests/bugs/snapshot/bug-1316437.t test is causing spurious failure
  • #1405577: [GANESHA] failed to create directory of hostname of new node in var/lib/nfs/ganesha/ in already existing cluster nodes
  • #1405886: Fix potential leaks in INODELK cbk in protocol/client
  • #1405890: Fix spurious failure in bug-1402841.t-mt-dir-scan-race.t
  • #1405951: NFS-Ganesha:Volume reset for any option causes reset of ganesha enable option and bring down the ganesha services
  • #1406740: Fix spurious failure in tests/bugs/replicate/bug-1402730.t
  • #1408414: Remove-brick rebalance failed while rm -rf is in progress
  • #1408772: [Arbiter] After Killing a brick writes drastically slow down
  • #1408786: with granular-entry-self-heal enabled i see that there is a gfid mismatch and vm goes to paused state after migrating to another host
  • #1410073: Fix failure of split-brain-favorite-child-policy.t in CentOS7
  • #1410369: Dict_t leak in dht_migration_complete_check_task and dht_rebalance_inprogress_task
  • #1410699: [geo-rep]: Config commands fail when the status is 'Created'
  • #1410708: glusterd/geo-rep: geo-rep config command leaks fd
  • #1410764: Remove-brick rebalance failed while rm -rf is in progress
  • #1411011: atime becomes zero when truncating file via ganesha (or gluster-NFS)
  • #1411613: Fix the place where graph switch event is logged
by on September 15, 2016

GlusterFS 3.8.4 is available, Gluster users are advised to update

Even though the last release 3.8 was just two weeks ago, we're sticking to the release schedule and have 3.8.4 ready for all our current and future users. As with all updates, we advise users of previous versions to upgrade to the latest and greatest. Several bugs have been fixed, and upgrading is one way to prevent hitting known problems in future.

Release notes for Gluster 3.8.4

This is a bugfix release. The Release Notes for 3.8.0, 3.8.1, 3.8.2 and 3.8.3 contain a listing of all the new features that were added and bugs fixed in the GlusterFS 3.8 stable release.

Bugs addressed

A total of 23 patches have been merged, addressing 22 bugs:
  • #1332424: geo-rep: address potential leak of memory
  • #1357760: Geo-rep silently ignores config parser errors
  • #1366496: 1 mkdir generates tons of log messages from dht xlator
  • #1366746: EINVAL errors while aggregating the directory size by quotad
  • #1368841: Applications not calling glfs_h_poll_upcall() have upcall events cached for no use
  • #1368918: tests/bugs/cli/bug-1320388.t: Infrequent failures
  • #1368927: Error: quota context not set inode (gfid:nnn) [Invalid argument]
  • #1369042: thread CPU saturation limiting throughput on write workloads
  • #1369187: fix bug in protocol/client lookup callback
  • #1369328: [RFE] Add a count of snapshots associated with a volume to the output of the vol info command
  • #1369372: gluster snap status xml output shows incorrect details when the snapshots are in deactivated state
  • #1369517: rotated FUSE mount log is using to populate the information after log rotate.
  • #1369748: Memory leak with a replica 3 arbiter 1 configuration
  • #1370172: protocol/server: readlink rsp xdr failed while readlink got an error
  • #1370390: Locks xlators is leaking fdctx in pl_release()
  • #1371194: segment fault while join thread reaper_thr in fini()
  • #1371650: [Open SSL] : Unable to mount an SSL enabled volume via SMB v3/Ganesha v4
  • #1371912: gluster system:: uuid get hangs
  • #1372728: Node remains in stopped state in pcs status with "/usr/lib/ocf/resource.d/heartbeat/ganesha_mon: line 137: [: too many arguments ]" messages in logs.
  • #1373530: Minor improvements and cleanup for the build system
  • #1374290: "gluster vol status all clients --xml" doesn't generate xml if there is a failure in between
  • #1374565: [Bitrot]: Recovery fails of a corrupted hardlink (and the corresponding parent file) in a disperse volume
by on September 13, 2016

Making gluster play nicely with others

These days hyperconverged strategies are everywhere. But when you think about it, sharing the finite resources within a physical host requires an effective means of prioritisation and enforcement. Luckily, the Linux kernel already provides an infrastructure for this in the shape of cgroups, and the interface to these controls is now simplified with systemd integration.

So lets look at how you could use these capabilities to make Gluster a better neighbour in a collocated or hyperconverged  model. 

First some common systemd terms, we should to be familiar with;
slice : a slice is a concept that systemd uses to group together resources into a hierarchy. Resource constraints can then be applied to the slice, which defines 
  • how different slices may compete with each other for resources (e.g. weighting)
  • how resources within a slice are controlled (e.g. cpu capping)
unit : a systemd unit is a resource definition for controlling a specific system service
NB. More information about control groups with systemd can be found here

In this article, I'm keeping things simple by implementing a cpu cap on glusterfs processes. Hopefully, the two terms above are big clues, but conceptually it breaks down into two main steps;
  1. define a slice which implements a CPU limit
  2. ensure gluster's systemd unit(s) start within the correct slice.
So let's look at how this is done.

Defining a slice

Slice definitions can be found under /lib/systemd/system, but systemd provides a neat feature where /etc/systemd/system can be used provide local "tweaks". This override directory is where we'll place a slice definition. Create a file called glusterfs.slice, containing;

[Slice]
CPUQuota=200%

CPUQuota is our means of applying a cpu limit on all resources running within the slice. A value of 200% defines a 2 cores/execution threads limit.

Updating glusterd


Next step is to give gluster a nudge so that it shows up in the right slice. If you're using RHEL7 or Centos7, cpu accounting may be off by default (you can check in /etc/systemd/system.conf). This is OK, it just means we have an extra parameter to define. Follow these steps to change the way glusterd is managed by systemd

# cd /etc/systemd/system
# mkdir glusterd.service.d
# echo -e "[Service]\nCPUAccounting=true\nSlice=glusterfs.slice" > glusterd.service.d/override.conf

glusterd is responsible for starting the brick and self heal processes, so by ensuring glusterd starts in our cpu limited slice, we capture all of glusterd's child processes too. Now the potentially bad news...this 'nudge' requires a stop/start of gluster services. If your doing this on a live system you'll need to consider quorum, self heal etc etc. However, with the settings above in place, you can get gluster into the right slice by;

# systemctl daemon-reload
# systemctl stop glusterd
# killall glusterfsd && killall glusterfs
# systemctl daemon-reload
# systemctl start glusterd


You can see where gluster is within the control group hierarchy by looking at it's runtime settings

# systemctl show glusterd | grep slice
Slice=glusterfs.slice
ControlGroup=/glusterfs.slice/glusterd.service
Wants=glusterfs.slice
After=rpcbind.service glusterfs.slice systemd-journald.socket network.target basic.target

or use the systemd-cgls command to see the whole control group hierarchy

├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 19
├─glusterfs.slice
│ └─glusterd.service
│   ├─ 867 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
│   ├─1231 /usr/sbin/glusterfsd -s server-1 --volfile-id repl.server-1.bricks-brick-repl -p /var/lib/glusterd/vols/repl/run/server-1-bricks-brick-repl.pid 

 │   └─1305 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log
├─user.slice
│ └─user-0.slice
│   └─session-1.scope
│     ├─2075 sshd: root@pts/0  
│     ├─2078 -bash
│     ├─2146 systemd-cgls
│     └─2147 less
└─system.slice

At this point gluster is exactly where we want it! 

Time for some more systemd coolness ;) The resource constraints that are applied by the slice are dynamic, so if you need more cpu, you're one command away from getting it;

# systemctl set-property glusterfs.slice CPUQuota=350%

Try the 'systemd-cgtop' command to show the cpu usage across the complete control group hierarchy.

Now if jumping straight into applying resource constraints to gluster is a little daunting, why not test this approach with a tool like 'stress'. Stress is designed to simply consume components of the system - cpu, memory, disk. Here's an example .service file which uses stress to consume 4 cores

[Unit]
Description=CPU soak task

[Service]
Type=simple
CPUAccounting=true
ExecStart=/usr/bin/stress -c 4
Slice=glusterfs.slice

[Install]
WantedBy=multi-user.target

Now you can tweak the service, and the slice with different thresholds before you move on to bigger things! Use stress to avoid stress :)

And now the obligatory warning. Introducing any form of resource constraint may resort in unexpected outcomes especially in hyperconverged/collocated systems - so adequate testing is key.

With that said...happy hacking :)




by on September 10, 2016

10 minutes introduction to Gluster Eventing Feature

Demo video is included in the end, or you can directly watch it on Youtube

Gluster Eventing is the new feature as part of Gluster.Next initiatives, it provides close to realtime notification and alerts for the Gluster cluster state changes.

Websockets APIs to consume events will be added later. Now we emit events via another popular mechanism called "Webhooks".(Many popular products provide notifications via Webhooks Github, Atlassian, Dropbox, and many more)

Webhooks are similar to callbacks(over HTTP), on event Gluster will call the Webhook URL(via POST) which is configured. Webhook is a web server which listens on a URL, this can be deployed outside of the Cluster. Gluster nodes should be able to access this Webhook server on the configured port. We will discuss about adding/testing webhook later.

Example Webhook written in python,

from flask import Flask, request

app = Flask(__name__)

@app.route("/listen", methods=["POST"])
def events_listener():
    gluster_event = request.json
    if gluster_event is None:
        # No event to process, may be test call
        return "OK"

    # Process gluster_event
    # {
    #  "nodeid": NODEID,
    #  "ts": EVENT_TIMESTAMP,
    #  "event": EVENT_TYPE,
    #  "message": EVENT_DATA
    # }
    return "OK"

app.run(host="0.0.0.0", port=9000)

Eventing feature is not yet available in any of the releases, patch is under review in upstream master(http://review.gluster.org/14248). If anybody interested in trying it out can cherrypick the patch from review.gluster.org

git clone http://review.gluster.org/glusterfs
cd glusterfs
git fetch http://review.gluster.org/glusterfs refs/changes/48/14248/5
git checkout FETCH_HEAD
git checkout -b <YOUR_BRANCH_NAME>
./autogen.sh
./configure
make
make install

Start the Eventing using,

gluster-eventing start

Other commands available are stop, restart, reload and status. gluster-eventing --help for more details.

Now Gluster can send out notifications via Webhooks. Setup a web server listening to a POST request and register that URL to Gluster Eventing. Thats all.

gluster-eventing webhook-add <MY_WEB_SERVER_URL>

For example, if my webserver is running at http://192.168.122.188:9000/listen then register using,

gluster-eventing webhook-add ``http://192.168.122.188:9000/listen``

We can also test if web server is accessible from all Gluster nodes using webhook-test subcommand.

gluster-eventing webhook-test http://192.168.122.188:9000/listen

With the initial patch only basic events are covered, I will add more events once this patch gets merged. Following events are available now.

Volume Create
Volume Delete
Volume Start
Volume Stop
Peer Attach
Peer Detach

Created a small demo to show this eventing feature, it uses Web server which is included with the patch for Testing.(laptop hostname is sonne)

/usr/share/glusterfs/scripts/eventsdash.py --port 8080

Login to Gluster node and start the eventing,

gluster-eventing start
gluster-eventing webhook-add http://sonne:8080/listen

And then login to VM and run Gluster commands to probe/detach peer, volume create, start etc and Observe the realtime notifications for the same where eventsdash is running.

Example,

ssh root@fvm1
gluster peer attach fvm2
gluster volume create gv1 fvm1:/bricks/b1 fvm2:/bricks/b2 force
gluster volume start gv1
gluster volume stop gv1
gluster volume delete gv1
gluster peer detach fvm2

Demo also includes a Web UI which refreshes its UI automatically when something changes in Cluster.(I am still fine tuning this UI, not yet available with the patch. But soon will be available as seperate repo in my github)

FAQ:

  • Will this feature available in 3.8 release?

    Sadly No. I couldn't get this merged before 3.8 feature freeze :(

  • Is it possible to create a simple Gluster dashboard outside the cluster?

    It is possible, along with the events we also need REST APIs to get more information from cluster or to perform any action in cluster. (WIP REST APIs are available here)

  • Is it possible to filter only alerts or critical notifications?

    Thanks Kotresh for the suggestion. Yes it is possible to add event_type and event_group information to the dict so that it can be filtered easily.(Not yet available now, but will add this feature once this patch gets merged in Master)

  • Is documentation available to know more about eventing design and internals?

    Design spec available here (which discusses about Websockets, currently we don't have Websockets support). Usage documentation is available in the commit message of the patch(http://review.gluster.org/14248).

Comments and Suggestions Welcome.

by on September 2, 2016

delete, info, config : GlusterFS Snapshots CLI Part 2

Now that we know how to create GlusterFS snapshots, it will be handy to know, how to delete them as well.
Right now I have a cluster with two volumes at my disposal. As can be seen below, each volume has 1 brick.

# gluster volume info

Volume Name: test_vol
Type: Distribute
Volume ID: 74e21265-7060-48c5-9f32-faadaf986d85
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: VM1:/brick/brick-dirs1/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

Volume Name: test_vol1
Type: Distribute
Volume ID: b6698e0f-748f-4667-8956-ec66dd91bd84
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: VM2:/brick/brick-dirs/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

We are going to take a bunch of snapshots for both these volumes using the create command.

# gluster snapshot create snap1 test_vol no-timestamp
snapshot create: success: Snap snap1 created successfully
# gluster snapshot create snap2 test_vol no-timestamp
snapshot create: success: Snap snap2 created successfully
# gluster snapshot create snap3 test_vol no-timestamp
snapshot create: success: Snap snap3 created successfully
# gluster snapshot create snap4 test_vol1 no-timestamp
snapshot create: success: Snap snap4 created successfully
# gluster snapshot create snap5 test_vol1 no-timestamp
snapshot create: success: Snap snap5 created successfully
# gluster snapshot create snap6 test_vol1 no-timestamp
snapshot create: success: Snap snap6 created successfully
# gluster snapshot list
snap1
snap2
snap3
snap4
snap5
snap6
#

Now we have 3 snapshots for each volume. To delete a snapshot we have to use the delete command along with the snap name.

# gluster snapshot delete snap1
Deleting snap will erase all the information about the snap. Do you still want to continue? (y/n) y
snapshot delete: snap1: snap removed successfully
# gluster snapshot list
snap2
snap3
snap4
snap5
snap6
#

We can also choose to delete all snapshots that belong to a particular volume. Before doing that let’s see what snapshots are present for volume “test_vol”. Apart from snapshot list, there is also snapshot info command that provides more elaborate details of snapshots. Like snapshot list, snapshot info can also take volume name as an option to show information of snapshots belonging to only that volume.

# gluster snapshot list test_vol
snap2
snap3
# gluster snapshot info volume test_vol
Volume Name               : test_vol
Snaps Taken               : 2
Snaps Available           : 254
Snapshot                  : snap2
Snap UUID                 : d17fbfac-1cb1-4276-9b96-0b73b90fb545
Created                   : 2016-07-15 09:32:07
Status                    : Stopped

Snapshot                  : snap3
Snap UUID                 : 0f319761-eca2-491e-b678-75b56790f3a0
Created                   : 2016-07-15 09:32:12
Status                    : Stopped
#

As we can see from both list and info command, test_vol  has 2 snapshots snap2, and snap3. Instead of individually deleting these snapshots one by one, we can choose to delete all snapshots that belong to a particular volume, in this case test_vol.

# gluster snapshot delete volume test_vol
Volume (test_vol) contains 2 snapshot(s).
Do you still want to continue and delete them?  (y/n) y
snapshot delete: snap2: snap removed successfully
snapshot delete: snap3: snap removed successfully
#
# gluster snapshot list
snap4
snap5
snap6
# gluster snapshot list test_vol
No snapshots present
# gluster snapshot info volume test_vol
Volume Name               : test_vol
Snaps Taken               : 0
Snaps Available           : 256
#

With the above volume option we successfully deleted both the snapshots of test_vol with a single command. Now only 3 snapshots remain, and both belong to volume “test_vol1”. Before proceeding further let’s create one more snapshot for volume “test_vol”.

# gluster snapshot create snap7 test_vol no-timestamp
snapshot create: success: Snap snap7 created successfully
# gluster snapshot list
snap4
snap5
snap6
snap7
#

With this, we have four snapshots belonging, three of which belong to test_vol1, and one belongs to test_vol. Now with the ‘delete all’  command we will be able to delete all snapshots present irrespective of which volumes they belong to.

 # gluster snapshot delete all
System contains 4 snapshot(s).
Do you still want to continue and delete them?  (y/n) y
snapshot delete: snap4: snap removed successfully
snapshot delete: snap5: snap removed successfully
snapshot delete: snap6: snap removed successfully
snapshot delete: snap7: snap removed successfully
# gluster snapshot list
No snapshots present
#

So that is how you delete GlusterFS snapshots. There are some configurable options for Gluster snapshots, which can be viewed and modified using the snapshot config option.

# gluster snapshot config

Snapshot System Configuration:
snap-max-hard-limit : 256
snap-max-soft-limit : 90%
auto-delete : disable
activate-on-create : disable

Snapshot Volume Configuration:

Volume : test_vol
snap-max-hard-limit : 256
Effective snap-max-hard-limit : 256
Effective snap-max-soft-limit : 230 (90%)

Volume : test_vol1
snap-max-hard-limit : 256
Effective snap-max-hard-limit : 256
Effective snap-max-soft-limit : 230 (90%)
#

Just running the config option, as shown above displays the current configuration in the system. What we are looking at are the default configuration values. There are four different configurable parameters. Let’s go through them one by one.

  • snap-max-hard-limit: Set by default as 256, the snap-max-hard-limit is the maximum number of snapshots that can be present in the system. Once a volume reaches this limit, in terms of the number of snapshots it has, we are not allowed to create any more snapshot, unless we either delete a snapshot, or increase this limit.

    # gluster snapshot config test_vol snap-max-hard-limit 2
    Changing snapshot-max-hard-limit will limit the creation of new snapshots if they exceed the new limit.
    Do you want to continue? (y/n) y
    snapshot config: snap-max-hard-limit for test_vol set successfully
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 256
    snap-max-soft-limit : 90%
    auto-delete : disable
    activate-on-create : disable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 2
    Effective snap-max-hard-limit : 2
    Effective snap-max-soft-limit : 1 (90%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 256
    Effective snap-max-soft-limit : 230 (90%)
    #
    #
    # gluster snapshot info volume test_vol
    Volume Name               : test_vol
    Snaps Taken               : 0
    Snaps Available           : 2
    #

    As can be seen with the config option, I have modified the snap-max-hard-limit for the volume test_vol to 2. This means after taking 2 snapshots it will not allow me to take any more snapshots, till I either delete one of them, or increase this value. See how the snapshot info for the volume test_vol shows ‘Snaps Available’ as 2.

    # gluster snapshot create snap1 test_vol no-timestamp
    snapshot create: success: Snap snap1 created successfully
    # gluster snapshot create snap2 test_vol no-timestamp
    snapshot create: success: Snap snap2 created successfully
    Warning: Soft-limit of volume (test_vol) is reached. Snapshot creation is not possible once hard-limit is reached.
    #
    #
    # gluster snapshot info volume test_vol
    Volume Name               : test_vol
    Snaps Taken               : 2
    Snaps Available           : 0
    Snapshot                  : snap1
    Snap UUID                 : 2ee5f237-d4d2-47a6-8a0c-53a887b33b26
    Created                   : 2016-07-15 10:12:55
    Status                    : Stopped

    Snapshot                  : snap2
    Snap UUID                 : 2c74925e-4c75-4824-b39e-7e1e22f3b758
    Created                   : 2016-07-15 10:13:02
    Status                    : Stopped

    #
    # gluster snapshot create snap3 test_vol no-timestamp
    snapshot create: failed: The number of existing snaps has reached the effective maximum limit of 2, for the volume (test_vol). Please delete few snapshots before taking further snapshots.
    Snapshot command failed
    #

    What we have done above is we created 2 snapshots for the volume test_vol and we reached it’s snap-max-hard-limit. Notice two things here, first is when we created the second snapshot it gave us a warning that the soft-limit is reached for this volume (we will come to the soft-limit in a while), and second is that the ‘Snaps Available’ in snapshot info has now become 0. As explained, when we try to take the third snapshot it fails to do so, while explaining that we have reached the maximum limit, and asking us to delete a few snapshots.

    # gluster snapshot delete snap1
    Deleting snap will erase all the information about the snap. Do you still want to continue? (y/n) y
    snapshot delete: snap1: snap removed successfully
    # gluster snapshot create snap3 test_vol no-timestamp
    snapshot create: success: Snap snap3 created successfully
    Warning: Soft-limit of volume (test_vol) is reached. Snapshot creation is not possible once hard-limit is reached.
    #
    # gluster snapshot config test_vol snap-max-hard-limit 3
    Changing snapshot-max-hard-limit will limit the creation of new snapshots if they exceed the new limit.
    Do you want to continue? (y/n) y
    snapshot config: snap-max-hard-limit for test_vol set successfully
    # gluster snapshot info volume test_vol
    Volume Name               : test_vol
    Snaps Taken               : 2
    Snaps Available           : 1
    Snapshot                  : snap2
    Snap UUID                 : 2c74925e-4c75-4824-b39e-7e1e22f3b758
    Created                   : 2016-07-15 10:13:02
    Status                    : Stopped

    Snapshot                  : snap3
    Snap UUID                 : bfd080f3-848e-490a-83ed-066858bd96fc
    Created                   : 2016-07-15 10:19:17
    Status                    : Stopped

    # gluster snapshot create snap4 test_vol no-timestamp
    snapshot create: success: Snap snap4 created successfully
    Warning: Soft-limit of volume (test_vol) is reached. Snapshot creation is not possible once hard-limit is reached.
    #

    As seen above, once we delete a snapshot the system allows us to create another one. It also allows us to do so when we increase the snap-max-hard-limit. I am curious to see what happens when we have hit the snap-max-hard-limit, and I go ahead and further decrease the limit. Does the system delete snapshots to bring the number of snapshots to the set limit.

    # gluster snapshot config test_vol snap-max-hard-limit 1
    Changing snapshot-max-hard-limit will limit the creation of new snapshots if they exceed the new limit.
    Do you want to continue? (y/n) y
    snapshot config: snap-max-hard-limit for test_vol set successfully
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 256
    snap-max-soft-limit : 90%
    auto-delete : disable
    activate-on-create : disable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 1
    Effective snap-max-hard-limit : 1
    Effective snap-max-soft-limit : 0 (90%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 256
    Effective snap-max-soft-limit : 230 (90%)
    # gluster snapshot info volume test_vol
    Volume Name               : test_vol
    Snaps Taken               : 3
    Snaps Available           : 0
    Snapshot                  : snap2
    Snap UUID                 : 2c74925e-4c75-4824-b39e-7e1e22f3b758
    Created                   : 2016-07-15 10:13:02
    Status                    : Stopped

    Snapshot                  : snap3
    Snap UUID                 : bfd080f3-848e-490a-83ed-066858bd96fc
    Created                   : 2016-07-15 10:19:17
    Status                    : Stopped

    Snapshot                  : snap4
    Snap UUID                 : bd9a5297-0eb5-47d1-b250-9b57f4e57427
    Created                   : 2016-07-15 10:20:08
    Status                    : Stopped

    #
    # gluster snapshot create snap5 test_vol no-timestamp
    snapshot create: failed: The number of existing snaps has reached the effective maximum limit of 1, for the volume (test_vol). Please delete few snapshots before taking further snapshots.
    Snapshot command failed
    #

    So the answer to that question is a big NO. We don’t explicitly delete snapshots when you decrease the snap-max-hard-limit to a number below the current number of snapshots. The reason for not doing so, is it will become very easy to lose important snapshots. However, what we do is, we do not allow you to create snapshots, till you… (yeah you guessed it right), either delete a snapshot or increase the snap-max-hard-limit.

    snap-max-hard-limit is both a system config and a volume config. What it means is we can set this value for indiviudal volumes, and we can also set a system value.

    # gluster snapshot config snap-max-hard-limit 10
    Changing snapshot-max-hard-limit will limit the creation of new snapshots if they exceed the new limit.
    Do you want to continue? (y/n) y
    snapshot config: snap-max-hard-limit for System set successfully
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 10
    snap-max-soft-limit : 90%
    auto-delete : disable
    activate-on-create : disable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 1
    Effective snap-max-hard-limit : 1
    Effective snap-max-soft-limit : 0 (90%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 10
    Effective snap-max-soft-limit : 9 (90%)
    #

    Notice, how not mentioning a volume name for a snapshot config, sets that particular config for the whole system, instead of a particular volume. The same is clearly visible in the ‘Snapshot System Configuration’ section of the snapshot config output. Look at this system option as an umbrella limit for the entire cluster. You are allowed to still configure individual volume’s snap-max-hard-limit. If the individual volume’s limit is lesser than the system’s limit, then it will be honored, else the system limit will be honored.

    For example, we can see that the system snap-max-hard-limit is set to 10. Now, in case of the volume test_vol, the snap-max-hard-limit for the volume is set to 1, which is lower than the system’s limit and is hence honored, making the effective snap-max-hard-limit as 1. This effective snap-max-hard-limit is the limit that is taken into consideration during snapshot create, and is displayed as ‘Snaps Available’ in snapshot info. Similarly, for volume test_vol1, the snap-max-hard-limit is 256, which is higher than the system’s limit, and is hence not honored, making the effective snap-max-hard-limit of that volume as 10, which is the system’s snap-max-hard-limit. Pretty intuitive huh!!!

  • snap-max-soft-limit: This option is set as a percentage (of snap-max-hard-limit), and as we have seen in examples above, on crossing this limit, a warning is shown saying the soft-limit is reached. It serves as a reminder to the user, that he is nearing the hard-limit and should do something about it in order to be able to keep on taking snapshots. By default the snap-max-hard-limit is set to 90%, and can be modified using the snapshot config option.

    # gluster snapshot config test_vol snap-max-soft-limit 50
    Soft limit cannot be set to individual volumes.
    Usage: snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])| ([activate-on-create <enable|disable>])
    #

    So what do we have here… Yes, the snap-max-soft-limit is a system option only and cannot be set to individual volumes. When the snap-max-soft-limit option is set for the system, it applies on the effective snap-max-hard-limit of individual volumes, to get the effective snap-max-soft-limit of those respective volumes.

    # gluster snapshot config snap-max-soft-limit 50
    If Auto-delete is enabled, snap-max-soft-limit will trigger deletion of oldest snapshot, on the creation of new snapshot, when the snap-max-soft-limit is reached.
    Do you want to change the snap-max-soft-limit? (y/n) y
    snapshot config: snap-max-soft-limit for System set successfully
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 10
    snap-max-soft-limit : 50%
    auto-delete : disable
    activate-on-create : disable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 1
    Effective snap-max-hard-limit : 1
    Effective snap-max-soft-limit : 0 (50%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 10
    Effective snap-max-soft-limit : 5 (50%)
    #

    As we can see above, on setting the option for the system, it applies to the individual volume’s (see test_vol1) snap-max-soft-limit, to procure that particular volume’s snap-max-soft-limit.

    I am sure the keen-eyed observer in you has noticed, the Auto-delete warning in the output above, and it’s just as well because it is our third configurable parameter.

  • auto-delete: This option is tightly tied with snap-max-soft-limit, or rather effective snap-max-soft-limit of individual volumes. It is however a system option and cannot be set for individual volumes. On enabling this option, once we exceed the effective snap-max-soft-limit, of a particular volume, we automatically delete the oldest snapshot for this volume, making sure the total number of snapshots don’t increase the effective snap-max-soft-limit, and never reach the effective snap-max-hard-limit, enabling you to keep taking snapshots without hassle.

    NOTE: Extreme Caution Should Be Exercised When Enabling This Option, As It Automatically Deletes The Oldest Snapshot Of A Volume, When The Number Of Snapshots For That Volume Exceeds The Effective snap-max-soft-limit Of That Volume.

    # gluster snapshot config auto-delete enable
    snapshot config: auto-delete successfully set
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 10
    snap-max-soft-limit : 50%
    auto-delete : enable
    activate-on-create : disable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 1
    Effective snap-max-hard-limit : 1
    Effective snap-max-soft-limit : 0 (50%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 10
    Effective snap-max-soft-limit : 5 (50%)
    #
    # gluster snapshot list
    snap2
    snap3
    snap4
    # gluster snapshot delete all
    System contains 3 snapshot(s).
    Do you still want to continue and delete them?  (y/n) y
    snapshot delete: snap2: snap removed successfully
    snapshot delete: snap3: snap removed successfully
    snapshot delete: snap4: snap removed successfully
    # gluster snapshot create snap1 test_vol1 no-timestamp
    snapshot create: success: Snap snap1 created successfully
    # gluster snapshot create snap2 test_vol1 no-timestamp
    snapshot create: success: Snap snap2 created successfully
    # gluster snapshot create snap3 test_vol1 no-timestamp
    snapshot create: success: Snap snap3 created successfully
    # gluster snapshot create snap4 test_vol1 no-timestamp
    snapshot create: success: Snap snap4 created successfully
    # gluster snapshot create snap5 test_vol1 no-timestamp
    snapshot create: success: Snap snap5 created successfully

    In the above example, we first set the auto-delete option in snapshot config,  followed by deleting all the snapshots currently in the system. Then we create 5 snapshots for test_vol1, whose effective snap-max-soft-limit is 5. On creating one more snapshot, we will exceed the limit, and the oldest snapshot will be deleted.

    # gluster snapshot create snap6 test_vol1 no-timestamp
    snapshot create: success: Snap snap6 created successfully
    #
    # gluster snapshot list volume test_vol1
    snap2
    snap3
    snap4
    snap5
    snap6
    #

    As soon as we create snap6, the total number of snapshots become 6, thus exceeding the effective snap-max-soft-limit for test_vol1. The oldest snapshot for test_vol1(which is snap1) is then deleted in the background,  bringing the total number of snapshots to 5.

  • activate-on-create: As we discussed during creation of snapshot, a snapshot on creation is in deactivated state by default, and needs to be activated to be used. On enabling this option in snapshot config, every snapshot created thereafter, will be activated by default. This too is a system option, and cannot be set for individual volumes.

    # gluster snapshot status snap6

    Snap Name : snap6
    Snap UUID : 7fc0a0e7-950d-4c1b-913d-caea6037e633

    Brick Path        :   VM2:/var/run/gluster/snaps/db383315d5a448d6973f71ae3e45573e/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   No
    Brick PID         :   N/A
    Data Percentage   :   1.80
    LV Size           :   616.00m

    #
    # gluster snapshot config activate-on-create enable
    snapshot config: activate-on-create successfully set
    # gluster snapshot config

    Snapshot System Configuration:
    snap-max-hard-limit : 10
    snap-max-soft-limit : 50%
    auto-delete : enable
    activate-on-create : enable

    Snapshot Volume Configuration:

    Volume : test_vol
    snap-max-hard-limit : 1
    Effective snap-max-hard-limit : 1
    Effective snap-max-soft-limit : 0 (50%)

    Volume : test_vol1
    snap-max-hard-limit : 256
    Effective snap-max-hard-limit : 10
    Effective snap-max-soft-limit : 5 (50%)
    # gluster snapshot create snap7 test_vol1 no-timestamp
    snapshot create: success: Snap snap7 created successfully
    # gluster snapshot status snap7

    Snap Name : snap7
    Snap UUID : b1864a86-1fa4-4d42-b20a-3d95c2f9e277

    Brick Path        :   VM2:/var/run/gluster/snaps/38b1d9a2f3d24b0eb224f142ae5d33ca/brick1/brick
    Volume Group      :   snap_lvgrp
    Brick Running     :   Yes
    Brick PID         :   6731
    Data Percentage   :   1.80
    LV Size           :   616.00m

    #

    As can be seen when this option was disabled, snap6 wasn’t activated by default. After enabling this option, snap7 on creation was in activated state. In the next post we will be discussing snapshot restore and snapshot clone.

(more…)

by on August 30, 2016

Run Gluster systemd containers [without privileged mode] in Fedora/CentOS

Today we will discuss about how to run gluster systemd containers without ‘privilege’ mode !! Awesome .. Isnt it ?

I owe this blog to few people latest being twitter.com/dglushenok/status/740265552258682882
Here is some details about my docker host setup:
[root@dhcp35-111 ~]# cat /etc/redhat-release
Fedora release 24 (Twenty Four)
[root@dhcp35-111 ~]# docker version
Client:
Version: 1.10.3
API version: 1.22
Package version: docker-1.10.3-21.git19b5791.fc24.x86_64
Go version: go1.6.2
Git commit: 19b5791/1.10.3
Built:
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Package version: docker-1.10.3-21.git19b5791.fc24.x86_64
Go version: go1.6.2
Git commit: 19b5791/1.10.3
Built:
OS/Arch: linux/amd64
[root@dhcp35-111 ~]#

I have pulled gluster/gluster-centos image from docker hub and kept in my docker image registry.

[root@dhcp35-111 ~]# docker images |grep gluster
docker.io/gluster/gluster-centos latest 759691b0beca 4 days ago 406.1 MB
gluster/gluster-centos experiment fd8cd51f47fb 2 weeks ago 351.2 MB
gluster/gluster-centos latest 9b46174d3366 3 weeks ago 351.1 MB
gluster/gluster-centos gluster_3_7_centos_7 5809addca906 4 weeks ago 351.1 MB

The beauty is that we don’t need any extra steps to be performed in our host system.

NOTE: We havent submitted ‘privileged’ flag/option with below ‘docker run’ command. The volume mounts like ‘/etc/glusterfs’, ‘/var/lib/glusterd’, ‘/var/log/glusterfs’..etc are kept for glusterfs metadata and logs to be persistent across container spawning.


[root@dhcp35-111 docker-host]# docker run -d --name gluster3 -v /etc/glusterfs:/etc/glusterfs:z -v /var/lib/glusterd:/var/lib/glusterd:z -v /var/log/glusterfs:/var/log/glusterfs:z -v /sys/fs/cgroup:/sys/fs/cgroup:ro gluster/gluster-centos
8b1dd6f0aa88197bdcd022802f7c0c16d642630a21b2b43accfa5ed8023c197a
[root@dhcp35-111 docker-host]#

As we now have the container id ( 8b1dd6f0aa88197bdcd022802f7c0c16d642630a21b2b43accfa5ed8023c197a), let’s get inside the container and examine the service and its behavior.

[root@dhcp35-111 docker-host]# docker exec -ti 8b1dd6f0aa88197bdcd022802f7c0c16d642630a21b2b43accfa5ed8023c197a /bin/bash
[root@8b1dd6f0aa88 /]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 122764 4688 ? Ss 13:34 0:00 /usr/sbin/init
root 22 0.0 0.0 36832 6348 ? Ss 13:34 0:00 /usr/lib/systemd/systemd-journald
root 23 0.0 0.0 118492 2744 ? Ss 13:34 0:00 /usr/sbin/lvmetad -f
root 29 0.0 0.0 24336 2884 ? Ss 13:34 0:00 /usr/sbin/crond -n
rpc 42 0.0 0.0 64920 3244 ? Ss 13:34 0:00 /sbin/rpcbind -w
root 44 0.0 0.2 430272 17300 ? Ssl 13:34 0:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
root 68 0.0 0.0 82572 6212 ? Ss 13:34 0:00 /usr/sbin/sshd -D
root 197 0.0 0.0 11788 2952 ? Ss 13:35 0:00 /bin/bash
root 219 0.0 0.0 47436 3360 ? R+ 13:44 0:00 ps aux
[root@8b1dd6f0aa88 /]#
[root@8b1dd6f0aa88 /]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-06-28 13:34:53 UTC; 27s ago
Process: 43 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 44 (glusterd)
CGroup: /system.slice/docker-8b1dd6f0aa88197bdcd022802f7c0c16d642630a21b2b43accfa5ed8023c197a.scope/system.slice/glusterd.service
└─44 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
Jun 28 13:34:51 8b1dd6f0aa88 systemd[1]: Starting GlusterFS, a clustered file-system server...
Jun 28 13:34:53 8b1dd6f0aa88 systemd[1]: Started GlusterFS, a clustered file-system server.
Jun 28 13:35:15 8b1dd6f0aa88 systemd[1]: Started GlusterFS, a clustered file-system server.
[root@8b1dd6f0aa88 /]#
[root@8b1dd6f0aa88 /]# glusterd --version
glusterfs 3.7.11 built on Apr 18 2016 13:20:46
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@8b1dd6f0aa88 /]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@8b1dd6f0aa88 /]# rpm -qa |grep glusterfs
glusterfs-3.7.11-1.el7.x86_64
glusterfs-fuse-3.7.11-1.el7.x86_64
glusterfs-cli-3.7.11-1.el7.x86_64
glusterfs-libs-3.7.11-1.el7.x86_64
glusterfs-client-xlators-3.7.11-1.el7.x86_64
glusterfs-api-3.7.11-1.el7.x86_64
glusterfs-server-3.7.11-1.el7.x86_64
glusterfs-geo-replication-3.7.11-1.el7.x86_64
[root@8b1dd6f0aa88 /]#

Let’s examine this container from docker host and verify these containers are running without privileged mode.

[root@dhcp35-111 docker-host]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8b1dd6f0aa88 gluster/gluster-centos "/usr/sbin/init" 6 minutes ago Up 6 minutes 111/tcp, 245/tcp, 443/tcp, 2049/tcp, 2222/tcp, 6010-6012/tcp, 8080/tcp, 24007/tcp, 38465-38466/tcp, 38468-38469/tcp, 49152-49154/tcp, 49156-49162/tcp gluster3
[root@dhcp35-111 docker-host]# docker inspect 8b1dd6f0aa88|grep -i privil
"Privileged": false,
[root@dhcp35-111 docker-host]#

All is well, but what will be missing if you run these containers without ‘privilged’ mode? Not much! However, if you want to create gluster snapshots from container we may need to export ‘/dev/’ to the container and operations to create devices from container need privileged mode.

by on August 26, 2016

Possible configurations of GlusterFS in Kubernetes/OpenShift setup

In previous blog posts we discussed, how to use GlusterFS as a persistent storage in Kubernetes and Openshift. In nutshell, the GlusterFS can be deployed/used in a kubernetes/openshift environment as : *) Contenarized GlusterFS ( Pod ) *) GlusterFS as Openshift service and Endpoint (Service and Endpoint). *) GlusterFS volume as Persistent Volume (PV) and using GlusterFS volume plugin to bind this PV to a Persistent Volume Claim ( PVC) *) GlusterFS template to deploy GlusterFS pods in an Openshift Environment. All the configuration files that can be used to deploy GlusterFS can be found @ github.com/humblec/glusterfs-kubernetes-openshift/ or github.com/gluster/glusterfs-kubernetes-openshift. Lets see how to use these files to deploy GlusterFS in kubernetes and Openshift. We will start with Deploying GlusterFS pods in an Openshift/Kubernetes Environment. Deploying GlusterFS Pod:
[Update] The pod file is renamed to gluster-pod.yaml in the mentioned repo. More details about Gluster Containers can be found @http://www.slideshare.net/HumbleChirammal/gluster-containers
GlusterFS pods can be deployed in Kubernetes/Openshift, so that Gluster Nodes are deployed in containers and it can provide persistent storage for Openshift/Kubernetes setup. The examples files in this repo are used for this demo. Step 1: Create GlusterFS pod [root@atomic-node2 gluster_pod]# oc create -f gluster-1.yaml Step 2: Get details about the GlusterFS pod. [root@atomic-node2 gluster_pod]# oc describe pod gluster-1 Name: gluster-1 Namespace: default Image(s): gluster/gluster-centos Node: atomic-node1/10.70.43.174 Start Time: Tue, 17 May 2016 10:19:17 +0530 Labels: name=gluster-1 Status: Running Reason: Message: IP: 10.70.43.174 Replication Controllers: Containers: glusterfs: Container ID: docker://ff8f4af700d725dfe0e08939ec011c34ddf9dedc7204e0ced1cc355a56150742 Image: gluster/gluster-centos Image ID: docker://033de9c44a8aabde55ce8a2b751ccf5bc345fdb534ea30e79a8fa70b82dc7761 QoS Tier: cpu: BestEffort memory: BestEffort State: Running Started: Tue, 17 May 2016 10:20:35 +0530 Ready: True Restart Count: 0 Environment Variables: Conditions: Type Status Ready True Volumes: brickpath: Type: HostPath (bare host directory volume) Path: /mnt/brick1 default-token-72d89: Type: Secret (a secret that should populate this volume) SecretName: default-token-72d89 Events: FirstSeen LastSeen Count From SubobjectPath Reason Message ───────── ──────── ───── ──── ───────────── ────── ─────── 1m 1m 1 {scheduler } Scheduled Successfully assigned gluster-1 to atomic-node1 1m 1m 1 {kubelet atomic-node1} implicitly required container POD Pulled Container image "openshift3/ose-pod:v3.1.1.6" already present on machine 1m 1m 1 {kubelet atomic-node1} implicitly required container POD Created Created with docker id f55ce55e6ea3 1m 1m 1 {kubelet atomic-node1} implicitly required container POD Started Started with docker id f55ce55e6ea3 1m 1m 1 {kubelet atomic-node1} spec.containers{glusterfs} Pulling pulling image "gluster/gluster-centos" 8s 8s 1 {kubelet atomic-node1} spec.containers{glusterfs} Pulled Successfully pulled image "gluster/gluster-centos" 8s 8s 1 {kubelet atomic-node1} spec.containers{glusterfs} Created Created with docker id ff8f4af700d7 8s 8s 1 {kubelet atomic-node1} spec.containers{glusterfs} Started Started with docker id ff8f4af700d7 From above logs, you can see it pulled `gluster/gluster-centos` container image and deployed containers from it. [root@atomic-node2 gluster_pod]# oc get pods NAME READY STATUS RESTARTS AGE gluster-1 1/1 Running 0 1m Examine the container and make sure it has a running GlusterFS daemon. [root@atomic-node2 gluster_pod]# oc exec -ti gluster-1 /bin/bash Examine the processes running in this container and the `glusterd` service information. [root@atomic-node1 /]# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.4 0.0 40780 2920 ? Ss 04:50 0:00 /usr/sbin/init root 20 0.3 0.0 36816 4272 ? Ss 04:50 0:00 /usr/lib/syste root 21 0.0 0.0 118476 1332 ? Ss 04:50 0:00 /usr/sbin/lvme root 37 0.0 0.0 101344 1228 ? Ssl 04:50 0:00 /usr/sbin/gssp rpc 44 0.1 0.0 64904 1052 ? Ss 04:50 0:00 /sbin/rpcbind root 209 0.1 0.1 364716 13444 ? Ssl 04:50 0:00 /usr/sbin/glus root 341 1.1 0.0 13368 1964 ? Ss 04:51 0:00 /bin/bash root 354 0.0 0.0 49020 1820 ? R+ 04:51 0:00 ps aux [root@atomic-node1 /]# service glusterd status Redirecting to /bin/systemctl status glusterd.service ● glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2016-05-17 04:50:41 UTC; 35s ago Process: 208 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 209 (glusterd) CGroup: /system.slice/docker-ff8f4af700d725dfe0e08939ec011c34ddf9dedc7204e0ced1cc355a56150742.scope/system.slice/glusterd.service └─209 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO... ‣ 209 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO... May 17 04:50:36 atomic-node1 systemd[1]: Starting Gluste... May 17 04:50:41 atomic-node1 systemd[1]: Started Gluster... Hint: Some lines were ellipsized, use -l to show in full. Let's fetch some more details about GlusterFS in this container. [root@atomic-node1 /]# gluster --version glusterfs 3.7.9 built on Mar 20 2016 03:19:49 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@atomic-node1 /]# [root@atomic-node1 /]# mount |grep mnt /dev/mapper/atomic-node1-root on /mnt/brick1 type xfs (rw,relatime,seclabel,attr2,inode64,noquota) This container is built on top of CentOS base image as shown below. [root@atomic-node1 /]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@atomic-node1 /]# In this article we discussed, how to run GlusterFS as a pod in Kubernetes or Openshift setup. [Part 2] covers `how to use GlusterFS as a service, Persistent Volume for a Persistent Volume Claim`. [Part 3] covers `how to use GlusterFS template to deploy GlusterFS pods in an Openshift/kubernetes setup`.
by on August 24, 2016

[Coming Soon] Dynamic Provisioning of GlusterFS volumes in Kubernetes/Openshift!!

In this context I am talking about the dynamic provisioning capability of ‘glusterfs’ plugin in Kubernetes/Openshift. I have submitted a Pull Request to Kubernetes to add this functionality for GlusterFS. At present, there is no existing network storage provisioners in kubernetes eventhough there are cloud providers. The idea here is to make the glusterfs plugin capable of provisioning volumes on demand from kubernetes/openshift .. Cool, Isnt it ? Indeed this is a nice feature to have. That said, an OSE user request for a space for example : 20G and the glusterfs plugin takes this request and create 20G and bound that to the claim. The plugin can use any REST service, but the example patch is based on ‘heketi’. Here is the workflow: Start your kubernetes controller manager with highlighted options:

 ...kube controller-manager --v=3 
 --service-account-private-key-file=/tmp/kube-serviceaccount.key
 --root-ca-file=/var/run/kubernetes/apiserver.crt --enable-hostpath-provisioner=false

 --enable-network-storage-provisioner=true --storage-config=/tmp --net-provider=glusterfs
 --pvclaimbinder-sync-period=15s --cloud-provider= --master=127.0.0.1:8080

 
Create a file called `gluster.json` in `/tmp` directory. The important fields in this config file are ‘endpoint’ and ‘resturl’. The endpoint has to be defined and match the setup. The `resturl` has been filled with the rest service which can take the input and create a gluster volume in the backend. As mentioned earlier I am using `heketi` for the same.

 [hchiramm@dhcp35-111 tmp]$ cat gluster.json
 {
 "endpoint": "glusterfs-cluster",
 "resturl": "http://127.0.0.1:8081",
 "restauthenabled":false,
 "restuser":"",
 "restuserkey":""
 }
 [hchiramm@dhcp35-111 tmp]$
 

We have to define an ENDPOINT and SERVICE. Below are the example configuration files. ENDPOINT : “ip” has to be filled with your gluster trusted pool IP.


[hchiramm@dhcp35-111 ]$ cat glusterfs-endpoint.json
{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "glusterfs-cluster"
},
"subsets": [
{
"addresses": [
{
"ip": "10.36.4.112"
}
],
"ports": [
{
"port": 1
}
]
},
{
"addresses": [
{
"ip": "10.36.4.112"
}
],
"ports": [
{
"port": 1
}
]
}
]
}

SERVICE: Please note that the Service Name is matching with ENDPOINT name


[hchiramm@dhcp35-111 ]$ cat gluster-service.json
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "glusterfs-cluster"
},
"spec": {
"ports": [
{"port": 1}
]
}
}
[hchiramm@dhcp35-111 ]$

Finally we have a Persistent Volume Claim file as shown below: NOTE: The size of the volume is mentioned as ’20G’:


[hchiramm@dhcp35-111 ]$ cat gluster-pvc.json
{
"kind": "PersistentVolumeClaim",
"apiVersion": "v1",
"metadata": {
"name": "glusterc",
"annotations": {
"volume.alpha.kubernetes.io/storage-class": "glusterfs"
}
},
"spec": {
"accessModes": [
"ReadOnlyMany"
],
"resources": {
"requests": {
"storage": "20Gi"
}
}
}
}
[hchiramm@dhcp35-111 ]$

Let's start defining the endpoint, service and PVC.


[hchiramm@dhcp35-111 ]$ ./kubectl create -f glusterfs-endpoint.json
endpoints "glusterfs-cluster" created
[hchiramm@dhcp35-111 ]$ ./kubectl create -f gluster-service.json
service "glusterfs-cluster" created
[hchiramm@dhcp35-111 ]$ ./kubectl get ep,service
NAME ENDPOINTS AGE
ep/glusterfs-cluster 10.36.6.105:1 14s
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/glusterfs-cluster 10.0.0.10 1/TCP 9s
svc/kubernetes 10.0.0.1 443/TCP 13m
[hchiramm@dhcp35-111 ]$ ./kubectl get pv,pvc
[hchiramm@dhcp35-111 ]$

Now, let's request a claim!

[hchiramm@dhcp35-111 ]$ ./kubectl create -f glusterfs-pvc.json
persistentvolumeclaim "glusterc" created
[hchiramm@dhcp35-111 ]$ ./kubectl get pv,pvc
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
pv/pvc-39ebcdc5-442b-11e6-8dfa-54ee7551fd0c  20Gi ROX  Bound  default/glusterc 2s
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
pvc/glusterc Bound pvc-39ebcdc5-442b-11e6-8dfa-54ee7551fd0c 0 3s
[hchiramm@dhcp35-111 ]$

Awesome! Based on the request it created a PV and BOUND to the PVClaim!!


[hchiramm@dhcp35-111 ]$ ./kubectl describe pv pvc-39ebcdc5-442b-11e6-8dfa-54ee7551fd0c
Name: pvc-39ebcdc5-442b-11e6-8dfa-54ee7551fd0c
Labels:
Status: Bound
Claim: default/glusterc
Reclaim Policy: Delete
Access Modes: ROX
Capacity: 20Gi
Message:
Source:
Type: Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
EndpointsName: glusterfs-cluster
 Path: vol_038b56756f4e3ab4b07a87494097941c
ReadOnly: false
No events.
[hchiramm@dhcp35-111 ]$
 

Verify the volume exist in backend:

 [root@ ~]# heketi-cli volume list |grep 038b56756f4e3ab4b07a87494097941c
 038b56756f4e3ab4b07a87494097941c
 [root@ ~]#

Let's delete the PV claim --


[hchiramm@dhcp35-111 ]$ ./kubectl delete pvc glusterc
persistentvolumeclaim "glusterc" deleted
[hchiramm@dhcp35-111 ]$ ./kubectl get pv,pvc
[hchiramm@dhcp35-111 ]$

It got deleted! Verify it from backend:


 [root@ ~]# heketi-cli volume list |grep 038b56756f4e3ab4b07a87494097941c
 [root@ ~]# 

We can use the Volume for app pods by referring the claim name. Hope this is a nice feature to have !

Please let me know if you have any comments/suggestions.

Also, the patch - https://github.com/kubernetes/kubernetes/pull/30888 is undergoing review in upstream as mentioned earlier and hopefully it will make it soon to the kubernetes release. I will provide an update here as soon as its available in upstream.