all posts tagged Software Defined Storage


by on November 26, 2015

Usenix LISA 2015 Tutorial on GlusterFS

I have been working in GlusterFS for quite some time now.  As you might know that GlusterFS is an open source distributed file-system. What differentiate Gluster from other distributed file-system is its scale-out nature, data access without metadata server and it’s modular design.

I feel that in general there is lack of awareness of Gluster as a distributed file-system. Therefore I wanted to do some kind of tutorial or workshop on Gluster. LISA gave me that perfect opportunity. LISA is one of the largest gathering of system engineers and operations professionals.

Me and my colleague Poornima gave a half-day tutorial on Gluster. Our tutorial was the First tutorial of the conference, i.e. on 8th Nov 2015.

This time LISA was held in Wasington DC from 8th Nov to 13th Nov. The conference was divided into multiple programs – Training, Workshops,
Technical sessions (or Conference Program), LISA Build, Lab, Expo, etc. Training, Workshops and LISA build started on 8th Nov, whereas Conference Program and Expo started from 11th Nov.

The tutorial was divided into two parts first part gives an overview of Gluster and its architecture and the second part was more hands-on. I felt that hands-on session would be much more helpful if the audience understood what is gluster and how various component gel together.

So in the first part I covered the basics of Gluster. e.g. trusted storage pool, nodes, bricks, volumes, etc. It is crucial that the audience understand these terms before we jump into anything. After that we introduced the translator concepts to the audience. This is one of the major feature which makes gluster so modular. Each translators provide unique functionality. Translators in Gluster are stackable therefore you can stack one or more translators to get your required feature set.

Then we explained various volume types in detail. Each volume type provide a unique functionality. It is important to understand the volume type because based on the type of workload or the use-case one has to choose a specific volume type. Apart from the regular volume types we also explained recently introduced volume types, e.g. sharding, disperse, etc.

Apart from the general GlusterFS architecture we also introduced various features like Geo-replication, Snapshots, Data-Tiering, etc.

The second part of the tutorial started after the tea break. Before the start of the tutorial we distributed USB pen drives to all the attendees. These pen drive contains a VirtualBox and KVM image of CentOS with GlusterFS pre-installed. Most of them managed to copy VirtualBox image without any problem, but as usual one or two people did face some minor hiccups.

Poornima started the hands-on session with Gluster installation and setup. We wanted to showcase how easily we can install and configure Gluster. she also gave a live demo so that attendees can try Gluster by themselves. We took some examples to show case how to setup volumes, how to expand and shrink the storage associated with the volume. In this context we also discussed the role of rebalance and Self-Heal Daemon (SHD). Near the end we also covered general Gluster troubleshooting.

Overall the session was very interactive and attendees asked lots of questions during the course of the training. Most of the questions were based on the tutorial slides, but some questions were based on previous experience with Gluster as well.

During the break time and after the session we had informal discussions with few attendees, which was very informative and helpful. This also made me realize that we should do more such sessions to spread the Gluster word.

The tutorial materials and the presentations can be found in the following link:

https://github.com/gluster/gluster-tutorial

We also have pre-recorded multiple demos for various Gluster use-cases. These recordings are also available in the above mentioned link. These links would be useful to learn more about Gluster.

by on

Usenix LISA 2015 Tutorial on GlusterFS

I have been working in GlusterFS for quite some time now.  As you might know that GlusterFS is an open source distributed file-system. What differentiate Gluster from other distributed file-system is its scale-out nature, data access without metadata server and it’s modular design.

I feel that in general there is lack of awareness of Gluster as a distributed file-system. Therefore I wanted to do some kind of tutorial or workshop on Gluster. LISA gave me that perfect opportunity. LISA is one of the largest gathering of system engineers and operations professionals.

Me and my colleague Poornima gave a half-day tutorial on Gluster. Our tutorial was the First tutorial of the conference, i.e. on 8th Nov 2015.

This time LISA was held in Wasington DC from 8th Nov to 13th Nov. The conference was divided into multiple programs – Training, Workshops,
Technical sessions (or Conference Program), LISA Build, Lab, Expo, etc. Training, Workshops and LISA build started on 8th Nov, whereas Conference Program and Expo started from 11th Nov.

The tutorial was divided into two parts first part gives an overview of Gluster and its architecture and the second part was more hands-on. I felt that hands-on session would be much more helpful if the audience understood what is gluster and how various component gel together.

So in the first part I covered the basics of Gluster. e.g. trusted storage pool, nodes, bricks, volumes, etc. It is crucial that the audience understand these terms before we jump into anything. After that we introduced the translator concepts to the audience. This is one of the major feature which makes gluster so modular. Each translators provide unique functionality. Translators in Gluster are stackable therefore you can stack one or more translators to get your required feature set.

Then we explained various volume types in detail. Each volume type provide a unique functionality. It is important to understand the volume type because based on the type of workload or the use-case one has to choose a specific volume type. Apart from the regular volume types we also explained recently introduced volume types, e.g. sharding, disperse, etc.

Apart from the general GlusterFS architecture we also introduced various features like Geo-replication, Snapshots, Data-Tiering, etc.

The second part of the tutorial started after the tea break. Before the start of the tutorial we distributed USB pen drives to all the attendees. These pen drive contains a VirtualBox and KVM image of CentOS with GlusterFS pre-installed. Most of them managed to copy VirtualBox image without any problem, but as usual one or two people did face some minor hiccups.

Poornima started the hands-on session with Gluster installation and setup. We wanted to showcase how easily we can install and configure Gluster. she also gave a live demo so that attendees can try Gluster by themselves. We took some examples to show case how to setup volumes, how to expand and shrink the storage associated with the volume. In this context we also discussed the role of rebalance and Self-Heal Daemon (SHD). Near the end we also covered general Gluster troubleshooting.

Overall the session was very interactive and attendees asked lots of questions during the course of the training. Most of the questions were based on the tutorial slides, but some questions were based on previous experience with Gluster as well.

During the break time and after the session we had informal discussions with few attendees, which was very informative and helpful. This also made me realize that we should do more such sessions to spread the Gluster word.

The tutorial materials and the presentations can be found in the following link:

https://github.com/gluster/gluster-tutorial

We also have pre-recorded multiple demos for various Gluster use-cases. These recordings are also available in the above mentioned link. These links would be useful to learn more about Gluster.

by on October 22, 2014

Gluster Volume Snapshot Howto

This article will provide details on how to configure GlusterFS volume to make use of Gluster Snapshot feature. As we have already discussed earlier that Gluster Volume Snapshot is based of thinly provisioned logical volume (LV). Therefore here I will guide you regarding creating LVs, bricks and Gluster volume. And then we will discuss various features of GlusterFS volume snapshot and how to use them.

Gluster Volume Snapshot

Gluster volume snapshot feature is based of thinly provisioned LVs. Therefore to make use of this feature following guidelines have to be followed:

  • All bricks should be carved out from an independent thinly provisioned logical volume (LV). In other words, no two brick should share a common LV. More details about thin provisioning and thin provisioned snapshot can be found here.
  • This thinly provisioned LV should only be used for forming a brick.
  • Thin pool from which the thin LVs are created should have sufficient space and also it should have sufficient space for pool metadata.

Given the above prerequisites I will take you through an example on how to create bricks in such case and how to create a volume. Click here to get the details of LVM and it’s various options.

The above diagram gives an overview of LVM. The volume group is formed out of storage devices and one or more thin pools can be carved out of this volume group. Once you have a thin pool, you can create one or more thinly provisioned logical volume (LV). All the thin LVs are created with a virtual size. This virtual size can even be greater than the total pool size itself. This gives the admin flexibility to procure their hardware when there is demand. e.g. even though i have 1 TB of storage I can create LVs of size 2 TB. Thin pool tracks the total strorage usage by each LV and the overall pool size. So you can add more storage as your utilization grows.

Let’s start with storage devices and see how we can create volume groups and thin LVs. If you have a storage device attached to your server then you can use them or else if you just want to try how GlusterFS work then you can even make use of loopback devices.

Note: If you already have a storage device then skip this section and goto pvcreate section. The loopback device should only be used for testing purpose.

The loopback device should point to a file and the device size is dependent on the file size. So lets create a file of required length. You can create files by any means but I prefer to use fallocate as it creates big files really fast.

fallocate -l 2G dev1.img

The above command will create a file of size 2GB. If you already have a loopback device then associate the newly created file with the device or else create a loopback device and then associate it. Following command can be used to create a loopback device (loop0).

mknod /dev/loop0 b 7 0

Once we have a loopback device we need to associate it with a file.

losetup /dev/loop0 dev1.img

Now, your test device is ready. Lets see how we can create a thinly provisioned LV and for that the first step is to create a physical volume (PV). Use the following command to create a PV.

pvcreate /dev/loop0

The above command will initialize the storage disk or partition so that it can be used by the LVM. Once we have the PVs ready we are ready to create the  volume group (VG). You can create one VG per PV or you can combine mulitple PVs to form a single VG. In this example we are using a single PV to create a VG.

vgcreate mygroup /dev/loop0

Now, we have a VG named “mygroup”. The next step is to create a thin pool. You can allocate one or more thin pool inside a volume group. In this example we are creating a single thin pool inside the volume group.

lvcreate -L 2G -T mygroup/mythinpool

The above command will create a thin pool named “mythinpool” with 2 GB as storage. Once you have a thin pool you are ready to create thin volumes. Use the following command to create thin volume.

lvcreate -V 1G -T mygroup/mythinpool -n thinv1

The above command will create a thin volume named “thinv1” of size 1GB. Now, before creating a brick out of this volume you need to create a file-system on these thin volume. Ideally you should use XFS or Ext4 as the file-system. Use mkfs to create a valid file-system. e.g. you can create a simple XFS file-system using the following command.

mkfs.xfs /dev/mygroup/thinv1

Now, we need to mount these LVs to form the bricks. Use the mount command to do so.

mount /dev/mygroup/thinv1 /bricks/brick1

Using the above method you can create one or more bricks. For our testing I created one more brick in “anotherhost” machine. Lets create a Gluster volume from these LVs. Use the following command to do so.

gluster volume create vol1 replica 2 myhost:/bricks/brick1/b1 anotherhost:/bricks/brick1/b1

We have a volume with bricks created out of thin LVs. We are now ready to  test and use the snapshot feature. Ah, forgot one thing. You need to first start the volume.

gluster volume start vol1

Gluster Volume Snapshot

With that we have a running Gluster volume. Lets create a snapshot of this volume. Snapshot can be taken only for a started Gluster volume. This snapshot creates a read-only Gluster volume. The following diagram represents the snapshot volume. As you can see the snapshot volume is the exact copy of the Gluster volume.

 

Snapshot Commands

This section provides details about various commands provided by Gluster to manage snapshots.

Snapshot creation

gluster snapshot create <snapname> <volname(s)> [description <description>] [force]

This command will create a snapshot of a Gluster volume. Snap-name is a mandatory field and the name should be unique in the entire cluster. vol-name is the volume name of the origin volume whose snapshot is to be taken. Users can also provide an optional description to be saved along with the snap (max 1024 characters).

Following pre-requisites need to be met before snapshot can be taken:

  • Gluster volume should be in started state.
  • All the bricks associated with the volume should be in started state, unless it is a n-way replication where n <= 3. In such case quorum is checked.
  • Snapshot name should be unique in the cluster.
  • No other volume operation, like rebalance, add-brick etc, should be running on the volume.
  • Total number of snapshots in the volume should not be equal to Effective snap-max-hard-limit.

e.g.

gluster snapshot create snap1 vol1

The above command will create a Gluster snapshot named “snap1” for volume “vol1”. This snapshot is a read-only Gluster volume. The bricks of this read-only volume is mounted under /var/run/gluster/snaps/ folder as /var/run/gluster/snaps/<snap-volume-name>/brick<bricknumber>, e.g.

/var/run/gluster/snaps/ee1c2c74f70a4043a2bbba94362eaeb6/brick1
/var/run/gluster/snaps/ee1c2c74f70a4043a2bbba94362eaeb6/brick2

Listing of available snaps

gluster snapshot list [volname]

This command is used to list all the snapshots present in the trusted storage pool, or for a specified volume.

Info of snapshots

gluster snapshot info [(snapname | volume <volname>)]

Shows the information of all the snapshots or the specified snapshot. The status will include the brick details, UUID and snapshot volume status.

Status of snapshots

gluster snapshot status [(snapname | volume <volname>)]

Shows the running status of all the snapshots or the specified snapshot. The status will include the brick details, LVM details, process details, etc.

Deleting snaps

gluster snapshot delete <snapname>

This command will delete the specified snapshot.

Activating a snap volume

Use the following commands to activate snapshot.

gluster snapshot activate <snapname> [force]

If some of the bricks of the snapshot volume are down then use the force command to start them.

Deactivating a snap volume

gluster snapshot deactivate <snapname>

By default the created snapshot is in active state. The above command will deactivate an active snapshot

Configuring snapshot

The configurable parameters for snapshot are:

  • snap-max-hard-limit: This is a hard limit beyond which snapshot creation is not allowed. This limit can be set for the trusted storage pool or per volume. The effective max limit is lowest of volume and the trusted storage limit.
  • snap-max-soft-limit: This is the soft limit beyond which user will get a warning on snapshot creation. If auto-delete feature is enabled then snapshot creation beyond this point will lead to deletion of the oldest snapshot. This is a percentage value and the default value is 90%.
  • auto-delete: This will enable or disable auto-delete feature. When enabled it will delete the oldest snapshot when the snapshot count in a volume crosses snap-max-soft-limit. By default this feature is disabled.
The following command displays the existing config values for a volume. If the volume name is not provided then config values of all the volumes are displayed.

gluster snapshot config [vol-name]

To change the existing configuration values, run the following command. If vol-name is provided then config value of that volume is changed, else it will set/change the trusted storage pool limit.

gluster snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])

Volume specific limit cannot cross the trusted storage pool limit. If a volume specific limit is not provided then the trusted storage pool limit will be considered.
snap-max-hard-limit: Maximum hard limit for the system or the specified volume.
snap-max-soft-limit: Soft limit mark for the system.
auto-delete: This will enable or disable auto-delete feature. By default auto-delete is disabled.

 

Restoring snaps

gluster snapshot restore <snapname>

This command restores an already taken snapshot of the volume. Snapshot restore is an offline activity therefore if any volume which is part of the given snap is online then the restore operation will fail.

Once the snapshot is restored it will be deleted from the list of snapshots. Therefore if you want to retain the snapshot then you should take explicit snapshot after the restore operation.

 

Accessing Snapshots

As I mentioned before Gluster snapshot creates a read-only volume. This volume can be accessed via fuse mount. Currenly other protocols such as NFS and CIFS access are not supported. Use the following command to mount the snapshot volume

mount -t glusterfs :/snaps/<snap-name>/<parent-volname> /mount_point

e.g.

mount -t glusterfs myhost:/snaps/snap1/vol1 /mnt/snapvol

Another way of accessing snapshots is via User Serviceable Snapshots, which I will explain later.

by on

Gluster Volume Snapshot Howto

This article will provide details on how to configure GlusterFS volume to make use of Gluster Snapshot feature. As we have already discussed earlier that Gluster Volume Snapshot is based of thinly provisioned logical volume (LV). Therefore here I will guide you regarding creating LVs, bricks and Gluster volume. And then we will discuss various features of GlusterFS volume snapshot and how to use them.

Gluster Volume Snapshot

Gluster volume snapshot feature is based of thinly provisioned LVs. Therefore to make use of this feature following guidelines have to be followed:

  • All bricks should be carved out from an independent thinly provisioned logical volume (LV). In other words, no two brick should share a common LV. More details about thin provisioning and thin provisioned snapshot can be found here.
  • This thinly provisioned LV should only be used for forming a brick.
  • Thin pool from which the thin LVs are created should have sufficient space and also it should have sufficient space for pool metadata.

Given the above prerequisites I will take you through an example on how to create bricks in such case and how to create a volume. Click here to get the details of LVM and it’s various options.

The above diagram gives an overview of LVM. The volume group is formed out of storage devices and one or more thin pools can be carved out of this volume group. Once you have a thin pool, you can create one or more thinly provisioned logical volume (LV). All the thin LVs are created with a virtual size. This virtual size can even be greater than the total pool size itself. This gives the admin flexibility to procure their hardware when there is demand. e.g. even though i have 1 TB of storage I can create LVs of size 2 TB. Thin pool tracks the total strorage usage by each LV and the overall pool size. So you can add more storage as your utilization grows.

Let’s start with storage devices and see how we can create volume groups and thin LVs. If you have a storage device attached to your server then you can use them or else if you just want to try how GlusterFS work then you can even make use of loopback devices.

Note: If you already have a storage device then skip this section and goto pvcreate section. The loopback device should only be used for testing purpose.

The loopback device should point to a file and the device size is dependent on the file size. So lets create a file of required length. You can create files by any means but I prefer to use fallocate as it creates big files really fast.

fallocate -l 2G dev1.img

The above command will create a file of size 2GB. If you already have a loopback device then associate the newly created file with the device or else create a loopback device and then associate it. Following command can be used to create a loopback device (loop0).

mknod /dev/loop0 b 7 0

Once we have a loopback device we need to associate it with a file.

losetup /dev/loop0 dev1.img

Now, your test device is ready. Lets see how we can create a thinly provisioned LV and for that the first step is to create a physical volume (PV). Use the following command to create a PV.

pvcreate /dev/loop0

The above command will initialize the storage disk or partition so that it can be used by the LVM. Once we have the PVs ready we are ready to create the  volume group (VG). You can create one VG per PV or you can combine mulitple PVs to form a single VG. In this example we are using a single PV to create a VG.

vgcreate mygroup /dev/loop0

Now, we have a VG named “mygroup”. The next step is to create a thin pool. You can allocate one or more thin pool inside a volume group. In this example we are creating a single thin pool inside the volume group.

lvcreate -L 2G -T mygroup/mythinpool

The above command will create a thin pool named “mythinpool” with 2 GB as storage. Once you have a thin pool you are ready to create thin volumes. Use the following command to create thin volume.

lvcreate -V 1G -T mygroup/mythinpool -n thinv1

The above command will create a thin volume named “thinv1” of size 1GB. Now, before creating a brick out of this volume you need to create a file-system on these thin volume. Ideally you should use XFS or Ext4 as the file-system. Use mkfs to create a valid file-system. e.g. you can create a simple XFS file-system using the following command.

mkfs.xfs /dev/mygroup/thinv1

Now, we need to mount these LVs to form the bricks. Use the mount command to do so.

mount /dev/mygroup/thinv1 /bricks/brick1

Using the above method you can create one or more bricks. For our testing I created one more brick in “anotherhost” machine. Lets create a Gluster volume from these LVs. Use the following command to do so.

gluster volume create vol1 replica 2 myhost:/bricks/brick1/b1 anotherhost:/bricks/brick1/b1

We have a volume with bricks created out of thin LVs. We are now ready to  test and use the snapshot feature. Ah, forgot one thing. You need to first start the volume.

gluster volume start vol1

Gluster Volume Snapshot

With that we have a running Gluster volume. Lets create a snapshot of this volume. Snapshot can be taken only for a started Gluster volume. This snapshot creates a read-only Gluster volume. The following diagram represents the snapshot volume. As you can see the snapshot volume is the exact copy of the Gluster volume.

 

Snapshot Commands

This section provides details about various commands provided by Gluster to manage snapshots.

Snapshot creation

gluster snapshot create <snapname> <volname(s)> [description <description>] [force]

This command will create a snapshot of a Gluster volume. Snap-name is a mandatory field and the name should be unique in the entire cluster. vol-name is the volume name of the origin volume whose snapshot is to be taken. Users can also provide an optional description to be saved along with the snap (max 1024 characters).

Following pre-requisites need to be met before snapshot can be taken:

  • Gluster volume should be in started state.
  • All the bricks associated with the volume should be in started state, unless it is a n-way replication where n <= 3. In such case quorum is checked.
  • Snapshot name should be unique in the cluster.
  • No other volume operation, like rebalance, add-brick etc, should be running on the volume.
  • Total number of snapshots in the volume should not be equal to Effective snap-max-hard-limit.

e.g.

gluster snapshot create snap1 vol1

The above command will create a Gluster snapshot named “snap1” for volume “vol1”. This snapshot is a read-only Gluster volume. The bricks of this read-only volume is mounted under /var/run/gluster/snaps/ folder as /var/run/gluster/snaps/<snap-volume-name>/brick<bricknumber>, e.g.

/var/run/gluster/snaps/ee1c2c74f70a4043a2bbba94362eaeb6/brick1
/var/run/gluster/snaps/ee1c2c74f70a4043a2bbba94362eaeb6/brick2

Listing of available snaps

gluster snapshot list [volname]

This command is used to list all the snapshots present in the trusted storage pool, or for a specified volume.

Info of snapshots

gluster snapshot info [(snapname | volume <volname>)]

Shows the information of all the snapshots or the specified snapshot. The status will include the brick details, UUID and snapshot volume status.

Status of snapshots

gluster snapshot status [(snapname | volume <volname>)]

Shows the running status of all the snapshots or the specified snapshot. The status will include the brick details, LVM details, process details, etc.

Deleting snaps

gluster snapshot delete <snapname>

This command will delete the specified snapshot.

Activating a snap volume

Use the following commands to activate snapshot.

gluster snapshot activate <snapname> [force]

If some of the bricks of the snapshot volume are down then use the force command to start them.

Deactivating a snap volume

gluster snapshot deactivate <snapname>

By default the created snapshot is in active state. The above command will deactivate an active snapshot

Configuring snapshot

The configurable parameters for snapshot are:

  • snap-max-hard-limit: This is a hard limit beyond which snapshot creation is not allowed. This limit can be set for the trusted storage pool or per volume. The effective max limit is lowest of volume and the trusted storage limit.
  • snap-max-soft-limit: This is the soft limit beyond which user will get a warning on snapshot creation. If auto-delete feature is enabled then snapshot creation beyond this point will lead to deletion of the oldest snapshot. This is a percentage value and the default value is 90%.
  • auto-delete: This will enable or disable auto-delete feature. When enabled it will delete the oldest snapshot when the snapshot count in a volume crosses snap-max-soft-limit. By default this feature is disabled.
The following command displays the existing config values for a volume. If the volume name is not provided then config values of all the volumes are displayed.

gluster snapshot config [vol-name]

To change the existing configuration values, run the following command. If vol-name is provided then config value of that volume is changed, else it will set/change the trusted storage pool limit.

gluster snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])

Volume specific limit cannot cross the trusted storage pool limit. If a volume specific limit is not provided then the trusted storage pool limit will be considered.
snap-max-hard-limit: Maximum hard limit for the system or the specified volume.
snap-max-soft-limit: Soft limit mark for the system.
auto-delete: This will enable or disable auto-delete feature. By default auto-delete is disabled.

 

Restoring snaps

gluster snapshot restore <snapname>

This command restores an already taken snapshot of the volume. Snapshot restore is an offline activity therefore if any volume which is part of the given snap is online then the restore operation will fail.

Once the snapshot is restored it will be deleted from the list of snapshots. Therefore if you want to retain the snapshot then you should take explicit snapshot after the restore operation.

 

Accessing Snapshots

As I mentioned before Gluster snapshot creates a read-only volume. This volume can be accessed via fuse mount. Currenly other protocols such as NFS and CIFS access are not supported. Use the following command to mount the snapshot volume

mount -t glusterfs :/snaps/<snap-name>/<parent-volname> /mount_point

e.g.

mount -t glusterfs myhost:/snaps/snap1/vol1 /mnt/snapvol

Another way of accessing snapshots is via User Serviceable Snapshots, which I will explain later.

by on October 7, 2014

Gluster Volume

GlusterFS

GlusterFS is an open source distributed file system. It incorporates automatic fail-over as a primary feature. All of this is accomplished without a centralized metadata server. Which also guarantees no single point of failure.

The detail documentation and getting started document can be found at Gluster.org. In this article I want to give an overview of Gluster so that you can understand GlusterFS volume snapshot better.

Let’s say you have some machines (or virtual machines) where you want to host GlusterFS. So the first thing you want to install is a POSIX compliant operating system, e.g. Fedora, CentOS, RHEL, etc. Install GlusterFS server on all these machines. Click here to get the detailed instruction on how to install GlusterFS. Once GlusterFS server is installed on each machine you have to start the server. Run the following command to start the GlusterFS server:

service glusterd start

Or, start the server using the following command:

glusterd

Now, you have multiple GlusterFS servers, but they are not part of the Gluster “Trusted Storage Pool” yet. All the servers should be part of the “Trusted Storage Pool” before they can be accessed.  Lets say you have 3 servers, Host1, Host2, and Host3. Run the following command to add them to the Gluster Trusted Storage Pool.

[root@Host1]# gluster peer probe Host2
peer probe: success

Now, Host1 and Host2 are in the Trusted Storage Pool. You can check the status of the peer probe using the peer status command.

[root@Host1]# gluster peer status
Number of Peers: 1
Hostname: Host2
Uuid: 3b51894a-6cc1-43d0-a996-126a347056c8
State: Peer in Cluster (Connected)

If you have any problems during the peer probe, make sure that your firewall is not blocking Gluster ports. Preferably, your storage environment should be located on a safe segment of your network where firewall is not necessary. In the real world, that simply isn’t possible for all environments. If you are willing to accept the potential performance loss of running a firewall, you need to know the following. Gluster makes use of ports 24007 for the Gluster Daemon, 24008 for Infiniband management (optional unless you are using IB), and one port for each brick in a volume. So, for example, if you have 4 bricks in a volume, port 49152 – 49155 would be used . Gluster uses ports 34865 – 34867 for the inline Gluster NFS server. Additionally, port 111 is used for portmapper, and should have both TCP and UDP open.

Once Host1 and Host2 are part of the Trusted Storage Pool you have to add Host3 to the trusted storage pool. You should run the same gluster peer probe command from either Host1 or Host2 to add Host3 to the Trusted Storage Pool. You will see the following output when you check the peer status:

[root@Host1]# gluster peer status
Number of Peers: 2
Hostname: Host2
Uuid: 3b51894a-6cc1-43d0-a996-126a347056c8
State: Peer in Cluster (Connected)
Hostname: Host3
Uuid: fa751bde-1f34-4d80-a59e-fec4113ba8ea
State: Peer in Cluster (Connected)

Now, you have a Trusted Storage Pool with multiple servers or nodes, but still we are not ready for serving files from the trusted storage pool. GlusterFS volume is the unified namespace through which an user can access his/her files on the distributed storage. A Trusted Storage Pool can host multiple volumes. And each volume is made up of one or more bricks. The brick provides a mapping between the local file-system and the Gluster volume.

The above diagram shows an example of Gluster volume. Here we have three nodes (Host1, Host2 and Host3) and a Gluster Volume is created from the bricks present in those nodes.

Until now we have learned how to create a Trusted Pool, and now to create a volume you need to create bricks. These bricks can be a simple directory in your storage node, but to make use of snapshot feature these bricks have to adhere to some guidelines. In this document I provide you those guidelines and will take you through an example setup.

See guidelines for creating snapshot supportable volumes.

by on

Gluster Volume

GlusterFS

GlusterFS is an open source distributed file system. It incorporates automatic fail-over as a primary feature. All of this is accomplished without a centralized metadata server. Which also guarantees no single point of failure.

The detail documentation and getting started document can be found at Gluster.org. In this article I want to give an overview of Gluster so that you can understand GlusterFS volume snapshot better.

Let’s say you have some machines (or virtual machines) where you want to host GlusterFS. So the first thing you want to install is a POSIX compliant operating system, e.g. Fedora, CentOS, RHEL, etc. Install GlusterFS server on all these machines. Click here to get the detailed instruction on how to install GlusterFS. Once GlusterFS server is installed on each machine you have to start the server. Run the following command to start the GlusterFS server:

service glusterd start

Or, start the server using the following command:

glusterd

Now, you have multiple GlusterFS servers, but they are not part of the Gluster “Trusted Storage Pool” yet. All the servers should be part of the “Trusted Storage Pool” before they can be accessed.  Lets say you have 3 servers, Host1, Host2, and Host3. Run the following command to add them to the Gluster Trusted Storage Pool.

[root@Host1]# gluster peer probe Host2
peer probe: success

Now, Host1 and Host2 are in the Trusted Storage Pool. You can check the status of the peer probe using the peer status command.

[root@Host1]# gluster peer status
Number of Peers: 1
Hostname: Host2
Uuid: 3b51894a-6cc1-43d0-a996-126a347056c8
State: Peer in Cluster (Connected)

If you have any problems during the peer probe, make sure that your firewall is not blocking Gluster ports. Preferably, your storage environment should be located on a safe segment of your network where firewall is not necessary. In the real world, that simply isn’t possible for all environments. If you are willing to accept the potential performance loss of running a firewall, you need to know the following. Gluster makes use of ports 24007 for the Gluster Daemon, 24008 for Infiniband management (optional unless you are using IB), and one port for each brick in a volume. So, for example, if you have 4 bricks in a volume, port 49152 – 49155 would be used . Gluster uses ports 34865 – 34867 for the inline Gluster NFS server. Additionally, port 111 is used for portmapper, and should have both TCP and UDP open.

Once Host1 and Host2 are part of the Trusted Storage Pool you have to add Host3 to the trusted storage pool. You should run the same gluster peer probe command from either Host1 or Host2 to add Host3 to the Trusted Storage Pool. You will see the following output when you check the peer status:

[root@Host1]# gluster peer status
Number of Peers: 2
Hostname: Host2
Uuid: 3b51894a-6cc1-43d0-a996-126a347056c8
State: Peer in Cluster (Connected)
Hostname: Host3
Uuid: fa751bde-1f34-4d80-a59e-fec4113ba8ea
State: Peer in Cluster (Connected)

Now, you have a Trusted Storage Pool with multiple servers or nodes, but still we are not ready for serving files from the trusted storage pool. GlusterFS volume is the unified namespace through which an user can access his/her files on the distributed storage. A Trusted Storage Pool can host multiple volumes. And each volume is made up of one or more bricks. The brick provides a mapping between the local file-system and the Gluster volume.

The above diagram shows an example of Gluster volume. Here we have three nodes (Host1, Host2 and Host3) and a Gluster Volume is created from the bricks present in those nodes.

Until now we have learned how to create a Trusted Pool, and now to create a volume you need to create bricks. These bricks can be a simple directory in your storage node, but to make use of snapshot feature these bricks have to adhere to some guidelines. In this document I provide you those guidelines and will take you through an example setup.

See guidelines for creating snapshot supportable volumes.