The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

A Gluster Block Interface – Performance and Configuration


This post shares some experiences I’ve had in simulating a block device in gluster.

The block device is a file based image, which acts as the backend for the Linux SCSI target. The file resides in gluster, so enjoys gluster’s feature set. But the client only sees a block device. The Linux SCSI target presents it over iSCSI.

Some more information on how to set this up is at the bottom of this post.

But where to mount gluster?

There are three options to consider. Call them client, server, and gateway.

The configurations are depicted in the diagram below:

  • Configuring the block device at the server means the client only requires an iSCSI initiator.

  • Configuring the block device at the client allows I/O to fan out to different nodes.

  • Configuring the block device at a gateway allows I/O to fan out to different nodes without changing the client.


These options have their pros and cons. For example, a gateway minimizes client overhead while providing fan-out. OTOH, the customer must provide an additional node.

In my case, the objective was to minimize customer burden. Server-side configuration is probably the best choice. After chatting with some colleagues here at Red Hat, thats what we settled on.

I ran some simple performance tests using the “fio” tool to generate I/O.

  • Up to 10 fio processes were started.

  • The queue depth for each was 32.

  • Each process sends I/O to its own slice of the volume.

  • client and server block cache is flushed between tests

    • echo 3 > /proc/sys/vm/drop_caches

  • 64k records

  • 50G volume replicated over two nodes

  • Gluster version:




  • Only two nodes were used in performance testing. Ideally more nodes would be used, but that equipment is not readily available.

  • There are numerous options to tune, including the queue depth, number of streams, number of paths, gluster volume configuration, block size, number of targets etc. Parameters were chosen based on trial and error rather than formal methodology.




1 KVM client – 6VCPU x 20GB client (pinned)  <===> 2 file servers baremetal 12x20GB

4 x 64GB  blk virtio file systems (cache=none)

(1 x 12disk RAID6 gluster brick / server)


For the interested, here is a cookbook to set it up. For more information, see [1] and [2].


  1. Mount gluster locally on your server.


$ mount -t glusterfs /mnt


  1. Create a large file representing your block device within the gluster fs.


$ dd if=/dev/zero of=/mnt/disk bs=2G count=25


  1. Create a target using the file as the backend storage.


If necessary, download the Linux SCSI target. Then start the service.


$ yum install scsi-target-utils

$ service tgtd start


You must give an iSCSI Qualified name (IQN), in the format :


  yyyy-mm represents the 4-digit year and 2-digit month the device was started (for example: 2011-07);


$ tgtadm –lld iscsi –op new –mode target –tid 1 -T


You can look at the target:


$ tgtadm –lld iscsi –op show –mode conn –tid 1

Session: 11

Connection: 0


    IP Address:


Next, add a logical unit to the target


$ tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /mnt/disk


Allow any initiator to access the target.


$ tgtadm –lld iscsi –op bind –mode target –tid 1 -I ALL

  1. Now it’s time to set up your client.


Discover your targets.


$ iscsiadm –mode discovery –type sendtargets –portal


Login to your target session.

$ iscsiadm –mode node –targetname –portal –login


You should have a new SCSI disk. You will see it created in /var/log/messages. You will see it in lsblk.


You can send I/O to it:


$ dd if=/dev/zero of=/dev/sda bs=4K count=100


To tear down the session when you are finished:

$ iscsiadm  -m node -T  -p -u



Linux Journal article on making software backed iSCSI targets


How to set up Linux iSCSI targets with tgt



  • 13 Nov 2019
    Announcing Gluster 7.0

    The Gluster community is pleased to announce the release of 7.0, our latest release. This is a major release that includes a range of code improvements and stability fixes along with a few features as noted below. A selection of the key features and bugs addressed are documented in this...

    Read more
  • 15 Oct 2019
    Gluster and CentOS Stream

    Progress cannot be made without change. As technologists, we recognize this every day. Most of the time, these changes are iterative: progresssive additions of features to projects like Gluster. Sometimes those changes are small, and sometimes not. And that’s, of course, just talking about our project. But one of the...

    Read more
  • 26 Apr 2019
    Gluster Monthly Newsletter, April 2...

    Upcoming Community Happy Hour at Red Hat Summit! Tue, May 7, 2019, 6:30 PM – 7:30 PM EDT has all the details. Gluster 7 Roadmap Discussion kicked off for our 7 roadmap on the mailing lists, see [Gluster-users] GlusterFS v7.0 (and v8.0) roadmap discussion for more details. Community...

    Read more