The Gluster Blog

Gluster blog stories provide high-level spotlights on our users all over the world

A Gluster Block Interface – Performance and Configuration

Gluster
November 26, 2013

This post shares some experiences I’ve had in simulating a block device in gluster.

The block device is a file based image, which acts as the backend for the Linux SCSI target. The file resides in gluster, so enjoys gluster’s feature set. But the client only sees a block device. The Linux SCSI target presents it over iSCSI.

Some more information on how to set this up is at the bottom of this post.

But where to mount gluster?

There are three options to consider. Call them client, server, and gateway.

The configurations are depicted in the diagram below:

  • Configuring the block device at the server means the client only requires an iSCSI initiator.

  • Configuring the block device at the client allows I/O to fan out to different nodes.

  • Configuring the block device at a gateway allows I/O to fan out to different nodes without changing the client.

 

These options have their pros and cons. For example, a gateway minimizes client overhead while providing fan-out. OTOH, the customer must provide an additional node.

In my case, the objective was to minimize customer burden. Server-side configuration is probably the best choice. After chatting with some colleagues here at Red Hat, thats what we settled on.

I ran some simple performance tests using the “fio” tool to generate I/O.

  • Up to 10 fio processes were started.

  • The queue depth for each was 32.

  • Each process sends I/O to its own slice of the volume.

  • client and server block cache is flushed between tests

    • echo 3 > /proc/sys/vm/drop_caches

  • 64k records

  • 50G volume replicated over two nodes

  • Gluster version: 3.4.0.33

 

Warning!

 

  • Only two nodes were used in performance testing. Ideally more nodes would be used, but that equipment is not readily available.

  • There are numerous options to tune, including the queue depth, number of streams, number of paths, gluster volume configuration, block size, number of targets etc. Parameters were chosen based on trial and error rather than formal methodology.

 

chart_2

 

1 KVM client – 6VCPU x 20GB client (pinned)  <===> 2 file servers baremetal 12x20GB

4 x 64GB  blk virtio file systems (cache=none)

(1 x 12disk RAID6 gluster brick / server)

—————–

For the interested, here is a cookbook to set it up. For more information, see [1] and [2].

 

  1. Mount gluster locally on your server.

 

$ mount -t glusterfs 127.0.0.1:gserver /mnt

 

  1. Create a large file representing your block device within the gluster fs.

 

$ dd if=/dev/zero of=/mnt/disk bs=2G count=25

 

  1. Create a target using the file as the backend storage.

 

If necessary, download the Linux SCSI target. Then start the service.

 

$ yum install scsi-target-utils

$ service tgtd start

 

You must give an iSCSI Qualified name (IQN), in the format :

 

  iqn.yyyy-mm.reversed.domain.name:OptionalIdentifierText

where:

  yyyy-mm represents the 4-digit year and 2-digit month the device was started (for example: 2011-07);

 

$ tgtadm –lld iscsi –op new –mode target –tid 1 -T iqn.20013-10.com.redhat

 

You can look at the target:

 

$ tgtadm –lld iscsi –op show –mode conn –tid 1

Session: 11

Connection: 0

    Initiator: iqn.1994-05.com.redhat:cf75c8d4274d

    IP Address: 172.17.11.25

 

Next, add a logical unit to the target

 

$ tgtadm –lld iscsi –op new –mode logicalunit –tid 1 –lun 1 -b /mnt/disk

 

Allow any initiator to access the target.

 

$ tgtadm –lld iscsi –op bind –mode target –tid 1 -I ALL

  1. Now it’s time to set up your client.

 

Discover your targets.

 

$ iscsiadm –mode discovery –type sendtargets –portal 192.168.1.2

 

Login to your target session.

$ iscsiadm –mode node –targetname iqn.2001-04.com.example:storage.disk1.amiens.sys1.xyz –portal 192.168.1.2:3260 –login

 

You should have a new SCSI disk. You will see it created in /var/log/messages. You will see it in lsblk.

 

You can send I/O to it:

 

$ dd if=/dev/zero of=/dev/sda bs=4K count=100

 

To tear down the session when you are finished:

$ iscsiadm  -m node -T iqn.2001-04.com.redhat  -p 192.168.1.2 -u

 

[1]

Linux Journal article on making software backed iSCSI targets

[2]

How to set up Linux iSCSI targets with tgt

 

BLOG

  • 06 Nov 2018
    Gluster Monthly Newsletter, October...

    Gluster Monthly Newsletter, October 2018 Gluster 5 is out and our retrospective is currently open! This feedback is anonymous and goes to our release team. https://lists.gluster.org/pipermail/gluster-users/2018-October/035171.html https://www.gluster.org/gluster-5-0-retrospective/ Upcoming Community Meeting  – November 7, November 21 – 15:00 UTC in #gluster-meeting on freenode. https://bit.ly/gluster-community-meetings has the agenda. We’re participating in Outreachy...

    Read more
  • 23 Oct 2018
    Announcing Gluster 5

    The Gluster community is pleased to announce the release of 5.0, our latest release. This is a major release that includes a range of code improvements and stability fixes with some management and standalone features as noted below. A selection of the key features and changes are documented on this...

    Read more