all posts tagged Samba


by on September 30, 2014

Gluster, CIFS, ZFS – kind of part 2

A while ago I put together a post detailing the installation and configuration of 2 hosts running glusterfs, which was then presented as CIFS based storage.

http://jonarcher.info/2014/06/windows-cifs-fileshares-using-glusterfs-ctdb-highly-available-data/

This post gained a bit of interest through the comments and social networks, one of the comments I got was from John Mark Walker suggesting I look at the samba-gluster vfs method instead of mounting the filesystem using fuse (directly access the volume from samba, instead of mounting then presenting). On top of this I’ve also been looking quite a bit at ZFS, whereas previously I had a Linux RAID as the base filesystem. So here is a slightly different approach to my previous post.

Getting prepared

As before, we’re looking at 2 hosts, virtual in the case of this build but more than likely physical in a real world scenario, either way it’s irrelevant. Both of these hosts are running CentOS 6 minimal installs (I’ll update to 7 at a later date), static IP addresses assigned and DNS entries created. I’ll also be running everything under a root session, if you don’t do the same just prefix the commands with sudo. For purposes of this I have also disabled SELINUX and removed all firewall rules. I will one day leave SELINUX enabled in this configuration but for now lets leave it out of the equation.

In my case these names and addresses are as follows:

arcstor01 – 192.168.1.210

arcstor02 – 192.168.1.211

First off lets get the relevant repositories installed (EPEL, ZFS and Gluster)

yum localinstall --nogpgcheck http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm
curl -o /etc/yum.repos.d/gluster.repo http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo
curl -o /etc/yum.repos.d/glusterfs-samba-epel.repo http://download.gluster.org/pub/gluster/glusterfs/samba/EPEL.repo/glusterfs-samba-epel.repo

Local filesystem

As previously mentioned, this configuration will be hosted from 2 virtual machines, each will have 3 disks. 1 for the OS, and the other 2 to be used in a ZFS pool.

First off we need to install ZFS itself, once you have the above zfs-release repo installed this can be done with the following command:

yum install kernel-devel zfs

Perform this on both hosts.

We can now create a zfs pool. In my case the disk device names are vdX but they could be sdX,

fdisk -l

can help you identify the device names, whatever they are just replace them in the following commands.

Create a ZFS pool

zpool create -f  -m /gluster gluster mirror /dev/vdb /dev/vdc

this command will create a zfs pool mounted at /gluster, without -m /gluster it would mount at /{poolname} while in this case it’s the same I just added the option for clarity. The volume name is gluster, the redundancy level is mirrored which is similar to RAID1, there are a number of raid levels available in ZFS all are best explained here: http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/. The final element to the command is where to host the pool, in our case on /dev/vdb and /dev/vdc. The -f option specified is to force creation of the pool, this is required remove the need to create partitions prior to the creation of the pool.

Running the command

zpool status

Will return the status of the created pool, which if successful should look something similar to:

[root@arcstor01 ~]# zpool status
 pool: gluster
 state: ONLINE
 scan: none requested
 config:
NAME STATE READ WRITE CKSUM
 gluster ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 vdb1 ONLINE 0 0 0
 vdc1 ONLINE 0 0 0

errors: No known data errors

A quick ls and df will also show us that the /gluster mountpoint is present and the pool is mounted, the df should show the size as being half the sum of both drives in the pool:

[root@arcstor01 ~]# ls /
 bin boot cgroup dev etc gluster home lib lib64 lost+found media mnt opt proc root sbin selinux srv sys tmp usr var
 [root@arcstor01 ~]# df -h
 Filesystem Size Used Avail Use% Mounted on
 /dev/vda1 15G 1.2G 13G 9% /
 tmpfs 498M 0 498M 0% /dev/shm
 gluster 20G 0 20G 0% /gluster

If this is the case, rinse and repeat on host 2. If this is also successful then we now have a resilient base filesystem on which to host our gluster volumes. There is a bucket load more to ZFS and it’s capabilities but it’s way outside the confines of this configuration, well worth looking into though.

Glusterising our pool

So now we have a filesystem, lets make it better. Next up, installing glusterfs, enabling it then preparing the directories, for this part it is pretty much identical to the previous post:

yum install glusterfs-server -y

chkconfig glusterd on

service glusterd start

mkdir  -p /gluster/bricks/share/brick1

This needs to be done on both hosts.

Now only on host1 lets make the two nodes friends, create and then start the gluster volume:

# gluster peer probe arcstor02
peer probe: success.

# gluster vol create share replica 2 arcstor01:/gluster/bricks/share/brick1 arcstor02:/gluster/bricks/share/brick1
volume create: share: success: please start the volume to access data

# gluster vol start share
volume start: share: success

[root@arcstor01 ~]# gluster vol info share

Volume Name: data1
Type: Replicate
Volume ID: 73df25d6-1689-430d-9da8-bff8b43d0e8b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: arcstor01:/gluster/bricks/share1/brick1
Brick2: arcstor02:/gluster/bricks/share1/brick1

If all goes well above we should have a gluster volume ready to go, this volume will be presented via samba directly. For this configuration a locally available shared area is required, for this we will create another gluster volume to mount locally in which to store lockfiles and shared config files.

mkdir  -p /gluster/bricks/config/brick1
gluster vol create config replica 2 arcstor01:/gluster/bricks/config/brick1 arcstor02:/gluster/bricks/config/brick1
gluster vol start config
mkdir  /opt/samba-config
mount -t glusterfs localhost:config /opt/samba-config

The share volume could probably be used by using a different path in the samba config but for simplicity we’ll keep them seperate for now.
The mountpoint for /opt/samba-config will need to be added to fstab to ensure it mounts at boot time.

echo "localhost:config /opt/samba-config glusterfs defaults,_netdev 0 0" >>/etc/fstab

Should take care of that, remember that needs to be on both hosts.

Samba and CTDB

We now have a highly resilient datastore which could withstand both disk and host downtime, but we need to make that datastore available for consumption and also highly available in the process, for this we will use CTDB, as in the previous post. CTDB is a cluster version of the TDB database which sits under Samba. The majority of this section will be the same as the previous post except for the extra packages and a slightly different config for samba. Lets install the required packages:

yum -y install ctdb samba samba-common samba-winbind-clients samba-client samba-vfs-glusterfs

For the majority of config files we will create them in our shared config volume and symlink them to their expected location. First file we need to create is /etc/sysconfig/ctdb but we will do this as /opt/samba-config/ctdb then link it afterwards

Note: The files which are created in the shared area should be done only on one host, but the linking needs to be done on both.

vi /opt/samba-config/ctdb
CTDB_RECOVERY_LOCK=/opt/samba-config/lockfile
 #CIFS only
 CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
 CTDB_MANAGES_SAMBA=yes
 #CIFS only
 CTDB_NODES=/etc/ctdb/nodes

We’ll need to remove the existing file in /etc/sysconfig then we can create the symlink

rm /etc/sysconfig/ctdb
ln -s /opt/samba-config/ctdb /etc/sysconfig/ctdb

Although we are using Samba the service we will be using is CTDB which allows for the extra clustering components, we need to stop and disable the samba services and enable the ctdb ones:

service smb stop
chkconfig smb off
chkconfig ctdb on

With this configuration being a cluster of essentially a single datapoint we should really use a single entry point, for this a 3rd “floating” or virtual IP address is employed, more than one could be used but lets keep this simple – 192.168.1.212. We also need to create a ctdb config file which contains a list of all the nodes in the cluster. Both these files need to be created in the shared location:

vi /opt/samba-config/public_addresses
192.168.1.212/24 eth0
vi /opt/samba-config/nodes
192.168.1.210
192.168.1.211

They both then need to be linked to their expected locations – neither of these exist so don’t need to be removed.

ln -s /opt/samba-config/nodes /etc/ctdb/nodes
ln -s /opt/samba-config/public_addresses /etc/ctdb/public_addresses

The last step is to modify the samba configuration to present the volume via cifs, I seemed to have issues using a linked file for samba so will only use the shared area for storing a copy of the config which can then be copied to both nodes to keep them identical.

cp /etc/samba/smb.conf /opt/samba-config/

Lets edit that file:

vi /opt/samba-config/smb.conf

Near the top add the following options

clustering = yes
idmap backend = tdb2
private dir = /opt/samba-config/

These turn the clustering (CTDB) features on and specify the shared directory where samba will create lockfiles. You can test starting ctdb at this point to ensure all is working, on both hosts:

cp /opt/samba-config/smb.conf /etc/samba/
service ctdb start

It should start OK, then health status of the cluster can be checked with

ctdb status

At this point I was finding that CTDB was not starting correctly, after a little bit of logwatching I found an error in the samba logs suggesting:

Failed to create pipe directory /run/samba/ncalrpc - No such file or directory

Also, to be search engine friendly the CTDB logfile was outputting

50.samba OUTPUT:ERROR: Samba tcp port 445 is not responding

This was a red herring, the port wasn’t responding as the samba part of CTDB wasn’t starting, 50.samba is a script in /etc/ctdb/events/ which actually starts the smb process.

So I created the directory /run/samba and restarted ctdb and the issue seems to have disappeared.

Now we have a started service, we can go ahead and add the configuration for the share. A regular samba share would look something like:

[share]
 comment = just a share
 path = /share
 read only = no
 guest ok = yes
 valid users = jon

In the previous post this would have been ideal if our gluster volume was mounted at share, but for this we are removing a layer and want samba to talk directly to gluster rather than via the fuse layer. This is achieved using a VFS object, we installed the samba-vfs-glusterfs package earlier. The configuration is slightly different within the smb.conf file also. Adding the following to our file should enable access to the share volume we created:

[share]
 comment = gluster vfs share
 path = /
 read only = No
 guest ok = Yes
 kernel share modes = No
 vfs objects = glusterfs
 glusterfs:loglevel = 7
 glusterfs:logfile = /var/log/samba/glusterfs-testvol.log
 glusterfs:volume = share

Notice the glusterfs: options near the bottom, these are specific to the glusterfs vfs object which is called further up (vfs objects = glusterfs). Another point to note is that the path is / this is relative to the volume rather than the filesystem, so a path to /test would be a test directory inside the gluster volume.

We can now reload the samba config, lets restart for completeness (on both nodes)

service ctdb restart

From a cifs client you should now be able to browse to \192.168.1.212share (or whatever IP you specified as the floating IP).

ctdb

 

All done!

To conclude, here we have created a highly resilient, highly available, very scalable storage solution using some fantastic technologies. We have created a single access method (Cifs on a floating  IP) to a datastore which is then stored on multiple hosts, which in turn store upon multiple disks. Talk about redundancy!

Useful links:

http://www.centos.org

http://zfsonlinux.org/

http://www.gluster.org/

http://ctdb.samba.org/

 

The post Gluster, CIFS, ZFS – kind of part 2 appeared first on Jon Archer.

flattr this!

by on June 30, 2014

Windows (CIFS) fileshares using GlusterFS and CTDB for Highly available data

This tutorial will walk through the setup and configuration of GlusterFS and CTDB to provide highly available file storage via CIFS. GlusterFS is used to replicate data between multiple servers. CTDB provides highly available CIFS/Samba functionality.

Prerequisites:

2 servers (virtual or physical) with RHEL 6 or derivative (CentOS, Scientific Linux). When installing create a partition for root of around 16Gb, but leave a large amount of disk space available for the shared data (you can add this in the installer but ensure the partition type is XFS and that the mountpoint is /gluster/bricks/data1) Once you have an installed system, ensure networking is configured and running, in this example the two servers will be:

server1 = storenode1 – 192.168.1.15

server2 = storenode2 – 192.168.1.16

lets add host entries (unless you have DNS available, in which case add an entry for both hosts in there.

echo "192.168.1.15 storenode1" >> /etc/hosts

echo "192.168.1.16 storenode2" >> /etc/hosts

Next make sure both of your systems are completely up to date:

yum -y update

Reboot if there are any kernel updates.

Filesystem layout

Now we have 2 fully updated working installs its time to start laying out the filesystem, in this instance we will have a partition dedicated to the underlying gluster volume.

If you didn’t add a partition for /gluster/bricks/data1 during the install do this now:

fdisk a partition on the disk (/dev/sda3?)

fdisk /dev/sda mkfs.xfs /dev/sda3

If mkfs.xfs isn’t installed, yum install xfsprogs will add it to your system.If you are running Red Hat you will need to subscribe to the Scalable filesystem channel to get this package.

The directory where this partition will be mounted:

mkdir /gluster/bricks/data1 -p

mount /dev/sda3 /gluster/bricks/data1

If the mount command worked correctly, lets add it to our fstab so it mounts at boot time.

echo "/dev/sda3 /gluster/bricks/data1 xfs default 0 0" >> /etc/fstab

You need to repeat the above steps to partition and mount the volume on server 2.

Introducing Gluster to the equation

Now we have a couple of working filesystems we are ready to bring gluster into the mix, we are going to use the /gluster/bricks/data1 as a location to store our brick for our Gluster volume. A Gluster volume is made up of many bricks, these bricks are essentially a directory on one or more servers that are grouped together to provide a storage array similar to RAID.

In our configuration we will have 2 servers, each with a directory used as a brick to create a replicated gluster volume. Also, for simplicity I have disabled both SELINUX and iptables for this build, however it’s fairly straight forward to get both working correctly with gluster, I may revisit at some point to add this configuration but for now I’m taking the stance that these servers are tucked away safely inside your network behind at least one firewall.

Lets install gluster, on both servers run the following:

cd /etc/yum.repos.d/

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo

yum install glusterfs-server -y

chkconfig glusterd on

service glusterd start

Woohoo, we have Gluster up and running, oh wait it’s not doing anything…

Lets get both servers talking to each other, on the first server run:

gluster peer probe storenode2

gluster-peer-probe

We now need a directory which we will use for the brick in our Gluster volume, run this command on both servers:

mkdir -p /gluster/bricks/data1/brick1

Everything should be now prepared for the volume to be created, run the following command on storenode1

gluster vol create data1 replica 2 storenode1:/gluster/bricks/data1/brick1 storenode2:/gluster/bricks/data1/brick1

gluster-vol-create

This will create a Gluster volume named data1 with 2 replicas which are then specified.

If this command returns ok we should be good to start the volume:

gluster vol start data1

gluster-vol-start

We can check the status of the volume:

gluster vol info data1

gluster-vol-info

Looks good!

Mounting

In order to start using the volume we have just created it needs to be mounted on our systems, lets create a directory on both servers where we will mount the volume:

mkdir /data/data1 -p

We need to ensure the glusterfs client tools are installed (it should have been installed during the initial gluster install, but it’s worth checking)

yum -y install glusterfs-fuse Now lets mount the volume:

mount -t glusterfs storenode1:data1 /data/data1

If that goes well we can add the mount statement to fstab:

echo "storenode1:data /data/data1 glusterfs defaults 0 0" >> /etc/fstab

Then repeat on storenode2:

mount -t glusterfs storenode2:data1 /data/data1

echo "storenode2:data /data/data1 glusterfs defaults 0 0" >> /etc/fstab

We now have a persistent mount for our gluster volume, each server mounts its own presentation of the gluster volume. Notice the mount paths are very similar to NFS, however they are slightly different, the format is hostname:volumename

We can test the Gluster side of things now by creating a file on one server and seeing it exists on the other

[root@storenode1 ~]# echo "hello world" >> /data/data1/test

[root@storenode2 ~]# cat /data/data1/test

gluster-hello-world

If you see the text “hello world” in the output then the Gluster setup is complete!

CTDB and Samba

All the above is good and well, but we need to present this storage to an end user don’t we?

The traditional way to present storage as a file share is using samba, however as we are using multiple servers we want to try and make use of them. This method will use traditional samba config files but using an extra overlay, CTDB. CTDB will present storage via cifs, but also create a VIP (Virtual IP) which “hovers” over the servers configured within.

Lets get the packages installed first:

yum -y install ctdb samba samba-common samba-winbind-clients (Resilient Storage subscription needed for RHEL)

On both nodes backup the default config, just in case:

mv /etc/sysconfig/ctdb{,.old}

CTDB requires a shared area in which to create a lock, and we also need a directory to share

On either node:

mkdir /data/data1/lock

mkdir /data/data1/share

In your favourite editor open /data/data1/lock/ctdb and add the following(In my case Vim):

vi /data/data1/lock/ctdb

CTDB_RECOVERY_LOCK=/data/data1/lock/lockfile
#CIFS only
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_MANAGES_SAMBA=yes
#CIFS only
CTDB_NODES=/etc/ctdb/nodes

The file we have just created will actually replace the config we backed up earlier but that will exist as a symlink (saves multiplication of config files which are the same) on both hosts:

ln -s /data/data1/lock/ctdb /etc/sysconfig/ctdb

Next we need to ensure the samba service won’t start on boot, but in turn the CTDB service will, on both nodes:

service smb stop

chkconfig smb off

chkconfig ctdb on

The /etc/ctdb/public_addresses file will contain a list of IP addresses which will be used as VIP’s, you can use as many as you like here, some configurations use multiple combinations of VIPs with round-robin DNS for true load-balanced scenarios, for our simple config we will just use the next IP. Note we are creating the file on our shared storage again to ensure that we have the same config on both boxes and will be later linked:

vi /data/data1/lock/public_addresses

192.168.1.17/24 eth0

Now we need to create the /etc/ctdb/nodes which contains the IP addresses of all servers which will present the storage, again this will be a shared file and linked:

vi /data/data1/lock/nodes

192.168.1.15

192.168.1.16

Lets link those two files, on both nodes:

ln -s /data/data1/lock/nodes /etc/ctdb/nodes

ln -s /data/data1/lock/public_addresses /etc/ctdb/public_addresses

The only thing we have left to do now is to modify the samba config file, there are 2 sections we are interested in. Firstly the general config section where we need to enable clustering and point it to the lock directory. Samba (or CTDB in this case) has some strange side effects if shared storage is used, however, it could be used to edit then copy in to place:

On storage node 1:

cp /etc/samba/smb.conf /data/data1/lock/smb.conf

vi /data/data1/lock/smb.conf

And add in the general section near the top:

clustering = yes

idmap backend = tdb2

private dir = /data/data1/lock

The second component is to create the share itself:

[share] comment = Gluster and CTDB based share
path = /data/data1/share
read only = no
guest ok = yes
valid users = jon

Once we are happy with the edit, the file can be copied to the correct location, on both hosts:

cp /data/data1/lock/smb.conf /etc/samba/

We need to ensure the user jon exists on both servers:

useradd jon

smbpasswd -a jon

and type a password.

Configuration is now done, all that is left to do is start the service, on both nodes:

service ctdb start

If the service starts successfully then after a short while the share becomes available, monitor its status using:

ctdb status

gluster-ctdb-status

Once both nodes get OK, we’re good to go. The share will now be accessible from a Windows PC (or anything that can access SMB/CIFS) using \\192.168.1.17\share

gluster-fileshare

If either storage server becomes unavailable the share will still exist.

 

We now have a resilient, highly available CIFS file server.

The post Windows (CIFS) fileshares using GlusterFS and CTDB for Highly available data appeared first on Jon Archer.

flattr this!