Over the last several years, I’ve had the honor of working with a talented and focused engineer named Brian Macbeth. Brian has spent a bit of time working with Citrix’s new Linux VDA technology and has run into a number of limitations in the area of roaming profiles and home directories, that he has resolved using the processes described below.  Brian has graciously volunteered to share his experiences with the extended virtualization community in an attempt to further advance the use of Linux VDA in enterprise organizations. Without further ado, below is the guest blog post by Brian Macbeth, feel free to comment below!

---

Part 1: Citrix Linux VDA Centralized Home Directories

Part 2: Configuring a High Availability NFS Server Cluster (This post)

Configuring a High Availability NFS Server Cluster

Welcome to part 2 of Citrix Linux VDA Centralized Home Directories blog post series.  In this post, I will describe how to stand up a high availability two-node Linux NFS server cluster using a shared iSCSI LUN via software initiator.

Installing a Linux HA cluster for the first time can be challenging endeavor; while there is a lot of info on the web, much of it is fragmented or dated, making the build troubling.  This post should set you on the path to successfully implementing high availability centralized NFS home directories.

I highly recommend getting comfortable with managing a Linux cluster in a non-production environment before moving into production. A clustered Linux NFS server failover is not nearly as seamless as a SMB/CIFS share failover on Windows failover cluster.  That said, you will need to understand how the NFS client “pause” during failovers will impact the user community and tune the NFS client recovery and timeout configurations to best meet your environment needs and expectations. The NFS man page should be your new friend.

Environment

Operating System:	CentOS 7, domain joined
Node 1 FQDN/IP:	vdinfs01a.demo.private 10.1.10.80
Node 2 FQDN/IP:	vdinfs01a.demo.private 10.1.10.81
Cluster Virtual IP:	vdinfs01.demo.private 10.1.10.82
iSCSI Target IP/LUN:	IP: 10.1.10.10  LUN Size: 50GB
Volume Group Name:	volgrp1
Logical Volume Name:	nfshome_vol1
NFS Mount Folder:	nfsmount

Cluster Node Base Configs

Perform the following configurations on both cluster nodes.

Host File Configuration

I originally had the host file configured the same way I configured for the Linux VDAs, which caused intra-cluster communication problems between the nodes.  It turns out, if the loopback address is associated with the hostname, Corosync seems to bind to the loopback address and then cannot reach the other node.  At a bare minimum, you only need to list the local node in its respected hosts file; however, listing both nodes ensures name resolution in the event of any DNS server connectivity issues.

Edit /etc/hosts using your editor of choice so that it looks like this:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

10.1.10.80  vdinfs01a.demo.private vdinfs01a

10.1.10.81  vdinfs01b.demo.private vdinfs01b

Disable Virtual Interface

I recommend disabling the virtual interface to prevent any chance of intra-cluster communication issues, similar to the issue experienced with incorrectly configured hosts files.

systemctl disable libvirtd.service

Disable IPv6 (Optional)

Add the following line to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1

Edit /etc/sysconfig/network-scripts/ifcfg-<interfacename> and change IPV6INIT to:

IPV6INIT=no

Firewall Configuration

The following firewall configurations allow intra-cluster communication and client access to NFS resources:

firewall-cmd –permanent –add-service=high-availability

firewall-cmd –permanent –add-port=111/tcp

firewall-cmd –permanent –add-port=54302/tcp

firewall-cmd –permanent –add-port=20048/tcp

firewall-cmd –permanent –add-port=2049/tcp

firewall-cmd –permanent –add-port=46666/tcp

firewall-cmd –permanent –add-port=42955/tcp

firewall-cmd –permanent –add-port=875/tcp

firewall-cmd –reload

Software Installation

yum -y install iscsi-initiator-utils

yum -y install pcs pacemaker fence-agents-all

Restart Node

shutdown -r 0

iSCSI Storage

Obtain the Node IQNs

Run the following command on both cluster nodes and make note of the IQN:

cat /etc/iscsi/initiatorname.iscsi

Note: if the machines are cloned, you will need to modify initiatorname.iscsi to ensure the IQNs are unique between the cluster nodes.

Provision iSCSI LUN

Provision the iSCSI LUN per your storage system requirements to each cluster node.

Connect Nodes to the iSCSI Target LUN

On both cluster nodes, perform the following command and make note of the IQN:

iscsiadm -m discovery -t sendtargets -p <storage.IP>

On both cluster nodes, perform the following command to connect to the iSCSI LUN:

iscsiadm -m node -T iqn.1994-11.com.netgear:nas:712ca734:vdinfs01 –login

Find the iSCSI disk name

On both cluster nodes, run the following command to find the device associated with the iSCSI LUN, such as /dev/sdb:

grep “Attached SCSI” /var/log/messages

The devices should be the same on each node.

Configure the Volume

On one cluster node, perform the following steps to configure the volume from shared iSCSI storage using the disk path found in the previous step.

Create an LVM physical volume on partition /dev/<sdb>:

pvcreate /dev/<sdb>

Create a volume group with a name of your choosing:

vgcreate volgrp1 /dev/<sdb>

Create a logical volume with a name of your choosing using all available space:

lvcreate -l 100%FREE -n nfshome_vol1 volgrp1

Create the filesystem on the logical volume:

mkfs.ext4 /dev/volgrp1/nfshome_vol1

NFS Shared Folder Configuration

On the same cluster node the volume was configured on, perform the following steps to configure the NFS shared folder.

Create a folder that will be used to mount the shared volume to:

mkdir /nfsmount

Mount the logical volume to the mount folder:

mount /dev/volgrp1/nfshome_vol1 /nfsmount

Create the shared folder structure and add some test files:

mkdir -p /nfsmount/exports/nfshome

chmod 777 /nfsmount/exports/nfshome

touch /nfsmount/exports/nfshome/file1

touch /nfsmount/exports/nfshome/file2

Unmount the filesystem and deactivate the volume group:

umount /nfsmount

vgchange -an volgrp1

Configure Volume Group for Cluster-only Activation

The following configurations are required to ensure that only the active cluster node will mount the shared volume. Otherwise both nodes would activate the volume, which could lead to corruption.

On both cluster nodes, run the following commands:

Ensure that locking_type is set to 1 and that use_lvmetad is set to 0:

lvmconf –enable-halvm –services –startstopservices

Run the following command to list the volume groups configured on the server:

vgs –noheadings -o vg_name

Edit /etc/lvm/lvm.conf uncomment the volume_list = line and add any configured volumes other than the volume created for NFS storage:

volume_list = [ “volume_name1”, “volume_name2” ]

or, if there are no volumes activated outside the cluster:

volume_list = []

Rebuild the initramfs boot image to prevent the boot image from activating a volume group controlled by the cluster (this can take a minute to complete):

dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

Restart the nodes:

restart -h 0

Configure the Cluster

Prior to configuring the cluster, validate the nodes can communicate with each other using both fqdn and host name.

Assign a Password to the Cluster Admin Account

On each cluster node, configure the same password for the hacluster user account:

passwd hacluster

Enable and configure the cluster service to autostart

On both cluster nodes, run the following commands to start and enable pcs:

systemctl start pcsd.service

systemctl enable pcsd.service

Authenticate the Cluster Admin Account with the Cluster

Run the following command from the node you will be running pcs from (vdinfs01a.demo.private):

pcs cluster auth vdinfs01a vdinfs01b

username: hacluster

Create the Cluster

Run the following command from vdinfs01a.demo.private with a cluster name of your choosing:

pcs cluster setup –start –name nfshome vdinfs01a vdinfs01b

Enable Cluster Services to Run on Each Node at Boot

Run the following command from vdinfs01a.demo.private:

pcs cluster enable –all

Disable Fencing (Temporarily)

Node fencing will be temporarily disabled and configured later:

pcs property set stonith-enabled=false

Disable Quorum

Disable quorum for a two-node cluster:

pcs property set no-quorum-policy=ignore

Cluster Validation

Run pcs status to validate the cluster configuration.  At this point you should see both nodes online with 0 resources configured.

# pcs status

Cluster name: nfshome

Stack: corosync

Current DC: vdinfs01b (version 1.1.15-11.el7_3.4-e174ec8) – partition with quorum

Last updated: Wed May 24 08:13:34 2017          Last change: Wed May 24 08:12:37 2017 by root via cibadmin on vdinfs01a

2 nodes and 0 resources configured

Online: [ vdinfs01a vdinfs01b ]

No resources

Daemon Status:

corosync: active/enabled

pacemaker: active/enabled

pcsd: active/enabled

Configure Cluster Resources

From one cluster node, add the LVM (volume group) resource:

pcs resource create volgrp1 LVM volgrpname=volgrp1 exclusive=true –group nfsgroup

Add a filesystem resource:

pcs resource create nfshome Filesystem device=/dev/volgrp1/nfshome_vol1 directory=/nfsmount fstype=ext4 –group nfsgroup

Add the NFS server resource:

pcs resource create nfs-daemon nfsserver nfs_shared_infodir=/nfsmount/nfsinfo nfs_no_notify=true –group nfsgroup

Note: the nfsinfo directory will store various NFS server information such as the export file and state info used during failover.

Add the exportfs resources to the export file (assumes clients only reside on the 10.1.10.0/24 network):

pcs resource create nfs-root exportfs clientspec=10.1.10.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsmount/exports/nfshome fsid=0 –group nfsgroup

See part 1 for information on NFS export option configs.

Add the VIP resource:

pcs resource create nfs_ip IPaddr2 ip=10.1.10.82 cidr_netmask=24 –group nfsgroup

Note: Ensure DNS forward and reverse records are configured if NFS clients will be using an FQDN to connect to the NFS cluster.

Add nfsnotify resource:

pcs resource create nfs-notify nfsnotify source_host-10.1.10.82 –group nfsgroup

Run the following command to validate the new resources are configured and started:

pcs status

Run the following command to prevent resources from moving after node recovery:

pcs resource defaults resource-stickiness=100

Configure Fencing

Fencing is the cluster function of isolating a malfunctioning node with STONITH (Shoot The Other Node In The Head) to take a malfunctioning node offline.  Since the cluster nodes are virtual machines and we don’t have access to power controllers or out-of-band management cards such as iLO or DRAC, we will use fence_scsi and associate it to the shared iSCSI LUN.

Run the following command to the find the iSCSI device by WWN:

ls /dev/disk/by-id/wwn-*

Build your fencing configuration by including the cluster node names and the WWN:

pcs stonith create iscsi-stonith-device fence_scsi pcmk_host_list=”vdinfs01a vdinfs01b” pcmk_reboot_action=”off” devices=/dev/disk/by-id/wwn-0x6001405b6e8ef8f72eb404182ac935d7 meta provides=unfencing

 Note: this fencing configuration will require that you restart the malfunctioning node, even if you resolve the malfunction.

Test Cluster Resources

Cluster Failover Tests

Use pcs status to find the active cluster node and then run the following command to put the active node in standby mode:

pcs cluster standby vdinfs01a

Run pcs status again and you should see the cluster resources running on the other node and only one of the two nodes will show online.

Run the following command to make the standby node available again:

pcs cluster unstandby vdinfs01a

Repeat the above steps for the other node.

NFS Client Access Test

Run the following command to mount the NFS share:

mkdir /nfstest

mount -o “vers=4” 10.1.10.82:/ /nfstest

Validate the previously created files are accessible

ls /nfstest

Validate access after cluster failover by putting the active node in standby.

Note: NFS4 can take upwards of 90 seconds (or more) to recover from the failover.

NFS Client Configuration

It is assumed the NFS client machine is already joined to the domain.

Install NFS Package

yum -y install nfs-utils

Selinux Configuration

If selinux is enabled in your configuration, run the following command to allow NFS home directory functionality and restart the machine:

setsebool -P use_nfs_home_dirs 1

shutdown -r 0

Configure NFS Mount

This example assumes:

1. A domain user has already logged on to the machine and the /home/<DOMAIN> directory has been created.
2. The exported NFS directory is nfshome and the home directory path is /home/DEMO.

Edit /etc/fstab with your editor of choice and add the following to the end of the file:

vdinfs01.demo.private:/ /home/DEMO nfs4 _netdev,vers=4.0,rw,soft,timeo=300,retrans=2,local_lock=none  0 0

Note: if you only have one NFS export defined, your clients will need to mount “/” from the cluster instead of “/nfshome” as indicated in part 1 with a stand-alone NFS server.

See part 1 for information on NFS mount option configs.

Run the following command to mount the NFS export to /home/<DOMAIN>:

mount -a

Run the following command to validate that /home/<DOMAN> is mounted to the NFS export:

df -h

Validation

See part 1 for validation steps, it is the same regardless if home directories are being serviced from stand-alone or clustered NFS servers.

Conclusion

This concludes the two part blog series on deploying a NFS Share for Linux VDA home directories! If you have any questions or comments, please feel free to leave feedback below. Thanks!