Citrix Linux VDA Centralized Home Directories – Part 2: Guest Blog Post by Brian Macbeth
Over the last several years, I’ve had the honor of working with a talented and focused engineer named Brian Macbeth. Brian has spent a bit of time working with Citrix’s new Linux VDA technology and has run into a number of limitations in the area of roaming profiles and home directories, that he has resolved using the processes described below. Brian has graciously volunteered to share his experiences with the extended virtualization community in an attempt to further advance the use of Linux VDA in enterprise organizations. Without further ado, below is the guest blog post by Brian Macbeth, feel free to comment below!
Configuring a High Availability NFS Server Cluster
Welcome to part 2 of Citrix Linux VDA Centralized Home Directories blog post series. In this post, I will describe how to stand up a high availability two-node Linux NFS server cluster using a shared iSCSI LUN via software initiator.
Installing a Linux HA cluster for the first time can be challenging endeavor; while there is a lot of info on the web, much of it is fragmented or dated, making the build troubling. This post should set you on the path to successfully implementing high availability centralized NFS home directories.
I highly recommend getting comfortable with managing a Linux cluster in a non-production environment before moving into production. A clustered Linux NFS server failover is not nearly as seamless as a SMB/CIFS share failover on Windows failover cluster. That said, you will need to understand how the NFS client “pause” during failovers will impact the user community and tune the NFS client recovery and timeout configurations to best meet your environment needs and expectations. The NFS man page should be your new friend.
|Operating System:||CentOS 7, domain joined|
|Node 1 FQDN/IP:||vdinfs01a.demo.private 10.1.10.80|
|Node 2 FQDN/IP:||vdinfs01a.demo.private 10.1.10.81|
|Cluster Virtual IP:||vdinfs01.demo.private 10.1.10.82|
|iSCSI Target IP/LUN:||IP: 10.1.10.10 LUN Size: 50GB|
|Volume Group Name:||volgrp1|
|Logical Volume Name:||nfshome_vol1|
|NFS Mount Folder:||nfsmount|
Cluster Node Base Configs
Perform the following configurations on both cluster nodes.
Host File Configuration
I originally had the host file configured the same way I configured for the Linux VDAs, which caused intra-cluster communication problems between the nodes. It turns out, if the loopback address is associated with the hostname, Corosync seems to bind to the loopback address and then cannot reach the other node. At a bare minimum, you only need to list the local node in its respected hosts file; however, listing both nodes ensures name resolution in the event of any DNS server connectivity issues.
Edit /etc/hosts using your editor of choice so that it looks like this:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
10.1.10.80 vdinfs01a.demo.private vdinfs01a
10.1.10.81 vdinfs01b.demo.private vdinfs01b
Disable Virtual Interface
I recommend disabling the virtual interface to prevent any chance of intra-cluster communication issues, similar to the issue experienced with incorrectly configured hosts files.
systemctl disable libvirtd.service
Disable IPv6 (Optional)
Add the following line to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
Edit /etc/sysconfig/network-scripts/ifcfg-<interfacename> and change IPV6INIT to:
The following firewall configurations allow intra-cluster communication and client access to NFS resources:
firewall-cmd –permanent –add-service=high-availability
firewall-cmd –permanent –add-port=111/tcp
firewall-cmd –permanent –add-port=54302/tcp
firewall-cmd –permanent –add-port=20048/tcp
firewall-cmd –permanent –add-port=2049/tcp
firewall-cmd –permanent –add-port=46666/tcp
firewall-cmd –permanent –add-port=42955/tcp
firewall-cmd –permanent –add-port=875/tcp
yum -y install iscsi-initiator-utils
yum -y install pcs pacemaker fence-agents-all
shutdown -r 0
Obtain the Node IQNs
Run the following command on both cluster nodes and make note of the IQN:
Note: if the machines are cloned, you will need to modify initiatorname.iscsi to ensure the IQNs are unique between the cluster nodes.
Provision iSCSI LUN
Provision the iSCSI LUN per your storage system requirements to each cluster node.
Connect Nodes to the iSCSI Target LUN
On both cluster nodes, perform the following command and make note of the IQN:
iscsiadm -m discovery -t sendtargets -p <storage.IP>
On both cluster nodes, perform the following command to connect to the iSCSI LUN:
iscsiadm -m node -T iqn.1994-11.com.netgear:nas:712ca734:vdinfs01 –login
Find the iSCSI disk name
On both cluster nodes, run the following command to find the device associated with the iSCSI LUN, such as /dev/sdb:
grep “Attached SCSI” /var/log/messages
The devices should be the same on each node.
Configure the Volume
On one cluster node, perform the following steps to configure the volume from shared iSCSI storage using the disk path found in the previous step.
Create an LVM physical volume on partition /dev/<sdb>:
Create a volume group with a name of your choosing:
vgcreate volgrp1 /dev/<sdb>
Create a logical volume with a name of your choosing using all available space:
lvcreate -l 100%FREE -n nfshome_vol1 volgrp1
Create the filesystem on the logical volume:
NFS Shared Folder Configuration
On the same cluster node the volume was configured on, perform the following steps to configure the NFS shared folder.
Create a folder that will be used to mount the shared volume to:
Mount the logical volume to the mount folder:
mount /dev/volgrp1/nfshome_vol1 /nfsmount
Create the shared folder structure and add some test files:
mkdir -p /nfsmount/exports/nfshome
chmod 777 /nfsmount/exports/nfshome
Unmount the filesystem and deactivate the volume group:
vgchange -an volgrp1
Configure Volume Group for Cluster-only Activation
The following configurations are required to ensure that only the active cluster node will mount the shared volume. Otherwise both nodes would activate the volume, which could lead to corruption.
On both cluster nodes, run the following commands:
Ensure that locking_type is set to 1 and that use_lvmetad is set to 0:
lvmconf –enable-halvm –services –startstopservices
Run the following command to list the volume groups configured on the server:
vgs –noheadings -o vg_name
Edit /etc/lvm/lvm.conf uncomment the volume_list = line and add any configured volumes other than the volume created for NFS storage:
volume_list = [ “volume_name1”, “volume_name2” ]
or, if there are no volumes activated outside the cluster:
volume_list = 
Rebuild the initramfs boot image to prevent the boot image from activating a volume group controlled by the cluster (this can take a minute to complete):
dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)
Restart the nodes:
restart -h 0
Configure the Cluster
Prior to configuring the cluster, validate the nodes can communicate with each other using both fqdn and host name.
Assign a Password to the Cluster Admin Account
On each cluster node, configure the same password for the hacluster user account:
Enable and configure the cluster service to autostart
On both cluster nodes, run the following commands to start and enable pcs:
systemctl start pcsd.service
systemctl enable pcsd.service
Authenticate the Cluster Admin Account with the Cluster
Run the following command from the node you will be running pcs from (vdinfs01a.demo.private):
pcs cluster auth vdinfs01a vdinfs01b
Create the Cluster
Run the following command from vdinfs01a.demo.private with a cluster name of your choosing:
pcs cluster setup –start –name nfshome vdinfs01a vdinfs01b
Enable Cluster Services to Run on Each Node at Boot
Run the following command from vdinfs01a.demo.private:
pcs cluster enable –all
Disable Fencing (Temporarily)
Node fencing will be temporarily disabled and configured later:
pcs property set stonith-enabled=false
Disable quorum for a two-node cluster:
pcs property set no-quorum-policy=ignore
Run pcs status to validate the cluster configuration. At this point you should see both nodes online with 0 resources configured.
# pcs status
Cluster name: nfshome
Current DC: vdinfs01b (version 1.1.15-11.el7_3.4-e174ec8) – partition with quorum
Last updated: Wed May 24 08:13:34 2017 Last change: Wed May 24 08:12:37 2017 by root via cibadmin on vdinfs01a
2 nodes and 0 resources configured
Online: [ vdinfs01a vdinfs01b ]
Configure Cluster Resources
From one cluster node, add the LVM (volume group) resource:
pcs resource create volgrp1 LVM volgrpname=volgrp1 exclusive=true –group nfsgroup
Add a filesystem resource:
pcs resource create nfshome Filesystem device=/dev/volgrp1/nfshome_vol1 directory=/nfsmount fstype=ext4 –group nfsgroup
Add the NFS server resource:
pcs resource create nfs-daemon nfsserver nfs_shared_infodir=/nfsmount/nfsinfo nfs_no_notify=true –group nfsgroup
Note: the nfsinfo directory will store various NFS server information such as the export file and state info used during failover.
Add the exportfs resources to the export file (assumes clients only reside on the 10.1.10.0/24 network):
pcs resource create nfs-root exportfs clientspec=10.1.10.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsmount/exports/nfshome fsid=0 –group nfsgroup
See part 1 for information on NFS export option configs.
Add the VIP resource:
pcs resource create nfs_ip IPaddr2 ip=10.1.10.82 cidr_netmask=24 –group nfsgroup
Note: Ensure DNS forward and reverse records are configured if NFS clients will be using an FQDN to connect to the NFS cluster.
Add nfsnotify resource:
pcs resource create nfs-notify nfsnotify source_host-10.1.10.82 –group nfsgroup
Run the following command to validate the new resources are configured and started:
Run the following command to prevent resources from moving after node recovery:
pcs resource defaults resource-stickiness=100
Fencing is the cluster function of isolating a malfunctioning node with STONITH (Shoot The Other Node In The Head) to take a malfunctioning node offline. Since the cluster nodes are virtual machines and we don’t have access to power controllers or out-of-band management cards such as iLO or DRAC, we will use fence_scsi and associate it to the shared iSCSI LUN.
Run the following command to the find the iSCSI device by WWN:
Build your fencing configuration by including the cluster node names and the WWN:
pcs stonith create iscsi-stonith-device fence_scsi pcmk_host_list=”vdinfs01a vdinfs01b” pcmk_reboot_action=”off” devices=/dev/disk/by-id/wwn-0x6001405b6e8ef8f72eb404182ac935d7 meta provides=unfencing
Note: this fencing configuration will require that you restart the malfunctioning node, even if you resolve the malfunction.
Test Cluster Resources
Cluster Failover Tests
Use pcs status to find the active cluster node and then run the following command to put the active node in standby mode:
pcs cluster standby vdinfs01a
Run pcs status again and you should see the cluster resources running on the other node and only one of the two nodes will show online.
Run the following command to make the standby node available again:
pcs cluster unstandby vdinfs01a
Repeat the above steps for the other node.
NFS Client Access Test
Run the following command to mount the NFS share:
mount -o “vers=4” 10.1.10.82:/ /nfstest
Validate the previously created files are accessible
Validate access after cluster failover by putting the active node in standby.
Note: NFS4 can take upwards of 90 seconds (or more) to recover from the failover.
NFS Client Configuration
It is assumed the NFS client machine is already joined to the domain.
Install NFS Package
yum -y install nfs-utils
If selinux is enabled in your configuration, run the following command to allow NFS home directory functionality and restart the machine:
setsebool -P use_nfs_home_dirs 1
shutdown -r 0
Configure NFS Mount
This example assumes:
- A domain user has already logged on to the machine and the /home/<DOMAIN> directory has been created.
- The exported NFS directory is nfshome and the home directory path is /home/DEMO.
Edit /etc/fstab with your editor of choice and add the following to the end of the file:
vdinfs01.demo.private:/ /home/DEMO nfs4 _netdev,vers=4.0,rw,soft,timeo=300,retrans=2,local_lock=none 0 0
Note: if you only have one NFS export defined, your clients will need to mount “/” from the cluster instead of “/nfshome” as indicated in part 1 with a stand-alone NFS server.
See part 1 for information on NFS mount option configs.
Run the following command to mount the NFS export to /home/<DOMAIN>:
Run the following command to validate that /home/<DOMAN> is mounted to the NFS export:
See part 1 for validation steps, it is the same regardless if home directories are being serviced from stand-alone or clustered NFS servers.
This concludes the two part blog series on deploying a NFS Share for Linux VDA home directories! If you have any questions or comments, please feel free to leave feedback below. Thanks!