Provisioning Services High Availability with a Read-Only vDisk Store
Provisioning Services 5.1 introduced another storage option for a vDisk store. The ability to use a read-only vDisk store. Provisioning Services 5.1 has added some nice new features to the product and the ability to use a read-only vDisk store is another one of those features. For more information on the new features in Provisioning Services 5.1 you can read one of my previous blogs Have you upgraded to Provisioning Services 5.1 yet? Why Not?. A read-only vDisk store is only required when you want to give direct read/write access to a SAN volume without a shared/clustered file system to all of your Provisioning Servers for high availability and load balancing in your Provisioning Services environment. In this post I am going to go over why you would use a read-only vDisk store, the limitations, the requirements, the steps involved for setting up, and the steps involved for updating a read-only vDisk store.
Why use a Read-Only vDisk Store?
A read-only vDisk store is only required when you want to have your Provisioning Servers directly access a SAN volume without a shared/clustered file system. If you gave all of your Provisioning Servers direct read/write access to the same SAN volume without a shared/clustered file system, you would corrupt the data on the volume. The data corruption comes from multiple server direct read/write access to the same volume that doesn’t have a shared/clustered file system. A shared/clustered file system is required to give direct read/write access to the same SAN volume for multiple Provisoning Servers. There are third party products such as Melio FS from Sanbolic that allow you to give multiple Provisioning Servers direct read/write access to the same SAN volume without corrupting data on the volume. Using a read-only vDisk store allows to have multiple Provisioning Servers directly access the same SAN volume without data corruption and without having to purchase a third party product such as Melio FS from Sanbolic.
So let’s go a little deeper into a shared/clustered file system to understand why a read-only vDisk store is needed. A shared/clustered file system is a storage file system that can be concurrently accessed for both reads and writes by multiple servers. Shared/clustered file systems are needed because without them servers accessing the same volume concurrently with read/write access would corrupt the data on the volume. This is due to the fact that without a shared/clustered file system there is nothing to prevent multiple servers from making modifications to the same part of the file system at the same time. Because of the way that server operating systems manipulate the file system directly, conventional file locking provides no aid in this since file locking operates above the file system level. A shared/clustered file system extends the file system by providing concurrency control. This prevents corruption and unintended data loss by providing each device accessing the file system with consistent and serializable views of the file system. Usually some sort of fencing is used by the file system to prevent data corruption in case a server accessing the volume fails. Now you know what a shared/clustered file system does you should understand why a read-only vDisk store is needed without one.
Using a read-only vDisk store creates a more complex Provisioning Services environment and administrative overhead. The Provisioning Services environment complexity depends on what vDisk access mode you want to use and where you want to place the target device write cache. There are some limitations for vDisk access modes and write cache placement when using a read-only vDisk store. The administrative overhead is when you want to modify the vDisk properties and/or update vDisks on the vDisk store. The vDisk store has to be placed in read/write mode to modify vDisk properties and/or update the vDisks. Maintenance plans and windows have to be strictly defined for your Provisioning Services enviroment when modifying the vDisk properties and/or updating vDisks because of the limitations of a read-only vDisk store. Only a single Provisioning Server can access the read-only vDisk store when the store is in read/write mode and all other Provisioning Servers access to the read-only vDisk store must be taken offline. If the steps required to update the read-only vDisk store are not followed, then you could possibly corrupt the volume and all the vDisks on it.
What are the limitations of a Read-Only vDisk Store?
Using a read-only vDisk store has its limitations. Some of things to consider when a using read-only vDisk store include vDisk access modes, target device write cache, modifying vDisk properties, and updating vDisks. The limitations when using a read-only vDisk store are:
- Private Image (single device, R/W access) access mode vDisks are not supported on a read-only vDisk store. This also includes booting to a Private Image vDisk. Master vDisk images will have to be created on a read/write vDisk store.
- Standard Image (multi-device, write-cache enabled) access mode vDisks with write cache or encrypted write cache on the Provisioning Servers are supported but a separate shared write cache location is required for the read-only vDisk store. The write cache has to be placed in a shared read/write location outside of the read-only vDisk store.
- Standard Image (multi-device, write cache enabled) access mode vDisks with write cache or encrypted write cache on the target device’s hard drive are supported but a separate shared fail back write cache location is required for the read-only vDisk store. The fail back write cache has to be placed in a shared read/write location outside of the read-only vDisk store. This is needed because if the target device’s hard drive is not found or fails then the write cache will fail back to the Provisioning Server.
- Differencing Disk Image access mode vDisks are supported but a separate shared write cache location is required for the read-only vDisk store. The write cache has to be placed in a shared read/write location outside of the read-only vDisk store. This is needed because write cache is on the Provisioning Servers when Differencing Disk Image access mode is used.
- Modifying vDisk properties can’t be done when the vDisk store is set to read-only. All Provisioning Servers (except for one) have to be taken offline and the vDisk store has to be changed to read/write to modify vDisk properties.
- Mounting vDisks on the Provisioning Servers can’t be done when the vDisk store is set to read-only. All Provisioning Servers (except for one) have to be taken offline and the vDisk store has to be changed to read/write to mount the vDisks on a Provisioning Server.
- Adding vDisks to the vDisk store can’t be done when the vDisk store is set to read-only. All Provisioning Servers (except for one) have to be taken offline and the vDisk store has to be changed to read/write to add vDisks to the read-only vDisk store.
What are the requirements to use a Read-Only vDisk Store?
Using a read-only vDisk store has several requirements that need to be met before use. The requirements to use a read-only vDisk store are:
- iSCSI SAN
- Provisioning Servers running Windows 2003 Server or Windows Server 2008 with access to the SAN volume.
- Microsoft iSCSI initiator installed on all Provisioning Servers that have access to the SAN volume. On Windows Server 2003 the iSCSI Initiator needs to be downloaded from Microsoft and installed. On Windows Server 2008 it’s already installed and the service just needs to be started and set to automatic.
- vDisk master images that have already been created on a read/write vDisk store. Creating vDisk master images on a read/write vDisk store and copying them to the read-only vDisk store is a lot easier than creating them on the read-only vDisk store.
- iSCSI SAN volume with the ability to be setup for shared read/write or shared read-only access without requiring a shared/clustered file system.
Note: The iSCSI SAN volume will be made read-only with NTFS attributes using DiskPart and not on the iSCSI SAN volume.
What are the steps to implement a Read-Only vDisk Store?
- iSCSI SAN
- Create a volume on your iSCSI SAN that is large enough to hold all VHD vDisks and associated PVP files that will be placed on the read-only vDisk store.
- Set read/write – shared access for the iSCSI SAN volume.
- Provisioning Servers
- Launch the iSCSI Initiator on only one of the Provisioning Servers and logon to the SAN volume. Note: Do not logon with the iSCSI Initiator to the SAN volume from any other Provisioning Servers at this time. Only logon with one of Provisioning Server’s iSCSI Initiator until the SAN volume has been changed to read-only. Accessing the SAN volume from more than one Provisioning Server while the SAN volume is in read/write mode will corrupt the data on the volume.
- Launch Computer Management and goto Disk Management. NTFS format the SAN volume and assign a drive letter or mount point. Make sure to assign the same drive letter or mount point to the remaining Provisioning Servers. If the same driver letter or mount point is not used on the remaining Provisioning Servers then the Provisioning Server store override paths will need to be used to point Provisioning Servers with different drive letters or mount points to the SAN volume.
- Now that the SAN volume is NTFS formatted with an assigned drive letter or mount point, the SAN volume should be read/write accessible from the Provisioning Server. Before copying the VHD vDisks and associated PVP files to the SAN volume, make sure all the vDisks properties have been set correctly. This includes access mode, write cache type, load balancing, high availability, and Active Directory machine account password management. Copy the VHD vDisks and associated PVP files to the SAN volume.
- After all VHD vDisks and associated PVP files have finished copying to the SAN volume, the volume can be made read-only from the command prompt using diskpart. Run diskpart.exe from the command prompt. Type list volume to find the volume number and make note of the volume number. Type select volume volumenumber where volumenumber is the volume number identified with the list volume command. Type attributes volume set readonly to set the volume as read-only. Type detail volume to make sure the readonly attribute is set on the volume. Once the readonly attribute is confirmed, type exit to exit out of diskpart.
- Logoff the volume using the iSCSI Initiator and then logon to the volume again as a persistent target (Enable Automatically restore this connection when the computer starts to enable persistent target. This makes sure the volume stays connected after the Provisioning Server is rebooted) to re-read the volume attributes. If you don’t logoff and log back onto the volume, the volume won’t be recognized as read-only. To ensure the iSCSI SAN volume is available at the proper time when Provisioning Server is rebooted, make the Provisioning Server’s Stream Service dependent on the Microsoft iSCSI Initiator Service. Edit the registry at HKLMSystemCurrentControlSetServicesStreamService and create a Multi-String entry for MSiSCSI or iscsi.exe to make the Provisioning Server’s Stream Service dependent on the Microsoft iSCSI Initiator Service.
- You may now logon to the iSCSI SAN volume with the iSCSI Initiator and mount the volume using Disk Management on the remaining Provisioning Servers in the farm. Try to use the same drive letter or mount point on the remaining Provisioning Servers so the Provisioning Server store override paths will not need to be used. Make sure to logon the volume as a persistent target so the volume stays connected after the Provisioning Servers are rebooted and Edit the registry at HKLMSystemCurrentControlSetServicesStreamService and create a Multi-String entry for MSiSCSI or iscsi.exe to make the Provisioning Server’s Stream Service dependent on the Microsoft iSCSI Initiator Service.
- In the Provisioning Services console create a store pointing to the iSCSI SAN volume and the select the Provisioning Servers in the farm that will have access to the read-only vDisk store. If Standard Image access mode with write cache on the Provisioning Server or Differencing Disk Image access mode vDisks will be used, then you must change the default write cache path to a shared read/write location outside of the read-only vDisk store.
Now you can add existing vDisks from the read-only vDisk store in the Provisioning Services console and assign them to Target Devices.
What are the steps to modify vDisk properties on or add new vDisks to a Read-Only vDisk store?
Remember the read-only vDisk store limitations – modifying vDisk properties and adding new vDisks can’t be done when the vDisk store is set to read-only. So how do you get around this? The read-only vDisk store has to be changed to read/write. To do this, we have to take all Provisioning Servers (except for one) offline. Use the following steps to change the vDisk store to read/write and modify vDisk properties or add new vDisks:
- Shutdown all target devices using vDisks on the read-only vDisk store.
- Logoff the volume using the iSCSI Initiator on all Provisioning Servers (except for one) to take them offline.
- After all Provisioning Servers (except for one) have been taken offline, the volume can be made read/write from the command prompt using diskpart. Run diskpart.exe from the command prompt. Type list volume to find the volume number and make note of the volume number. Type select volume volumenumber where volumenumber is the volume number identified with the list volume command. Type attributes volume clear readonly to set the volume as read/write. Type detail volumeto make sure the readonly attribute is cleared on the volume. Once the readonly attribute is confirmed to have been cleared, type exit to exit out of diskpart.
- Logoff the volume using the iSCSI Initiator and then logon to the volume again to re-read the volume attributes. If you don’t logoff and log back onto the volume, the volume won’t be recognized as read/write.
- Modify vDisk properties on or add new vDisks to the SAN volume. When adding new vDisks make sure to copy the VHD vDisks and associated PVP files to the SAN volume.
- After all vDisks properties have been modified or new vDisks have finished copying to the SAN volume, the volume can be made read-only from the command prompt using diskpart. Run diskpart.exe from the command prompt. Type list volume to find the volume number and make note of the volume number. Type select volume volumenumber where volumenumber is the volume number identified with the list volume command. Type attributes volume set readonly to set the volume as read-only. Type detail volume to make sure the readonly attribute is set on the volume. Once the readonly attribute is confirmed, type exit to exit out of diskpart.
- Logoff the volume using the iSCSI Initiator and then logon to the volume again to re-read the volume attributes. If you don’t logoff and log back onto the volume, the volume won’t be recognized as read-only.
- You may now logon to the iSCSI SAN volume with the iSCSI Initiator on the remaining Provisioning Servers in the farm.
Now you can add the new vDisks from the read-only vDisk store. Add the existing vDisks and assign them to Target Devices in the Provisioning Services console.
I have only been able to test this process with iSCSI on Windows Storage Server 2008 and OpenFiler in my lab. I don’t have a Fibre Channel SAN to test with. This same process may apply to Fibre Channel SANs. If you are able to test and verify this process on a Fibre Channel SAN, please let me know.
I used the Provisioning Services 5.1 Administrator’s Guide as source for this article. The Citrix Blog has another good article on configuring a read-only vDisk store – Provisioning Services Read-only vDisk Storage.
Before using a read-only vDisk store I would look at the other storage options for Provisioning Services high availability. Each storage option has its pros and cons. For more information on the pros and cons of different vDisk store options see Citrix support article CTX119286 – Provisioning Server High Availability Considerations. Another storage option for Provisioning Services high availability is to use a local vDisk store. For more information on using and configuring a local vDisk store for high availability read one of my previous blogs Provisioning Services High Availability with a Local vDisk Store. Also make sure to check out Rich Brumpton’s great blog post on How to use PVS without having to wait for file copies at The Generation V Blog.
If you have found this article interesting or if you have any other insights, please feel free to leave comments on this article.
Excellent article
Could you also write one for the best practices to Architect a Provisioning server environment with the Network consideration and SAN requirements as well
Thanks
Hi Jarian,
I just read through your blog and I am designing an implementation for a client. I am faced with decision to using a shared SAN RO for the PVS store or assingnig a different LUN to each PVS server on the SAN and with the same drive letter on each PVS server with means any update will have to be manually copy. The client doesn’t want to deal with the admiinistrative overhead on the RO SAN management. Is this better in 5.6. Please advise
Yes it is much better in 5.6. There is now a wizard for this. Blog coming soon.