Citrix Provisioning Services 7.6 Issues and Lessons Learned from the Trenches: Guest Blog Post by John Meek

Share Button

johnOver the last several years, I’ve had the honor of working with a Citrix and VMware virtualization focused engineer named John Meek. John has been focused on a large customer deployment over the last several months and has run into several interesting issues deploying the latest version of Citrix Provisioning Services in this customer’s environment.

John has graciously volunteered to share his experiences with the extended virtualization community in an attempt to minimize troubleshooting efforts he has recently gone through. If you have content you would like to contribute and be a guest blogger (or regular) on itvce.com, feel free to reach out to me at dane@itvce.com or on Twitter @youngtech so we can discuss. Without further ado, below is the guest blog post by John Meek, you can find him on LinkedIn or on Twitter.

Virtual Machine Stuck at Windows Splash Screen

Recently I was working with a client and we implemented Provisioning Services 7.6 with the newest updates for ESXi 5.5u2. A major issue (headache) we ran into was that if the Master image was VMware hardware version 10, then when we rebooted to get the PVS vDisk mounted the virtual server would hang at the Windows splash screen forever. We were also using Windows 2008 R2 Service Pack 1 for our VDA PVS Targets.

This issue would also occur when we the Master image was set to boot from hard disk in Provisioning Services with a vDisk from our PVS store mounting as a secondary drive. Because of this we could not complete the Imaging Wizard and BNImage failed as well after a reboot.

Stuck

Some testing and results:

  • Provisioning Services 7.1 – At this client we were also building a PVS 7.1 Hotfix 3 environment in parallel for testing purposes in case we ran into issues with 7.6. They had 7.1 deployed in other areas and it worked well for them. Provisioning Services 7.1 did not experience this issue, we could use VMware hardware version 10 without issue.
  • Changed the VMware NIC type to E1000 – We could get the vDisk to load and get past the splash screen if we changed the NIC type to E1000 while using VM hardware 10, but other online sources online mentioned  there may be issues registering to the DDC once you changed the network adapter back to VMXNET3.
  • Changing the virtual machine version to 8 – To do this we edited the .vmx file to change the parameter virtuaHW.version=”8”. After this we removed the virtual machine from inventory and then added it back. We still had the same issue.
  • Deployed a new virtual machine using VMware version 8 and mounted vDisk – This worked using the VMXNET3 adapter.

In summary, we could get the vDisk to load if we changed the NIC type to E1000 on VM hardware 10, but with potential issues registering to the DDC and the client not wanting to use the E1000 adapters we decided to try other solutions.

The next thing we tested was creating a new virtual machine with version 8 of the VMware hardware, and using the VMXNET3 NIC. This worked perfectly and we are now able to mount the PVS vDisk and create our images. The client ultimately determined that they did not need the new hardware features of hardware version 10, so for now we are using this solution. You can verify the additional features here.

I have heard this does not happen for everyone with PVS 7.6 and VMware 5.5u2, and I certainly did not experience it in my lab. In this case we are using Cisco UCS so perhaps it is related to their converged networking, LOM cards, drivers, or some other hardware related issue, but now that we have a working solution we didn’t have time to isolate further.

This was a pretty challenging issue so I thought I would share, hopefully this saves someone a few hours of time if they run into this issue.

Logging into the Provisioning Services Console generates the error “This domain/user does not have access to the farm.”

At my current client when a user that is not the PVS service account tries to access the PVS console, they get the error “This domain/user does not have access to the farm.” If you use the PVS service account you can login successfully, though there were some intermittent times when we got the error with that account as well. We worked with Citrix support for about a week until we finally found a resolution. I did not have this issue in my lab, but we did experience it in their production environment.

Error

Here are resolutions we tested…

  • If the PVS service account is a member of Account Operators or better (domain admin) in AD, then users can log into the PVS console without the error
  • This clients infosec team will not allow this to be a permanent solution
  • Once you remove the service account from this group it breaks again
  • Allowed the Domain Users account access to the PVS security groups, and we can log in successfully
  • Set the service accounts SPN for PVSSoap services in active Directory
  • Verified there were not related authentication errors in the SQL logs and no packets were being dropped at the firewall
  • This was part of a domain migration project, so we created a PVS service account and PVS Administrators group that were new and therefore not migrated from the original domain

After working with Citrix for quite some time they determined that the actual fix was to make the PVS service account a member of the Account Operators group in Active Directory. This was unacceptable to my client’s infosec team since these rights are way too elevated for a service account. Citrix even had a support article with this as the recommended fix. They mentioned they were going to update the article after we found a better resolution, however I can no longer find the Citrix support article online. I am assuming it is in the process of being updated with the resolution below.

The new method we ended up finding to resolve this issue was to give the PVS Service account configured in the PVS Configuration Wizard read/write permissions to the Provisioning Services Admin group we had created in Active Directory. This was acceptable to my client’s infosec team and after applying this setting all users in the Provisioning Services Admin group were able to login to the console without generating this error.

ActiveDirectory

Hopefully this blog post has been beneficial to you, and keeps you from several weeks of troubleshooting. If you have any comments, questions, or just want to leave feedback, please do so in the comments section below.

Cheers,

John Meek

Share Button
  1. Richard HofmanRichard Hofman06-09-2016

    Great article, John! This really helped me today in investigating the inability to open the PVS console. Here’s what worked for us:

    The difference was that it was the PVS service account that was unable to access the PVS console. After some investigation, we found that the PVS AuthGroup only had \built-in\administrators as an authorized group. “Domain Admins” is a member of this group. So if the PVS service account was made a domain admin, the console would open. That was also unacceptable to give the PVS service account that level of rights.

    We needed to add the security group that the PVS service account was a member of to the PVS AuthGroup list. To do this, we used the MCLI.exe tool on the PVS server.

    You can run “mcli get authgroup” to output the currently authorized groups.

    To add a group, run the command with this syntax:

    MCLI Add authgroup –r authgroupname=<domain/yourOU/yourSecurityGroup

    Hope this is helpful!

    Richard

  2. John MeekJohn Meek06-12-2015

    Just a note, I tried both using the IDE controller instead of the unsupported SATA that gets listed in the VM, and using interrupt safe mode. We actually tested recreating the VM with the supported IDE controller for the CD/ROM drive before installing the PVS target client and XD 7.6 VDA client. Both had no success.

    Then we tried VM hardware level 8 on that VM which also failed, we had to create a new VM with hardware level 8 before installing these clients and then the VMXNET3 adapter worked successfully. It is still working well.

  3. G. JongeneelG. Jongeneel06-11-2015

    The first problem described has been solved for quite some time now. I described the solution that was found after extensive testing in http://discussions.citrix.com/topic/351531-pvs-71-vsphere-55-w2012r2-master-target-device-hangs/. A few months later this solution appeared in Citrix KB articles http://support.citrix.com/article/CTX200289 (vSphere 5.5) and http://support.citrix.com/article/CTX200969 (vSphere 6.0).

  4. Alan OsborneAlan Osborne06-11-2015

    This CTX forum post suggests that the PVS 7.6 boot issue (stuck at splash screen) is likely caused by an incompatibility between PVS 7.6 and the SATA controller that’s part of VM hardware version 10:

    http://discussions.citrix.com/topic/351531-pvs-71-vsphere-55-w2012r2-master-target-device-hangs/?p=1820335

    The workaround is to make the virtual CD drive an IDE device and to remove the SATA controller.

  5. Alan OsborneAlan Osborne06-11-2015

    For the first issue mentioned, this discussion is worth reading:

    http://discussions.citrix.com/topic/363293-pvs-76-xa65-tds-taking-10mins-to-boot/

Leave a Reply to Richard Hofman Click here to cancel reply.