Automated Migration to VMxNet3 Network Adapters and Paravirtual SCSI Controllers for vSphere 4.x Virtual Machines
VMware vSphere 4.x was released for general availability nearly two years ago and now vSphere 5.x is rumored for release later this year. In June 2009, virtualization master Scott Lowe wrote a blog post illustrating the roughly 16 manual steps to upgrade virtual machines to VMxNet3 adapters and Paravirtual SCSI (PVSCSI) controllers. Interestingly, two years following I still encounter larger environments that are not fully taking advantage of the performance characteristics of these two virtual machine hardware types.
The reason is simple, this 16 step process can take up to 30 minutes of downtime per virtual machine if done manually, not to mention the inevitable human error. Unfortunately, what I have most frequently heard is that the time required to manually perform these virtual machine modifications enterprise-wide does not outweigh the advantages of utilizing VMxNet3 Adapters and PVSCSI Controllers. Instead, most larger environments update and implement these enhancements in their VM templates, but leave all other vSphere migrated/legacy virtual machines configured with either Flexible/E1000 NICs and/or LSI Logic Parallel SCSI/SAS Controllers.
What if we could change all that? What if this entire process could be automated, requiring less than 5 minutes of downtime for each virtual machine? What if we could cycle through all virtual machines in a given vCenter instance and automatically perform these modifications to all virtual machines? Two years after its release VMware certainly has not demonstrated an “Update Manager” migration approach for VMxNet3 and PVSCSI. Other scripts have attempted this in the past, but typically miss the bar when it comes to the Guest Operating System configuration. Most data center virtual machines have Static IP Addresses configured on the Virtual Machines and quite frankly, this can be a chore to remove and add network adapters, ensuring the correct settings are reconfigured for the new adapters.
The script provided in this post solves that problem by dynamically recording the static address and NIC configuration prior to removal, to ensure the original configuration is set on the new adapter. From a high-level, here are the processes performed by the script:
– Obtain a list of all virtual machines to process (Accepts * to process all machines within vCenter)
-Prompts for Guest Operating System credentials to use for all virtual machines
-Bypasses machines that are not online, VMware tools not up-to-date, VMware tools not running, or already have VMxNet3 & PVSCSI
-Obtain the current network configuration by invoking ipconfig /all
-Shutdown the virtual machine
-Upgrade the VM Hardware from Version 4 to Version 7
-Obtain the network name and remove the legacy (flexible/E100) adapter
-Add a VMxNet3 adapter on the same network as the legacy adapter (VM Network, etc)
-Add a temporary hard disk on a Paravirtual SCSI controller to install the PVSCSI drivers on boot
-Start the virtual machine
-Remove Hidden Devices utilizing DevCon.exe and automated RemoveHidden.bat file (Automated process for devmgr_show_nonpresent_devices)
-Use DevCon.exe to check for hardware changes
-Rename vmxnet3 network connection to Local Area Connection
-Apply original static ip address and network configuration to the new adapter
-Shutdown the virtual machine
-Remove the temporary hard disk and secondary PVSCSI controller
-Change the primary SCSI controller to Paravirtual
-Start the virtual machine
-Ensure the virtual machine boots with PVSCSI controller before moving forward
-Cycle through this process on the remaining virtual machines
Here is a screenshot of the output after running through this process from the beginning on a sample virtual machine:
Limitations:
– Only execute against virtual machines with static IP addresses configured. The current revision does not programatically determine if the NIC is set to DHCP and assumes all VMs have Static IP addresses. I intend to change this in future releases.
– Only execute against virtual machines with single NIC configurations and single SCSI controllers. The current revision does not address multi-homed systems or multiple SCSI controller configurations.
– Only execute against virtual machines with up-to-date VMware tools. Other scriptsare available to update VMware Tools using PowerCLI without a reboot. Outdated machines will be bypassed.
– Only execute locally from the virtual center server. For simplicity, in Revision 1 I intend for the script to be executed from the vCenter server locally. Therefore, the above components must be installed on the server locally.
– Test the script against test/dev VMs using the “comma-seperated virtual machines” option prior to running against “*” for all virtual machines within vCenter.
Requirements:
– PowerShell is installed on the Virtual Center server. Execution policy has been set to Unrestricted for the Local Machine scope (Run PowerShell command “Set-ExecutionPolicy Unrestricted -scope LocalMachine”)
– PowerCLI version 4.1.1 is installed on the Virtual Center server obtained here
– The included script (AutomatedMigration.VMxNet3.Paravirtual.ps1)
– DevCon.exe included in the zip file or available from http://support.microsoft.com/kb/311272. Devcon.exe for i386 must be placed in the same directory as the script.
– NVSPBind.exe included in this post or available from http://archive.msdn.microsoft.com/nvspbind. NVSPBind.exe for XP (Most stable for all operating systems and forward compatible) must be placed in the same directory as the script.
– Authentication credentials for the Guest Operating System of all virtual machines. Credentials will be requested once and used throughout the process for subsequent virtual machines.
– IMPORTANT: Outage windows have been scheduled and approved for virtual machines. Two reboots will be required in this process, so do not perform during production hours! 🙂
To execute, simply unzip all files and right click the Ps1 script and select “Run with PowerShell”. Do not execute from an existing PowerShell or PowerCLI instance. All necessary components will load automatically when you select Run with PowerShell.
Please feel free to leave a comment if you have any special requests or would like to leave feedback on results in your environment! Now that I have the framework fully laid out, I fully intend to utilize these processes to enable automated upgrades to components available in the new release of vSphere 5. Check back often for new revisions and updates on this script.
Click here to download the current version (Rev.3) along with Devcon, Devcon64 and nvspbind: AutomatedMigration.VMxNet3.Paravirtual
Updates for Revision 2:
Per Johnny Stanko’s request I have added functionality to read in the current WINS server list and assign these addresses to the new VMxNet3 adapter. Additionally, I have added functionality to remove the IPv6 component, QoS Packet Scheduler component, Link-Layer Topology Discovery Mapper component, Link-Layer Topology Discovery Responder component and enable NetBIOS over TCP/IP. To configure any of these settings, modify the Rev2 PS1 script in Notepad and change any of the variables at the top to $true before executing the script. These features required an additional command-line configuration utility NVSPBind.exe (Included in the updated Zip file). Here’s an updated screenshot of the script post-execution:
Updates for Revision 3:
Per Ryan’s request I have added functionality to check the Guest Operating System version prior to continuing. Processing will occur for the following “supported” operating systems:
– Windows Server 2003
– Windows Server 2008
– Windows Server 2008R2
– Windows XP
– Windows Vista
– Windows 7
All other operating system versions will report “Bypassing legacy or non-Windows virtual machine”.
Additionally, I have added functionality to support Devcon hidden device removal for 64-bit Operating Systems. Devcon64.exe is now included in the attached Zip file (Obtained from the Windows Driver Kit using the procedures documented here). The attached Zip file also includes an updated version of Devcon.exe for 32-bit systems (6.1.7600).
Thanks!
Hi youngtech
this looks great, any updates on the last post?
Is the rev3 version ready as most of the machines we need to address are Win 2008 R2
tx a lot
Harry,
Sorry for the delay. Rev. 3 has been posted above. Please download using the link above and send me any feedback from use in your environment! Thanks
Hi,
Haven’t tried this yet but looks really good. 2 questions before I run it:
1. How does this go against Windows 2000 servers? Windows 2000 doesn’t support Paravirtual or VMXNET3. What is its behaviour against this?
2. Devcon on Windows 2008 R2 64 bit and Windows 7 64 bit. I was attempting to do something similar however this got me stuck in my automation previously as Devcon 32bit will not work in the above OS’s. I was attempting to do some scripting to determine the architecture and this consequently selected which Devcon to use however I couldn’t find a devcon for the above OS’s.
Regards
Ryan,
I’ll repost later this week with an updated version that supports 2008R2 and Win7 64-bit. I’m not sure how I missed this, but the version that I packaged most certainly only works with 32-bit Operating Systems. According to a Technet blog post “If you don’t match the platform DevCon.exe might be partially functional. For example if you try to use x86 version of DevCon.exe with x64 OS it would correctly read and display information but return errors if you try to make some changes (e.g. update a driver).”
I’ve obtained the 64-bit version of devcon from the 600Mb Windows Deployment Kit (v 7.1.0) and validated that it removes hidden devices using the script as expected. At this point I just need to repackage it all up in a Rev. 3, which I can post later this week.
Thanks for the heads up Ryan. Good input!
youngtech
Ryan,
Sorry for the delay. Rev. 3 has been posted above. Please download using the link above and send me any feedback from use in your environment! Thanks
Hi “youngtech”,
This is Jean with Fidelity. The script worked great for a while. From sometime last week, I got timeout error for waiting VMtools running on each VM at the step “Running Devcon and remove hidden.bat…”. I saw the vmtools are on OK status. Don’t know why it behaved like this. I’ve changed the script from sleep 3 seconds to 10 seconds, rebooted vCenter server. Nothing seems to help.
Any idea?
Jean,
This may be related to running against 2008 R2. See Ryan’s feedback below. Up to this point I think all testing had been performed on 32-bit Operating Systems (mostly Server 2003 R2 and Server 2008 non-R2).
Please try the Revision that I’ll post later this week to see if that resolves the issue with the script hanging at “Running Devcon and RemoveHidden.bat”
Thanks!
This is perfect!!! Works great and I really appreciate you tweaking this out. Now if I can just get someone to click the button for me… 🙂
Thanks again Tech Guy you made my job a lot easier.
I meant YoungTech…DOH!
Does PowerShell need to be installed on all VMs or just on the local machine the script is being executed from?
Apologies if it is unclear, PowerShell and PowerCLI need to be installed on the vCenter server only. Script should be run locally on vCenter server since in the current version I’m pointing at localhost for the SDK connection. I’ll be releasing rev 2 shortly with some enhancements integrated from customer feedback. Thanks for the comment. Let me know if you have any recommendations once you migrate a couple test machines. Thanks!
I have sucessfully connected through PowerCLI locally on the vCenter server and was able execute Get-VM and other commands with no problem…however when I execute the script(with devcon in the same dir) I receive the following…any suggestions?
[vSphere PowerCLI] C:Program Files (x86)VMwareInfrastructurevSphere PowerCLI
Scripts> c:scriptsAutomatedMigration.VMxNet3.Paravirtual
The term ‘-f’ is not recognized as a cmdlet, function, operable program, or scr
ipt file. Verify the term and try again.
At line:1 char:3
+ -fi <<<
Try either launching the script directly or right clicking the Ps1 and selecting “Run with PowerShell”.
Thanks
Thank you very much. I was able to execute the script and it worked perfectly!!! Well, almost perfect…it only added only 1 DNS entry…probably something on my end. I have about 45 QA instances and 90 Prod instances to tackle and this will ease my worries of human error and not to mention all the time you have saved me.
I have one more question…is there code I can insert to add WINS addresses and also disable/uncheck QoS Packet Scheduler, TCP/IPv6, the two link layer topologies, register the connection in DNS and force NetBIOS over TCP/IP? I know…I am asking too much of you. 🙂
If not, I will look around and try to tweak on my own but I am horrible with scripts.
Again, thanks so much this is an AWESOME script!!!
Johnny, I have some ideas on how to tackle WINS, QoS, IPv6, LLTDMIO and LLTDR using NVSPBind (Will also need to be in the same directory along with Devcon). Also, I will be re-writing the DNS query function for cleaner and more consistent results. Stay tuned, I should have Rev2 with these modifications within the next several days. Thanks for the ideas!
Johnny, please download Rev. 2 and give it a go in your environment! Thanks