URGENT: VMware Tools 10.3.0 was recalled

VMware has just announced that VMware Tools 10.3.0 was recalled due to a functional issue with 10.3.0 in ESXi 6.5.

VMware-Tools-1030-Issue

As per KB 57796, the VMXNET3 driver released with VMware Tools 10.3.0 can result in a Purple Diagnostic Screen (PSOD) or guest network connectivity loss in certain configurations. Those configurations include:

  • VMware ESXi 6.5 hosts
  • VM Hardware version 13
  • Windows 8/Windows Server 2012 or higher guest operating system (OS).

As a workaround, VMware recommends uninstalling VMware Tools 10.3.0 and then reinstalling VMware Tools 10.2.5 for affected systems.

The vendor will be releasing a revised version of the VMware Tools 10.3 family at some point in the future.

More information is available in VMSA-2018-0017.

25/09/2018 – Update 1: VMware Tools were updated to version 10.3.2 to resolve the issue with VMXNET3 driver. VMware recommends to install VMware Tools 10.3.2, or VMware Tools 10.2.5 or an earlier version of VMware Tools.

VMware Tools 10.3.0: Additional considerations when updating Windows machines

Last week VMware released VMware Tools version 10.3.0.

VMware-Tools-1030-00

Not only does it include new features and a security update to address an out-of-bounds read vulnerability, it also introduces a significant change in a way VMware Tools install to Windows operating systems. In some circumstances, VMware Tools 10.3.0 installation or upgrade can even fail, as documented in VMware KB 55798.

Please bear in mind that the installer size for Windows has almost doubled with this release (for example, the 64-bit version has 72.6 MB in size compared to 47.4 MB for version 10.2.5), as it includes the following additional software:

  • Microsoft Visual C++ 2017 Redistributable packages (both 32- and 64-bit versions),

VMware-Tools-1030-02

  • VMware AppDefence (disabled by default).

VMware-Tools-1030-03

As a result, VMware Tools require to reboot the operating system at least once before actually proceed with the software install or upgrade.

VMware-Tools-1030-01

To reduce the maintenance window, the vendor’s recommendations are as follows:

  • Upgrade Windows with latest service pack available from Microsoft and install the Microsoft Visual C++ 2017 Redistributable manually before installing or upgrading to VMware Tools 10.3.x.

Note: If installing Microsoft Visual C++ 2017 Redistributable is not possible, consider installing the Windows Update KB2999226 manually to reduce the need for system restart in versions earlier to Windows 10.

  • When VMware Tools installation or upgrade is invoked with REBOOT=ReallySuppress argument and a system restart is required for completing Microsoft Visual C++ 2017 Redistributable install, re-attempt the VMware Tools installation or upgrade after restarting the Windows system. vSphere client can detect this situation by noticing no change in VMware Tools version and guestinfo.toolsInstallErrCode=3010 in the guest variables or in the advanced configuration of the virtual machine.

Note: When VMware Tools installation or upgrade is invoked without any arguments, a system restart may occur automatically to complete Microsoft Visual C++ 2017 Redistributable install. After Windows system restarts, re-attempt the VMware Tools installation or upgrade.

Also, VMware has enabled the receive data ring support for the VMXNET3 driver in Windows with this update. Thorough testing is required to understand how this change influences the virtual NIC performance.

07/09/2018 – Update 1: VMware has recalled VMware Tools 10.3.0 due to a functional issue with 10.3.0 in ESXi 6.5. More information is available here.

VMware Tools 10.2.5: Changes to VMXNET3 driver settings

Last week VMware released a new version of VMware Tools.

VMware Tools 10.2.5

It might look like a minor upgrade. However, it includes important changes to the Receive Side Scaling (RSS) and Receive Throttle options in VMXNET3 driver which require attention and careful planning when implemented.

According to the vendor:

RSS is a mechanism which allows the network driver to spread incoming TCP traffic across multiple CPUs, resulting in increased multi-core efficiency and processor cache utilization. If the driver or the operating system is not capable of using RSS, or if RSS is disabled, all incoming network traffic is handled by only one CPU. In this situation, a single CPU can be the bottleneck for the network while other CPUs might remain idle.

Despite all benefits, this technology has been disabled on Windows 8 and Windows 2012 Server or later due to an issue with the vmxnet3 driver which affects Windows guest operating systems with VMware Tools 9.4.15 and later.

It was finally resolved in mid-2017 with the release of VMware Tools 10.1.7. However, only vmxnet3 driver version 1.7.3.7 in VMware Tools 10.2.0 was recommended by VMware for Windows and Microsoft Business Critical applications.

Few months after, VMware introduces the following changes to vmxnet3 driver version 1.7.3.8:

  • Receive Side Scaling is enabled by default,
  • The default value of the Receive Throttle is set to 30.

If you install VMware Tools 10.2.5 on a new virtual machine with Windows 8 and Windows 2012 Server or later, those settings will apply automatically; with the VMware Tools upgrade, they remain the same as it was before.

To check the current status of RSS and the Receive Throttle, you can execute the following PowerShell script inside the VM:

Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Get-NetAdapterAdvancedProperty | Where-Object { $_.RegistryKeyword -like “*RSS” -or $_.RegistryKeyword -like “RxThrottle” } | Format-Table -AutoSize

If you would like to edit those advanced options for all VMXNET3 NICs inside the VM, it can be done with the following two lines:

Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Set-NetAdapterAdvancedProperty -DisplayName “Receive Side Scaling” -DisplayValue “Enabled” -NoRestart
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Set-NetAdapterAdvancedProperty -DisplayName “Receive Throttle” -DisplayValue “30” -NoRestart

Remember that after applying those settings, the virtual machine should be rebooted. As a result, the output will look similar to this:

VMXNET3-RSS

The only thing that is left is to perform thorough testing. Some ideas how to do it can be found in here.

25/04/2018 – Update 1: VMware released a knowledge base article about Windows 7 and 2008 virtual machines losing network connectivity on VMware Tools 10.2.0. To resolve this issue they recommend to upgrade to VMware Tools 10.2.5.

01/05/2018 – Update 2: VMware released VMware Tools 10.2.1. This minor update resolves an issue when ‘network ports are exhausted on Guest VM after a few days when using VMware Tools 10.2.0’.