VMware ESXi 6.0-6.5: Low network receive throughput for VMXNET3 on Windows VM

VMware has just released a new KB 57358 named ‘Low receive throughput when receive checksum offload is disabled and Receive Side Coalescing is enabled on Windows VM‘. This requires attention when configuring the VMXNET3 adapter on Windows operating systems (OS). However, it only affects virtual environments with VMware ESXi 6.0 and ESXi 6.5 only.

VMware states that it happens when the following conditions are met:

  • Guest OS is Windows 2012 / Windows 8 or later
  • VM hardware version 11 or later
  • Virtual network adapter is VMXNET3
  • Receive Side Coalescing (RSC) is enabled on the VMXNET3 driver on the guest OS
  • Some or all of following receive checksum offloads have value Disabled or only Tx Enabled on the VMXNET3 driver on the guest operating system:
    • IPv4 Checksum Offload
    • TCP Checksum Offload (IPv4)
    • TCP Checksum Offload (IPv6)
    • UDP Checksum Offload (IPv4)
    • UDP Checksum Offload (IPv6).

This shouldn’t be a problem if the VMXNET3 driver has the default settings.

VMXNET3-RSC-Issue-00

For example, copying 3.4 gigabytes of data to the test VM via the 1Gbps link took me seconds.

VMXNET3-RSC-Issue-01

With TCP Checksum Offload (IPv4) set to Tx Enabled on the VMXNET3 driver the same data takes ages to transfer.

VMXNET3-RSC-Issue-02

VMware provides a workaround for this issue: you either need to disable RSC, if any of receive checksum offloads is disabled, or manually enable receive checksum offloads. The knowledge base includes the PowerShell commands that help to automate the latter.

vSphere 6.5: How to improve performance of the Content Library data transfers

I came across this subject by chance. However, it worth considering the KB 2112692 when planning for the content library design in vSphere 6.5.

For all operations related to content libraries that involve data transfer, which cannot be conducted by the ESXi hosts, VMware suggests the following:

  • Switch the traffic to run through non-secure HTTP connection. This is to improve the transfer speed for synchronisation tasks.

This can be done by enforcing HTTP connection for Library Sync operations for all the libraries (global approach) or on particular libraries (individual approach) in your vCenter Server instance.

At the time of writing this, there is no support for editing the Content Library Service settings via vSphere Client (HTML5). All changes can be implemented using the vSphere Web Client though.

To set this policy globally, you need to change ‘Force HTTP for Library Sync’ option in Administration > Deployment > System Configuration > Services > Content Library Service to true. This applies immediately and restart is not required.

vSphere65-CL-Performance-01

With the individual approach, you need to edit the URL of the subscribed library to start with http, instead of https. Again, no need to restart the Content Library Service.

  • Bypass the rhttpproxy. This is to improve the transfer speed for synchronisation tasks, importing and exporting items.

This change applies globally. It requires editing the vCenter Server configuration file vdc.properties, changing the firewall settings to allow connections to TCP port 16666, and restarting the Content Library Service.

vSphere 6.5: Failed to deploy OVF package from the Content Library

Working on automating the virtual machine (VM) provisioning from an OVF template in a content library in vSphere 6.5 Update 2, I ran across an interesting behaviour. Some virtual machines were created without any issues, whereas others have been failing with the error message ‘Failed to deploy OVF package.’

OVF-Template-Issue-00

OVF-Template-Issue-01

In the vpxd.log on vCenter Server, this error message looks as follows:

info vpxd[7F83D22C5700] [Originator@6876 sub=Default opID=SelectResourcePageMediator-validate-335519-ngc:70031913-ae-71-7e-92-01] [VpxLRO] — ERROR task-51355 — TEST-VM-01 — ResourcePool.ImportVAppLRO: vim.fault.OvfImportFailed:
–> Result:
–> (vim.fault.OvfImportFailed) {
–> faultCause = (vim.fault.NotFound) {
–> faultCause = (vmodl.MethodFault) null,
–> faultMessage = (vmodl.LocalizableMessage) [
–> (vmodl.LocalizableMessage) {
–> key = “com.vmware.ovfs.ovfs-main.ovfs.object_not_found”,
–> arg = (vmodl.KeyAnyValue) [
–> (vmodl.KeyAnyValue) {
–> key = “0”,
–> value = “/TEST-VM-01/nvram”
–> }
–> ],
–> message = “The specified object /TEST-VM-01/nvram could not be found.”
–> }
–> ]
–> msg = “The specified object /TEST-VM-01/nvram could not be found.”
–> },
–> faultMessage = <unset>
–> msg = “”
–> }
–> Args:
–>

The NVRAM file contains information such as BIOS settings. As VMware states here, a new NVRAM file is created ‘when a virtual machine is migrated, migrated with vMotion, cloned or deployed from a template.’

Looking through the knowledge base articles on the vendor’s website, I have found the following hint in KB 2108718:

Error 2: Could not find the file

This issue occurs if the NVRAM file mentioned in the virtual machines configuration file(*.vmx) is not available anymore.

As you might know, templates are stored in OVF format in the content library. Two files compose a template: an OVF descriptor (.ovf) and a virtual disk file (.vmdk). During the VM provisioning phase, the settings listed in the OVF descriptor form the virtual machine’s configuration file (.vmx). So my assumption was that one of those settings caused a conflict that prevented vSphere from creating those VMs.

When we clone a template to the content library, only minimum required settings are stored in the OVF descriptor. However, there is an option available to include extra configuration.

OVF-Template-Issue-02

I decided to compare a vanilla OVF descriptor with the one with extra settings. While doing this comparison, one particular line in the file with extra configuration caught my eye:

<vmw:ExtraConfig ovf:required=”false” vmw:key=”firmware” vmw:value=”efi”/>

For some reason, the provisioning task is not able to succeed in creating new virtual machines with the EFI firmware from the template in vSphere 6.5. This issue happens regardless of the guest operating system version and content library type (local or subscribed).

Surprisingly, I do not see any problem with deploying VMs with the EFI firmware from the content library in vSphere 6.0. I need some time to test this functionality in vSphere 6.7, before opening a support request with VMware.

Workaround: Feel free to use a PowerCLI script below to set the firmware type to EFI after provisioning a new VM.

 

VMware Tools 10.3.0: Additional considerations when updating Windows machines

Last week VMware released VMware Tools version 10.3.0.

VMware-Tools-1030-00

Not only does it include new features and a security update to address an out-of-bounds read vulnerability, it also introduces a significant change in a way VMware Tools install to Windows operating systems. In some circumstances, VMware Tools 10.3.0 installation or upgrade can even fail, as documented in VMware KB 55798.

Please bear in mind that the installer size for Windows has almost doubled with this release (for example, the 64-bit version has 72.6 MB in size compared to 47.4 MB for version 10.2.5), as it includes the following additional software:

  • Microsoft Visual C++ 2017 Redistributable packages (both 32- and 64-bit versions),

VMware-Tools-1030-02

  • VMware AppDefence (disabled by default).

VMware-Tools-1030-03

As a result, VMware Tools require to reboot the operating system at least once before actually proceed with the software install or upgrade.

VMware-Tools-1030-01

To reduce the maintenance window, the vendor’s recommendations are as follows:

  • Upgrade Windows with latest service pack available from Microsoft and install the Microsoft Visual C++ 2017 Redistributable manually before installing or upgrading to VMware Tools 10.3.x.

Note: If installing Microsoft Visual C++ 2017 Redistributable is not possible, consider installing the Windows Update KB2999226 manually to reduce the need for system restart in versions earlier to Windows 10.

  • When VMware Tools installation or upgrade is invoked with REBOOT=ReallySuppress argument and a system restart is required for completing Microsoft Visual C++ 2017 Redistributable install, re-attempt the VMware Tools installation or upgrade after restarting the Windows system. vSphere client can detect this situation by noticing no change in VMware Tools version and guestinfo.toolsInstallErrCode=3010 in the guest variables or in the advanced configuration of the virtual machine.

Note: When VMware Tools installation or upgrade is invoked without any arguments, a system restart may occur automatically to complete Microsoft Visual C++ 2017 Redistributable install. After Windows system restarts, re-attempt the VMware Tools installation or upgrade.

Also, VMware has enabled the receive data ring support for the VMXNET3 driver in Windows with this update. Thorough testing is required to understand how this change influences the virtual NIC performance.

07/09/2018 – Update 1: VMware has recalled VMware Tools 10.3.0 due to a functional issue with 10.3.0 in ESXi 6.5. More information is available here.

vCenter Converter 6.2: Error 1053: The service did not respond to the start or control request in a timely fashion

I couldn’t believe something as simple as vCenter Converter Agent installation would cause me a headache. The service was properly registered on the remote machine. However, it was failing to start with the error message as follows:

Converter-Issue-01

For a moment I thought about any dependencies that VMware vCenter Converter Standalone Agent service could have, but it was a false assumption.

Converter-Issue-02

Looking for any hint from the community, I thought it might worth time to check the release notes for VMware vCenter Converter Standalone. To my great surprise, this issue has been documented as the known issue:

Converter-Issue-03

Who can imagine that the Internet access can be a requirement for this tool?

Gladly, it can be fixed with a workaround provided by VMware:

  1. Open HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control in the Windows Registry Editor and create or modify a ServicesPipeTimeout DWORD32 decimal value to at least 300000 (5 minutes).
  2. Restart the system for the changes to take effect.

16/08/2018 – Update 1: If any issues with VMware vCenter Converter Standalone, it makes sence to look into VMware KB 1016330 ‘Troubleshooting checklist for VMware Converter’ for possible solutions.

vSphere 6.0: The operation failed due to The operation is not allowed in the current state.

Working on the PowerCLI script to automate VM provisioning in vSphere 6.0, I ran across the following error message:

New-VM-Issue-02

The script was simply creating a new virtual machine from the Content Library template.

New-VM -Name $vmName -VMHost $vmHost -Location $vmLocation -Datastore $vmDatastore -ContentLibraryItem $vmTemplate

Investigating this behaviour further, I have found that it was related to a failing cross vMotion placement. In vSphere Client those errors were looking like this:

New-VM-Issue-01

It was then I concluded it might be related to DRS being disabled for that cluster. Setting DRS to the Manual mode resolved this issue.

The next step was to check whether this problem could be reproduced for other templates. Surprisingly, the VM provisioning from some of them went smoothly, regardless of the DRS status.

Remembering about my previous articles (here and here) related to the Content Library, I decided to have a look into the VM configuration files (*.ovf) for both templates to compare. The only difference was related to the following storage settings which exist for the faulty templates:

<vmw:StorageGroupSection ovf:required=”false” vmw:id=”group1″ vmw:name=”EMC VNX VM Storage Policy”>
<Info>Storage policy for group of disks</Info>
<vmw:Description>The EMC VNX VM Storage Policy storage policy group</vmw:Description>
</vmw:StorageGroupSection>

<vmw:StorageSection ovf:required=”false” vmw:group=”group1″>
<Info>Storage policy group reference</Info>
</vmw:StorageSection>

As expected, removing those block of data from the OVF-file and re-uploading it to the Content Library resolved provisioning issue completely.

A couple of notes:

  • There is no way to identify any problems with the VM configuration using the Content Library configuration page or ‘New VM from Library…’ dialogue in vCenter Web Client.
  • Even if a modified OVF-file has been uploaded to the local Content Library, this change is not going to propagate to the subscribed ones. You need to repeat this process for all subscribed Content Libraries manually.