VMware ESXi 6.0-6.5: Low network receive throughput for VMXNET3 on Windows VM

VMware has just released a new KB 57358 named ‘Low receive throughput when receive checksum offload is disabled and Receive Side Coalescing is enabled on Windows VM‘. This requires attention when configuring the VMXNET3 adapter on Windows operating systems (OS). However, it only affects virtual environments with VMware ESXi 6.0 and ESXi 6.5 only.

VMware states that it happens when the following conditions are met:

  • Guest OS is Windows 2012 / Windows 8 or later
  • VM hardware version 11 or later
  • Virtual network adapter is VMXNET3
  • Receive Side Coalescing (RSC) is enabled on the VMXNET3 driver on the guest OS
  • Some or all of following receive checksum offloads have value Disabled or only Tx Enabled on the VMXNET3 driver on the guest operating system:
    • IPv4 Checksum Offload
    • TCP Checksum Offload (IPv4)
    • TCP Checksum Offload (IPv6)
    • UDP Checksum Offload (IPv4)
    • UDP Checksum Offload (IPv6).

This shouldn’t be a problem if the VMXNET3 driver has the default settings.

VMXNET3-RSC-Issue-00

For example, copying 3.4 gigabytes of data to the test VM via the 1Gbps link took me seconds.

VMXNET3-RSC-Issue-01

With TCP Checksum Offload (IPv4) set to Tx Enabled on the VMXNET3 driver the same data takes ages to transfer.

VMXNET3-RSC-Issue-02

VMware provides a workaround for this issue: you either need to disable RSC, if any of receive checksum offloads is disabled, or manually enable receive checksum offloads. The knowledge base includes the PowerShell commands that help to automate the latter.

vSphere 6.5: How to improve performance of the Content Library data transfers

I came across this subject by chance. However, it worth considering the KB 2112692 when planning for the content library design in vSphere 6.5.

For all operations related to content libraries that involve data transfer, which cannot be conducted by the ESXi hosts, VMware suggests the following:

  • Switch the traffic to run through non-secure HTTP connection. This is to improve the transfer speed for synchronisation tasks.

This can be done by enforcing HTTP connection for Library Sync operations for all the libraries (global approach) or on particular libraries (individual approach) in your vCenter Server instance.

At the time of writing this, there is no support for editing the Content Library Service settings via vSphere Client (HTML5). All changes can be implemented using the vSphere Web Client though.

To set this policy globally, you need to change ‘Force HTTP for Library Sync’ option in Administration > Deployment > System Configuration > Services > Content Library Service to true. This applies immediately and restart is not required.

vSphere65-CL-Performance-01

With the individual approach, you need to edit the URL of the subscribed library to start with http, instead of https. Again, no need to restart the Content Library Service.

  • Bypass the rhttpproxy. This is to improve the transfer speed for synchronisation tasks, importing and exporting items.

This change applies globally. It requires editing the vCenter Server configuration file vdc.properties, changing the firewall settings to allow connections to TCP port 16666, and restarting the Content Library Service.