VMware Tools 10.2.5: Changes to VMXNET3 driver settings

Last week VMware released a new version of VMware Tools.

VMware Tools 10.2.5

It might look like a minor upgrade. However, it includes important changes to the Receive Side Scaling (RSS) and Receive Throttle options in VMXNET3 driver which require attention and careful planning when implemented.

According to the vendor:

RSS is a mechanism which allows the network driver to spread incoming TCP traffic across multiple CPUs, resulting in increased multi-core efficiency and processor cache utilization. If the driver or the operating system is not capable of using RSS, or if RSS is disabled, all incoming network traffic is handled by only one CPU. In this situation, a single CPU can be the bottleneck for the network while other CPUs might remain idle.

Despite all benefits, this technology has been disabled on Windows 8 and Windows 2012 Server or later due to an issue with the vmxnet3 driver which affects Windows guest operating systems with VMware Tools 9.4.15 and later.

It was finally resolved in mid-2017 with the release of VMware Tools 10.1.7. However, only vmxnet3 driver version 1.7.3.7 in VMware Tools 10.2.0 was recommended by VMware for Windows and Microsoft Business Critical applications.

Few months after, VMware introduces the following changes to vmxnet3 driver version 1.7.3.8:

  • Receive Side Scaling is enabled by default,
  • The default value of the Receive Throttle is set to 30.

If you install VMware Tools 10.2.5 on a new virtual machine with Windows 8 and Windows 2012 Server or later, those settings will apply automatically; with the VMware Tools upgrade, they remain the same as it was before.

To check the current status of RSS and the Receive Throttle, you can execute the following PowerShell script inside the VM:

Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Get-NetAdapterAdvancedProperty | Where-Object { $_.RegistryKeyword -like “*RSS” -or $_.RegistryKeyword -like “RxThrottle” } | Format-Table -AutoSize

If you would like to edit those advanced options for all VMXNET3 NICs inside the VM, it can be done with the following two lines:

Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Set-NetAdapterAdvancedProperty -DisplayName “Receive Side Scaling” -DisplayValue “Enabled” -NoRestart
Get-NetAdapter | Where-Object { $_.InterfaceDescription -like “vmxnet3*” } | Set-NetAdapterAdvancedProperty -DisplayName “Receive Throttle” -DisplayValue “30” -NoRestart

Remember that after applying those settings, the virtual machine should be rebooted. As a result, the output will look similar to this:

VMXNET3-RSS

The only thing that is left is to perform thorough testing. Some ideas how to do it can be found in here.

25/04/2018 – Update 1: VMware released a knowledge base article about Windows 7 and 2008 virtual machines losing network connectivity on VMware Tools 10.2.0. To resolve this issue they recommend to upgrade to VMware Tools 10.2.5.

01/05/2018 – Update 2: VMware released VMware Tools 10.2.1. This minor update resolves an issue when ‘network ports are exhausted on Guest VM after a few days when using VMware Tools 10.2.0’.

A few noticeable changes to vROps 6.6 and Log Insight 4.5

With the recent releases of VMware vRealize Operations Manager 6.6 and Log Insight 4.5, VMware has made some changes to its products. They are as follows:

  • vRealize Operations Manager Plugin for vSphere Web Client should be removed after upgrading to vROps 6.6.vROps Plugin - 01
    In KB 2150394, VMware provides detailed instructions on how to do it for both vCenter for Windows and vCenter Virtual Appliance.
  • Native support for Active Directory in vRealize Log Insight is now deprecated.Log Insight - AD Integration - 01
    As per KB 2148976, VMware Identity Manager (VIDM) should be configured as an alternative. It is not confirmed yet whether VIDM is available for free for Log Insight users. More information will be posted on this thread on VMware Technology Network.I understand that this change widens the business scenarios for the product. However, for those of us who use Log Insight purely for collecting and analysing vSphere logs, it would be great to have Active Directory replaced with vCenter Single Sign-On (vCenter SSO). It sounds more logical.You can vote for the Log Insight integration with vCenter SSO on this link. It will be great if more people request this feature.

21/06/2017 – Update 1: VMware Identity Manager for Log Insight has been officially released and it is free! The VIDM virtual appliance is available on the Log Insight 4.5 download page.

27/06/2017 – Update 2: VMware has changed its policy and continues to provide support for the Active Directory integration in Log Insight 4.5. However, “it may be removed in a future version” (and probably will).

vSphere 6.x: The beauty and ugliness of the Content Library – Part 2

In part 1 of this mini-series, I wrote about some technical problems that I had had with the Content Library and provided the workable solutions for them.

Here I am going to touch the security aspect of this technology. Fortunately, there are no complications with restricting the virtual machine provisioning. It is just not as straightforward, as I or some of the readers would expect.

Issue #3 – Preventing users from provisioning the virtual machine from the Content Library

Affected platform: vSphere 6.0 and 6.5, all versions.

In vCenter, permissions are assigned to objects in the object hierarchy called vSphere Inventory Hierarchy. The individual permissions are called privileges. They are combined into roles which then allocated to the users.

The Content Library has its own set of privileges under All Privileges > Content Library. They designed to manage different settings related to the configuration of the object. There is a predefined role in vSphere called Content Library Administrator. The primary purpose of it is to give a user privileges to monitor and manage a library and its contents.

However, if you would like to restrict the VM provisioning from the Content Library and look at the long list, there is no privilege which can help to achieve this task there.

After doing some testing and discussing this subject with VMware GSS, the only solution we were able to come up was removing all Content Library privileges from the role and assigning it to the users on the vCenter Server level. In this case, users won’t be able to get access to the items in the Content Library. I was a bit frustrated with this limitation and even contacted the engineering team at VMware directly about the issue.

Coincidentally, I was working on restricting the VM provisioning from other sources: vApps and OVA/OVF Templates. It was then I realised it was actually possible to implement the complete solution to my problem.

As you might know, the Content Library keeps the VM template objects in OVF format.

CL-02

So I decided to play with the privileges that control deploying process from OVF templates. Surprisingly, it was a vApp import that helped me to achieve my goal. Happy days!

CL-03

Resolution: Remove All Privileges > vApp > Import privilege from the user role, as described in VMware KB 2105932.

Configuring static network in Photon OS

As more virtual appliances from VMware come with Photon OS, I would like to share a few simple workarounds to assign a static IP address and other network parameters to the virtual machines based on this operating system.

In Photon OS, the process systemd-networkd is responsible for the network configuration. You can check its status by executing the following command:

[ ~ ]# systemctl status systemd-networkd -l

It should give you an output similar to one in the picture below.

PhotonOS-Net-01

 

By default, systemd-networkd receives its settings from the configuration file 10-dhcp-en.network located in /etc/systemd/network/ folder. It has the following format:

[Match]
Name=e*

[Network]
DHCP=yes

I would recommend renaming this file to 10-static-en.network. So it will be easy to troubleshoot network issues in the future.

The file syntax is similar to what is used in Arch Linux. With few additional lines in the file, the network configuration can be set to our requirements. They are as follows:

  • In section [Network]
    • Address – the IP address and subnet mask in the format of XXX.XXX.XXX.XXX/YY
    • Gateway – an IP address of the default gateway
    • DNS – IP addresses of one or more DNS servers (space-separated values)
    • Domains – domain name(s) in FQDN format (space-separated values)
    • NTP – IP addresses or FQDNs of NTP sources (space-separated values).

An example of the static network configuration is shown below.

[Match]
Name=e*

[Network]
DHCP=no
Address=192.168.1.101/24
Gateway=192.168.1.1
DNS=192.168.1.21 192.168.1.1
Domains=testorg.local
NTP=0.au.pool.ntp.org 1.au.pool.ntp.org

The hostname of the system can be added to /etc/hostname file in FQDN format.

All changes should apply after rebooting the virtual machine. To test the results, we can use the following commands:

  • ip a – shows the IP addresses of the network interfaces

PhotonOS-Net-02

  • ip route – shows the routing table,

PhotonOS-Net-03

  • systemctl status systemd-timesyncd -l – shows time synchronisation status.

PhotonOS-Net-04

vCenter Support Assistant 6.5: This type of network adapter is not supported by {0}Other Linux (64-bit)

VMware has just released a new version of vCenter Support Assistant 6.5 which officially supports vSphere 6.5 and has a few noticeable improvements comparing to the previous release.

In this appliance, SUSE Linux has been replaced with Photon OS. The shift looks quite logical, as VMware pushes their own Linux flavour to more and more new products. Not only is it help to maintain a holistic approach when distributing virtual appliances, but it also promises an improved performance of the operating system, as VMware heavily invested into making it lightweight and fast.

However, when I completed provisioning vSA 6.5 in my environment and checked the virtual machine settings; to my surprise, it was a warning message shown in the screenshot below.

vsa-issue-01

It is not problematic to understand a root cause of this issue and eliminate it completely.To keep backwards compatibility with previous versions of vCenter Server, the VM hardware was set to version 8 (ESXi 5.0 and later).

To keep backwards compatibility with earlier versions of vCenter Server, the VM hardware was set to version 8 (ESXi 5.0 and later).

vsa-issue-02

This choice of the OS is entirely unexpected, as ‘Other Linux (64-bit)‘ was classified as a Legacy operating system by the vendor.

vsa-issue-03

It is until the VM hardware version 10 when it is possible to change the guest operating system to ‘Other 3.x or later Linux (64-bit)‘ to resolve the problem. So the workaround would be upgrading the VM to at least hardware version 10, and then chose the compatible OS type.

My suggestion to VMware would be to introduce a new Guest OS version called ‘Linux / Photon OS’ with the compatible hardware profile to prevent similar warnings in the future.

vSphere 6.x: Force the datastore capability sets update

When a new datastore provisioned to the vSphere environment, it might be a delay in updating the information about the capability sets, and the datastore would be incompatible with a storage policy.

storage-provider-01

The vCenter Server periodically updates storage data in its database. I couldn’t find the exact time intervals when it occurs. Fortunately, it is possible to force the datastore capability sets update in the vSphere Web Client.

To complete this task, go to the vCenter Manage tab and choose ‘Storage providers’ option. A rescan button is available from the storage system settings.

storage-provider-02

Clicking on that icon initiates rescan and updates the storage capabilities of the datastore.

storage-provider-03

Now it is able to place the virtual machines on the datastore.

Configuring PERC H730/730p cards for VMware vSAN 6.x

One of the necessary steps to create a new VMware vSAN cluster is to configure the RAID controller.

I have found Joe’s post about setting up Dell PERC H730 cards very informative and easy to follow. However, the latest generation of Dell’s PowerEdge servers has a slightly different configuration interface. So I decided to document configuration process using the BIOS graphical interface.

You can get into it either pressing an F2 key during the server boot or choosing a BIOS Setup option in the Next Boot drop-down menu in the iDRAC Virtual Console.

step-00

The next step is to click on the Device Settings and select the RAID controller from the list of available devices.

step-01

Step-02.png

There are two configuration pages that we should be interested in, as follows:

  • Controller Management > Advanced Controller Management
  • Controller Management > Advanced Controller Properties.

The former gives us ability to switch from RAID mode to HBA mode.

Step-03.png

The latter allows disabling the controller caching and setting the BIOS Boot mode.

Step-04.png

Please note the system reboot is required for the change to take effect. It is always a good idea to double check that the parameters above were setup correctly.