vSphere 6.x: The datastore file browser converts VMDKs to thick-provisioned when copy/move data to the target VMFS datastore

It came as a surprise to me that the datastore file browser in vSphere 6.5 (and all other versions) would convert VMDK disks to thick-provisioned when you copy or move them across from one VMFS datastore to another. This is a default behaviour, even if the initial VMDK file was thin-provisioned and the target datastore supports thin provisioning.

This can potentially cause an outage to the virtual machines resided on the Virtual Machine File System. Unfortunately, when you initiate a copy/move operation in the datastore file browser, the system doesn’t warn you about this change. So you need to remember about it and calculate the required disk space ahead of transferring data.

What is more interesting, I haven’t been able to find any reference to this in the official documentation to vSphere. It actually states quite opposite:

“Virtual disk files are moved or copied without format conversion.”

To illustrate the observed behaviour, I had created a virtual machine named TEST-VM with one thin-provisioned disk of 10 GB.

VMFS-Thick-Issue-01

After the VM was powered on, it reported the following data usage in the datastore file browser:

VMFS-Thick-Issue-02

The Inflate button on the image above indicates that “you can convert the thin disk to a virtual disk in thick provision format.”

The PowerCLI commands below helped me to show the used and provisioned space for TEST-VM:

Get-VM -Name “TEST-VM” | Select Name,@{N=’Used Space (GB)’;E={[math]::Round($_.UsedSpaceGB,2)}},@{N=’Prov. Space (GB)’;E={[math]::Round($_.ProvisionedSpaceGB,2)}} | Format-List

The result was as expected:

VMFS-Thick-Issue-03

I powered off the VM and copied the TEST-VM folder via the datastore file browser to another VMFS datastore.

VMFS-Thick-Issue-04

After this task completes, the resulted VMDK looked different:

VMFS-Thick-Issue-05

Please note the Inflate button is greyed out now. This means the virtual disk is thick provisioned.

I have looked on the Internet for any information and found a few community threads from 2009 discussing this issue – here and here. So the problem exists for a while. According to those threads, during the copy/move operation initiated via the datastore file browser, the underlying vmkfstools utility executes with the default settings creating a thick provisioned disk.

The only workaround is to use the following command to convert the VMDK to thin again:

vmkfstools -i <source.vmdk> <destination.vmdk> -d thin

If the intent is to replace the source thick-provisioned VMDK with a new thin-provisioned one, make sure to use vmkfstools utility for that operation. It will change names for both *.vmdk and *flat.vmdk files, as well as the extent description value in *.vmdk.

[IMPORTANT] VMware ESXi 6.x: Denial-of-service vulnerability in 3D-acceleration feature

This week VMware published a security advisory VMSA-2018-0025 about the denial-of-service vulnerability in the 3D-acceleration feature in VMware ESXi, Workstation, and Fusion.

VM3DSupport-Issue-01

It affects all versions of those products if 3D-acceleration feature is enabled for virtual machines (VMs). This is a default setting for all VMs on VMware Workstation and Fusion and might be an issue for the VMs managed by VMware Horizon.

More information about this issue can be found here.

At the moment of writing this article, there were no patches or updates provided by VMware to mitigate this problem. So a workaround would be to disable the 3D-acceleration feature for affected systems.

To identify the VMs that have the 3D-acceleration feature enabled, I wrote the following PowerCLI script:

As soon as the permanent solution provided by the vendor, I will update this blog post with more information.

VMware vForum 2018 is coming to Sydney!

For those IT professionals who are based in Sydney or are planning to travel to this stunning city shortly, it is good to know that VMware vForum 2018 will be held here at Luna Park, Milsons Point on the 17th and 18th of October, 2018.

vForum2018

Even if the event takes two days, this time it is completely free for all participants. The main focus will be on Data Centre, Cloud, Workforce Transformation, and Security. A detailed agenda is available here.

For the technology geeks it is good to know the availability of Hands-On Labs, including the expert-led workshops, as well as generous discounts for the On Demand courses, VMware Learning Zone Premium subscription, and the VCP/VCAP certification exams.

Also, vBrownBag crew will host TechTalks at the event – a unique experience that shouldn’t be missed.

Considering the number of announcements at VMworld 2018 US, vForum 2018 is an essential place to be next week!

vSphere 6.5: Additional considerations when migrating to VMFS-6 – Part 1

For those who use the Virtual Machine File System (VMFS) datastores, one of the steps when upgrading to vSphere 6.5 is to migrate them to VMFS-6.

VMFS6-01

VMware provides a detailed overview of VMFS-6 on the StorageHub, as well as an example of how the migration from VMFS-5 can be automated using PowerCLI.

However, there are three edge cases that require extra steps to continue with the migration. They are as follows:

All those objects, if they exist, prevent the ESXi host from unmounting the datastore, and they need to be moved to a new location before migration continues. The required steps to relocate them will be reviewed in the paragraphs below.

Relocating the system swap

The system swap location can be checked and set via vSphere Client in Configure > System > System Swap settings of the ESXi host.

VMFS6-02

Alternatively, the system swap settings can be retrieved via PowerCLI:

The script above can be modified to create the system swap files on a new datastore:

Note: The host reboot is not required to apply this change.

Moving the persistent scratch location

A persistent scratch location helps when investigating the host failures. It preserves the host log files on a shared datastore. So they can be reachable for troubleshooting, even if the host experienced the Purple Screen of Death (PSOD) or went down.

To identify the persistent scratch location, filter the key column by the ‘scratch’ word in Settings > System > Advanced System Settings of the ESXi host in vSphere Client.

VMFS6-03

You only need to point the ScratchConfig.ConfiguredScratchLocation setting to a new location and reboot the host for this change to take effect.

Note: Before doing any changes, make sure that the .locker folder (should be unique for each configured host to avoid data mixing or overwrites) has been created on the desired datastore. Otherwise, the persistent scratch location remains the same.

To review and modify advanced host parameters including the persistent scratch location via PowerCLI, look for two cmdlets named Get-AdvancedSetting and Set-AdvancedSetting. This procedure is well-documented in KB 1033696.

An information about how to automate the diagnostic coredump file relocation will be covered in Part 2 or this series later this month. Keep you posted!

URGENT: VMDKs residing on vSAN 6.6 and later that have been extended may encounter data inconsistencies [RESOLVED]

Last week VMware published a KB 58715 reporting virtual machine disks residing on vSAN 6.6 and later that have been extended may encounter data inconsistencies. For those who subscribed to VMware email communications, the following message has been sent recently.

vSAN66-Issue-01

As stated in the article, this issue might happen in a rare occurrence. Still, VMware encourages their clients to check the value of the advanced setting VSAN.ClomEnableInplaceExpansion on all ESXi hosts that are part of the vSAN cluster. If it is set to the default value of “1”, the vendor recommends changing it to “0” immediately. This can be done using the following PowerCLI command:

Foreach ($VMHost in (Get-Cluster -Name (Read-Host “Cluster Name”) | Get-VMHost)) {Get-AdvancedSetting -Entity $VMHost -Name VSAN.ClomEnableInplaceExpansion | Where-Object {$_.Value -ne ‘0’} | Set-AdvancedSetting -Value ‘0’ -Confirm:$false}

Fortunately, no reboot or service restart is required for this change to take effect, and it will become effective within 60 seconds.

It is good to see how much effort the vendor put into supporting vSAN and proactively inform users about any problems. Great service, VMware!

04/10/2018 – Update 1: VMware has realeased patches for both vSAN 6.6 and 6.7 that remediate this issue. Please read the resolution section in KB 58715 for more information.

URGENT: VMware Tools 10.3.0 was recalled

VMware has just announced that VMware Tools 10.3.0 was recalled due to a functional issue with 10.3.0 in ESXi 6.5.

VMware-Tools-1030-Issue

As per KB 57796, the VMXNET3 driver released with VMware Tools 10.3.0 can result in a Purple Diagnostic Screen (PSOD) or guest network connectivity loss in certain configurations. Those configurations include:

  • VMware ESXi 6.5 hosts
  • VM Hardware version 13
  • Windows 8/Windows Server 2012 or higher guest operating system (OS).

As a workaround, VMware recommends uninstalling VMware Tools 10.3.0 and then reinstalling VMware Tools 10.2.5 for affected systems.

The vendor will be releasing a revised version of the VMware Tools 10.3 family at some point in the future.

More information is available in VMSA-2018-0017.

25/09/2018 – Update 1: VMware Tools were updated to version 10.3.2 to resolve the issue with VMXNET3 driver. VMware recommends to install VMware Tools 10.3.2, or VMware Tools 10.2.5 or an earlier version of VMware Tools.

vSphere 6.5: Switching to Native Drivers in ESXi 6.5

The Native Device Driver architecture is not something new. Since its introduction more than five years ago, VMware encourages their hardware ecosystem partners to work on developing native drivers. A list of supported hardware is growing with every major release of ESXi, with the company’s aim to deprecate the vmkLinux APIs and associated driver ecosystem completely in the future releases of vSphere.

The benefits of using the native drivers are as follows:

  • It removes the complexity of developing and maintaining Linux derived drivers,
  • It improves the system performance,
  • It frees from the functional limitations of Linux derived drivers,
  • It increases the stability and reliability of the hypervisor, as native drivers are designed specifically for VMware ESXi.

Saying that one of the steps when upgrading to a new version of vSphere is to check that the hardware supports native drivers. By default, if ESXi identifies a native driver for a device it will be loaded instead of Linux derived driver. However, it is not always a case, and you need to check whether native drivers are in use after the system upgrade.

Following steps in KB 1031534 and KB 1034674, you can pinpoint PCI devices and corresponding drivers loaded for each of them:

  • To identify a storage HBA (such as a fibre card or RAID controller), run this command:

# esxcfg-scsidevs -a

  • To identify a network card, run this command:

# esxcfg-nics -l

  • To list device state and note the hardware IDs, run this command:

# vmkchdev -l

The /etc/vmware/default.map.d/ folder on ESXi host contains a full list of map files referring to the native drivers available for your system.

ESXi-Native-Driver-01

To quickly identify the driver version, you can run this command:

# esxcli software vib list | grep <native_driver_name>

In addition, information about available vSphere Installation Bundles (VIBs) in vSphere 6.5 can be found via the web client or PowerCLI session:

  • To view all installed VIBs in vSphere Client (HTML5), open Configure > System > Packages tab in the host settings:

ESXi-Native-Driver-02

  • To view all installed VIBs in VMware Host Client, open Manage > Packages tab in the host settings:

ESXi-Native-Driver-03

  • To list all installed VIBs in PowerCLI, run this command:

# (Get-VMHost -Name ‘<host_name>‘ | Get-EsxCli).software.vib.list() | select Name,Vendor,Version | sort Name

Comparing findings above with information in the IO Devices section in VMware Hardware Compatibility List, you would be able to find out whether native drivers available for your devices, as well as the recommended combination of the driver and firmware, tested and supported by VMware.

It worth reading the release notes for the corresponding drivers and search any reference to it on VMware and the third-party vendors’ websites, in case there are any known issues or limitations that might affect how device function.

If everything seems good, it is time to enable the native driver following steps in KB 2147565:

# esxcli system module set –enabled=true –module=<native_driver_name>

This change requires a host reboot and a thorough testing afterwards. The following commands can be quite helpful when troubleshooting native drivers:

  • To get the driver supported module parameters, run this command:

# esxcfg-module -i <native_driver_name>

  • To get the driver info, run this command:

# esxcli network nic get -n <vmnic_name>

  • To get an uplink stats, run this command:

# esxcli network nic stats -n <vmnic_name>

31/08/2018 – Update 1: After some feedback provided, I have decided to list well-known issues with the native drivers that exist currently. They are as follows:

  • The Mellanox ConnectX-4/ConnectX-5 native ESXi driver might exhibit performance degradation when its Default Queue Receive Side Scaling (DRSS) feature is turned on (Reference: vSphere 6.7 Release Notes),
  • Native software FCoE adapters configured on an ESXi host might disappear when the host is rebooted (Reference: vSphere 6.7 Release Notes),
  • HP host with QFLE3 Driver Version 1.0.60.0 experienced a PSOD or stuck at “Shutting down device drivers…” shutdown or restart (Reference: KB 55088),
  • ESXi 6.5 Storage Performance Issues and Fix (Reference: Anthony Spiteri’s blog).