vSphere 6.5: Additional considerations when migrating to VMFS-6 – Part 2

In the Part 1 of this series, I was writing about the most common cases which might prevent a successful migration to VMFS-6. There is another one to cover.

For ESXi hosts that boot from a flash storage or from memory, a diagnostic core dump file can also be placed on a shared datastore. You won’t be able to un-mount this datastore without deleting a core dump first.

VMware recommends using an esxcli utility to view/edit the core dump settings. This also can be automated via PowerCLI.

To check if the core dump file exists and is active, please use the following code:

# File name: Get-VMHostCoredumpLocation.ps1
# Description: This script checks the core dump location for ESXi hosts.
#
# 11/10/2018 - Version 1.0
# - Initial Release
#
# Author: Roman Dronov (c)
# Get information about the core dump location for ESXi hosts
Write-Host "`nCore Dump Settings:`r" -ForegroundColor Green
ForEach ($vmhost in $(Get-VMHost | sort Name)) {
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$esxcli2.system.coredump.file.get.Invoke() | Select @{N='Host Name';E={$vmhost}},@{N='Active Core Dump File';E={$_.Active}} #| Format-List
}
Clear-Variable vmhost,esxcli2 -Scope Global

To delete an old configuration that points to the VMFS-5 datastore, the following script can help:

# File name: Remove-VMHostCoredumpLocation.ps1
# Description: This script removes a core dump file for the particular ESXi host.
#
# 11/03/2019 - Version 1.0
# - Initial release
#
# Author: Roman Dronov (c)
# Define common functions
function ex {exit}
# Get the host name and check it is valid
$vmhosts = Get-VMHost | ? {$_.ConnectionState -eq "Connected" -or $_.ConnectionState -eq "Maintenance"} | ForEach-Object {$_.Name.Split('.')[0]}
$vmhost = (Read-Host -Prompt "`n Please type in the ESXi host name").Split('.')[0]
While ($vmhosts.Contains("$vmhost") -ne "True") {
Write-Host "`n Checking the host exists..." -NoNewline
Write-Host " The host is not reachable." -ForegroundColor Yellow
$vmhost = Read-Host -Prompt "`n Please type in the host name correctly"
}
$vmhost = $vmhost + "*"
# Get the system configuration
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
# Activate the current coredump (this is to identify it properly later in this script)
Write-Host "`n Searching for a coredump file and trying to activat it..." -NoNewline
$arguments = $esxcli2.system.coredump.file.set.CreateArgs()
$arguments.Item('enable') = $true
$arguments.Item('smart') = $true
Try {
$activation = $esxcli2.system.coredump.file.set.Invoke($arguments)
}
Catch [Exception]{
Write-Host " File doen't exist!" -ForegroundColor Yellow
}
# Get the current coredump configuration
$dumpConfigured = $esxcli2.system.coredump.file.get.Invoke().Configured
# Prompt for the coredump removal
If ($dumpConfigured -ne ''){
Write-Host " File exists." -ForegroundColor Green
Write-Host "`n Current configuration: $dumpConfigured"
$choice = $null
While ("Yes","No" -notcontains $choice) {
$choice = Read-Host -Prompt "`n Would you like to remove this file? (Yes/No)"
}
Switch ($choice){
"Yes" {
# Remove the coredump file
Write-Host " Removing the old coredump file..." -NoNewline
$arguments = $esxcli2.system.coredump.file.remove.CreateArgs()
$arguments.Item('force') = $true
$arguments.Item('file') = "$dumpConfigured"
$remove = $esxcli2.system.coredump.file.remove.Invoke($arguments)
Write-Host " Done!" -ForegroundColor Green
}
"No" {
# Exit this script
Write-Host "`n Exiting..."
ex
}
}
}
Write-Host "`n Exiting..."

With this change made you would be able to continue migrating to VMFS-6 without any issue.

If you have any suggestions or concerns, feel free to share them in the comments below.

vSphere 6.x: SEsparse snapshot may cause guest OS file system corruption

Early this month, VMware published a KB 59216 named ‘Virtual Machines running on a SEsparse snapshot may report guest data inconsistencies’.

As per the vendor’s documentation, ‘SEsparse is a snapshot format introduced in vSphere 5.5 for large disks, and is the preferred format for all snapshots in vSphere 6.5 and above with VMFS-6‘. On VMFS-5 and NFS datastores, the SEsparse format is used for virtual disks that are 2 TB or larger; whereas on VMFS-6, SEsparse is the default format for all snapshots.

The knowledge base article states that the issue affects vSphere 5.5 and later versions. As of today, it has been fixed only in VMware ESXi 6.7 Update 1, with the Express Patches pending for VMware ESXi 6.0 and 6.5.

How is this related to your production environment? Well, it depends…

For example, when the backup software creates a system snapshot and it coexists with the operating system (OS) experiencing ‘a burst of non-contiguous write IO in a very short period of time‘, this can potentially trigger the data corruption. There might be other scenarios when a snapshot is used during the OS or software upgrades.

While waiting for a permanent solution, VMware provides a workaround that requires disabling SEsparse IO coalescing on each affected host. The advanced setting that controls IO Coalescing (COW.COWEnableIOCoalescing) is not available through the vSphere Client:

ESXi-SEspare-Issue-01

In spite of that, you can always determine and change its value via PowerCLI:

Get-VMHost | Get-AdvancedSetting -Name “COW.COWEnableIOCoalescing” | select Entity,Name,Value | Format-Table -AutoSize

Get-VMHost | Get-AdvancedSetting -Name “COW.COWEnableIOCoalescing” | ? {$_.Value -eq “1”} | Set-AdvancedSetting -Value “0”

Note: After disabling the IO coalescing, all virtual machines resided on that host ‘must be power-cycled or migrated (vMotion) to other hosts that have the config option set‘.

VMware states there will be a performance penalty when disabling IO coalescing and ‘the extent of degradation depends on the individual virtual machine workload‘.

Note: ‘After patches are released, the workaround needs to be rolled back to regain performance benefits of IO coalescing‘.

24/01/2019 – Update 1: This issue has been resolved with the following set of patches:

vSphere 6.5: Additional considerations when migrating to VMFS-6 – Part 1

For those who use the Virtual Machine File System (VMFS) datastores, one of the steps when upgrading to vSphere 6.5 is to migrate them to VMFS-6.

VMFS6-01

VMware provides a detailed overview of VMFS-6 on the StorageHub, as well as an example of how the migration from VMFS-5 can be automated using PowerCLI.

However, there are three edge cases that require extra steps to continue with the migration. They are as follows:

All those objects, if they exist, prevent the ESXi host from unmounting the datastore, and they need to be moved to a new location before migration continues. The required steps to relocate them will be reviewed in the paragraphs below.

Relocating the system swap

The system swap location can be checked and set via vSphere Client in Configure > System > System Swap settings of the ESXi host.

VMFS6-02

Alternatively, the system swap settings can be retrieved via PowerCLI:

# File name: Get-VMHostSystemSwapLocation.ps1
# Description: This script provides information about the system swap location for ESXi hosts.
#
# 08/10/2018 - Version 1.0
# - Initial Release (based on https://code.vmware.com/forums/2530/vsphere-powercli#534332)
#
# Author: Roman Dronov (c)
# Get information about the system swap location for ESXi hosts
Write-Host "`nSystem swap location:`r" -ForegroundColor Green
ForEach ($vmhost in $(Get-VMHost | sort Name)) {
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$esxcli2.sched.swap.system.get.Invoke() | Select `
@{N='Host Name';E={$vmhost.Name}}, `
@{N='Datastore Name';E={$_.DatastoreName}}, `
@{N='Datastore Active';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreActive)}}, `
@{N='Datastore Enabled';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreEnabled)}}
}
Clear-Variable vmhost,esxcli2 -Scope Global

The script above can be modified to create the system swap files on a new datastore:

# File name: New-VMHostSystemSwapLocation.ps1
# Description: This script creates a system swap file for the ESXi host on a desired datastore.
#
# 08/10/2018 - Version 1.0
# - Initial Release (based on https://code.vmware.com/forums/2530/vsphere-powercli#534332)
#
# Author: Roman Dronov (c)
# Get the host name and check it is valid
$vmhosts = Get-VMHost | ? {$_.ConnectionState -eq "Connected" -or $_.ConnectionState -eq "Maintenance"} | ForEach-Object {$_.Name.Split('.')[0]}
$vmhost = Read-Host -Prompt "`n Please type in the ESXi host name"
while ($vmhosts.Contains($vmhost) -ne "True") {
Write-Host "`n Checking the host exists..." -NoNewline
Write-Host " The host is not reachable." -ForegroundColor Yellow
$vmhost = Read-Host -Prompt "`n Please type in the host name correctly"
}
# Get the datastore name and check it is valid
$datastores = $(Get-Datastore | ? {$_.State -eq "Available"}).Name
$datastore = Read-Host -Prompt "`nPlease type in the datastore name"
while ($datastores.Contains($datastore) -ne "True") {
Write-Host "`n Checking the datastore exists..." -NoNewline
Write-Host " The datastore is not reachable." -ForegroundColor Yellow
$datastore = Read-Host -Prompt "`nPlease type in the datastore name correctly"
}
# Check the host has access to the datastore
$vmhost = $vmhost + "*"
$vmhost = $(Get-VMHost | ? {$_.Name -like "$vmhost"}).Name
$lookup = $(Get-VMHost | ? {$_.Name -like $vmhost} | Get-Datastore).Name
if ($lookup.Contains($datastore) -ne "True") {
Write-Host "`n The datastore is not reachable by the host." -ForegroundColor Yellow
}
else {
# Create the system swap file on the provided datastore
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$arguments = $esxcli2.sched.swap.system.set.CreateArgs()
$arguments.datastorename = $datastore
$arguments.datastoreenabled = "true"
$esxcli2.sched.swap.system.set.Invoke($arguments) | Out-Null
# Print out the results
Write-Host "`nSystem swap location:`r" -ForegroundColor Green
$esxcli2.sched.swap.system.get.Invoke() | Select `
@{N='Host Name';E={$vmhost}}, `
@{N='Datastore Name';E={$_.DatastoreName}}, `
@{N='Active';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreActive)}}, `
@{N='Enabled';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreEnabled)}}
}
Clear-Variable vmhost,vmhosts,datastore,datastores,esxcli2,arguments -Scope Global | Out-Null

Note: The host reboot is not required to apply this change.

Moving the persistent scratch location

A persistent scratch location helps when investigating the host failures. It preserves the host log files on a shared datastore. So they can be reachable for troubleshooting, even if the host experienced the Purple Screen of Death (PSOD) or went down.

To identify the persistent scratch location, filter the key column by the ‘scratch’ word in Settings > System > Advanced System Settings of the ESXi host in vSphere Client.

VMFS6-03

You only need to point the ScratchConfig.ConfiguredScratchLocation setting to a new location and reboot the host for this change to take effect.

Note: Before doing any changes, make sure that the .locker folder (should be unique for each configured host to avoid data mixing or overwrites) has been created on the desired datastore. Otherwise, the persistent scratch location remains the same.

To review and modify advanced host parameters including the persistent scratch location via PowerCLI, look for two cmdlets named Get-AdvancedSetting and Set-AdvancedSetting. This procedure is well-documented in KB 1033696.

An information about how to automate the diagnostic coredump file relocation will be covered in Part 2 or this series later on. Keep you posted!