vSphere 6.5: Additional considerations when migrating to VMFS-6 – Part 2

In the Part 1 of this series, I was writing about the most common cases which might prevent a successful migration to VMFS-6. There is another one to cover.

For ESXi hosts that boot from a flash storage or from memory, a diagnostic core dump file can also be placed on a shared datastore. You won’t be able to un-mount this datastore without deleting a core dump first.

VMware recommends using an esxcli utility to view/edit the core dump settings. This also can be automated via PowerCLI.

To check if the core dump file exists and is active, please use the following code:

# File name: Get-VMHostCoredumpLocation.ps1
# Description: This script checks the core dump location for ESXi hosts.
#
# 11/10/2018 - Version 1.0
# - Initial Release
#
# Author: Roman Dronov (c)
# Get information about the core dump location for ESXi hosts
Write-Host "`nCore Dump Settings:`r" -ForegroundColor Green
ForEach ($vmhost in $(Get-VMHost | sort Name)) {
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$esxcli2.system.coredump.file.get.Invoke() | Select @{N='Host Name';E={$vmhost}},@{N='Active Core Dump File';E={$_.Active}} #| Format-List
}
Clear-Variable vmhost,esxcli2 -Scope Global

To delete an old configuration that points to the VMFS-5 datastore, the following script can help:

# File name: Remove-VMHostCoredumpLocation.ps1
# Description: This script removes a core dump file for the particular ESXi host.
#
# 11/03/2019 - Version 1.0
# - Initial release
#
# Author: Roman Dronov (c)
# Define common functions
function ex {exit}
# Get the host name and check it is valid
$vmhosts = Get-VMHost | ? {$_.ConnectionState -eq "Connected" -or $_.ConnectionState -eq "Maintenance"} | ForEach-Object {$_.Name.Split('.')[0]}
$vmhost = (Read-Host -Prompt "`n Please type in the ESXi host name").Split('.')[0]
While ($vmhosts.Contains("$vmhost") -ne "True") {
Write-Host "`n Checking the host exists..." -NoNewline
Write-Host " The host is not reachable." -ForegroundColor Yellow
$vmhost = Read-Host -Prompt "`n Please type in the host name correctly"
}
$vmhost = $vmhost + "*"
# Get the system configuration
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
# Activate the current coredump (this is to identify it properly later in this script)
Write-Host "`n Searching for a coredump file and trying to activat it..." -NoNewline
$arguments = $esxcli2.system.coredump.file.set.CreateArgs()
$arguments.Item('enable') = $true
$arguments.Item('smart') = $true
Try {
$activation = $esxcli2.system.coredump.file.set.Invoke($arguments)
}
Catch [Exception]{
Write-Host " File doen't exist!" -ForegroundColor Yellow
}
# Get the current coredump configuration
$dumpConfigured = $esxcli2.system.coredump.file.get.Invoke().Configured
# Prompt for the coredump removal
If ($dumpConfigured -ne ''){
Write-Host " File exists." -ForegroundColor Green
Write-Host "`n Current configuration: $dumpConfigured"
$choice = $null
While ("Yes","No" -notcontains $choice) {
$choice = Read-Host -Prompt "`n Would you like to remove this file? (Yes/No)"
}
Switch ($choice){
"Yes" {
# Remove the coredump file
Write-Host " Removing the old coredump file..." -NoNewline
$arguments = $esxcli2.system.coredump.file.remove.CreateArgs()
$arguments.Item('force') = $true
$arguments.Item('file') = "$dumpConfigured"
$remove = $esxcli2.system.coredump.file.remove.Invoke($arguments)
Write-Host " Done!" -ForegroundColor Green
}
"No" {
# Exit this script
Write-Host "`n Exiting..."
ex
}
}
}
Write-Host "`n Exiting..."

With this change made you would be able to continue migrating to VMFS-6 without any issue.

If you have any suggestions or concerns, feel free to share them in the comments below.

PowerCLI: Housekeeping after upgrading to the latest version of PowerCLI

With a new version of VMware PowerCLI 11.1.0 released, it is time to plan for an upgrade.

Even if it is simply a matter of one command in PowerShell, you need to remember that the Update-Module cmdlet does not remove the old modules from the operating system (OS). So over the time, it might look a bit messy:

Following a few threads in regards to this issue (here and here), I wrote a simple script that removes the old versions of VMware.PowerCLI and its dependencies from Windows OS.

# File name: Remove-OldPowerCLI.ps1
# Description: This script removes the old versions of VMware.PowerCLI and its dependencies
#
# 21/12/2018 - Version 1.0
# - Initial Release (based on https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/14934876-update-module-needs-flag-to-remove-previous-versio)
#
# Author: Roman Dronov (c)
$modules = @((Get-Module -ListAvailable | ? {$_.Name -like "VMware*"}).Name | Get-Unique)
foreach ($module in $modules){
$latest = Get-InstalledModule -Name $module
Get-InstalledModule -Name $module -AllVersions | ? {$_.Version -ne $latest.Version} | Uninstall-Module -Force -Verbose
}

A few minutes later, it looks much cleaner with only the latest version of VMware.PowerCLI remaining:

Hope it will be useful for you as well.

[URGENT] vSAN 6.6.1: Potential data loss due to resynchronisation mixed with object expansion

Last week VMware released an urgent hotfix to remediate potential data loss in vSAN 6.6.1 due to resynchronisation mixed with object expansion.

This is a known issue affecting earlier versions of ESXi 6.5 Express Patch 9. The vendor states that a sequence of the following operations might cause it:

  1. vSAN initiates resynchronisation to maintain data availability.
  2. You expand a virtual machine disk (VMDK).
  3. vSAN initiates another resync after the VMDK expansion.

Detailed information about this problem is available in KB 60299.

If you are a vSAN customer, additional considerations are required before applying this hotfix:

  • If hosts have already been upgraded to ESXi650-201810001, you can proceed with this upgrade,
  • If hosts have not been upgraded to ESXi650-201810001, and if an expansion of a VMDK is likely, the in-place expansion should be disabled on all of them by setting the VSAN.ClomEnableInplaceExpansion advanced configuration option to ‘0‘.

The VSAN.ClomEnableInplaceExpansion advanced configuration option is not available in vSphere Client. I use the following one-liner scrips to determine and change its value via PowerCLI:

# To check the current status
Get-VMHost | Get-AdvancedSetting -Name “VSAN.ClomEnableInplaceExpansion” | select Entity, Name, Value | Format-Table -AutoSize

# To disable the in-place expansion
Get-VMHost | Get-AdvancedSetting -Name “VSAN.ClomEnableInplaceExpansion” | ? {$_.Value -eq “1”} | Set-AdvancedSetting -Value “0”

Note: No reboot is required after the change.

After hosts were upgraded to ESXi650-201810001 or ESXi650-201811002, you can set VSAN.ClomEnableInplaceExpansion back to ‘1‘ to enable the in-place expansion.

vSphere 6.5: Failed to deploy OVF package from the Content Library

Working on automating the virtual machine (VM) provisioning from an OVF template in a content library in vSphere 6.5 Update 2, I ran across an interesting behaviour. Some virtual machines were created without any issues, whereas others have been failing with the error message ‘Failed to deploy OVF package.’

OVF-Template-Issue-00

OVF-Template-Issue-01

In the vpxd.log on vCenter Server, this error message looks as follows:

info vpxd[7F83D22C5700] [Originator@6876 sub=Default opID=SelectResourcePageMediator-validate-335519-ngc:70031913-ae-71-7e-92-01] [VpxLRO] — ERROR task-51355 — TEST-VM-01 — ResourcePool.ImportVAppLRO: vim.fault.OvfImportFailed:
–> Result:
–> (vim.fault.OvfImportFailed) {
–> faultCause = (vim.fault.NotFound) {
–> faultCause = (vmodl.MethodFault) null,
–> faultMessage = (vmodl.LocalizableMessage) [
–> (vmodl.LocalizableMessage) {
–> key = “com.vmware.ovfs.ovfs-main.ovfs.object_not_found”,
–> arg = (vmodl.KeyAnyValue) [
–> (vmodl.KeyAnyValue) {
–> key = “0”,
–> value = “/TEST-VM-01/nvram”
–> }
–> ],
–> message = “The specified object /TEST-VM-01/nvram could not be found.”
–> }
–> ]
–> msg = “The specified object /TEST-VM-01/nvram could not be found.”
–> },
–> faultMessage = <unset>
–> msg = “”
–> }
–> Args:
–>

The NVRAM file contains information such as BIOS settings. As VMware states here, a new NVRAM file is created ‘when a virtual machine is migrated, migrated with vMotion, cloned or deployed from a template.’

Looking through the knowledge base articles on the vendor’s website, I have found the following hint in KB 2108718:

Error 2: Could not find the file

This issue occurs if the NVRAM file mentioned in the virtual machines configuration file(*.vmx) is not available anymore.

As you might know, templates are stored in OVF format in the content library. Two files compose a template: an OVF descriptor (.ovf) and a virtual disk file (.vmdk). During the VM provisioning phase, the settings listed in the OVF descriptor form the virtual machine’s configuration file (.vmx). So my assumption was that one of those settings caused a conflict that prevented vSphere from creating those VMs.

When we clone a template to the content library, only minimum required settings are stored in the OVF descriptor. However, there is an option available to include extra configuration.

OVF-Template-Issue-02

I decided to compare a vanilla OVF descriptor with the one with extra settings. While doing this comparison, one particular line in the file with extra configuration caught my eye:

<vmw:ExtraConfig ovf:required=”false” vmw:key=”firmware” vmw:value=”efi”/>

For some reason, the provisioning task is not able to succeed in creating new virtual machines with the EFI firmware from the template in vSphere 6.5. This issue happens regardless of the guest operating system version and content library type (local or subscribed).

Surprisingly, I do not see any problem with deploying VMs with the EFI firmware from the content library in vSphere 6.0. I need some time to test this functionality in vSphere 6.7, before opening a support request with VMware.

Workaround: Feel free to use a PowerCLI script below to set the firmware type to EFI after provisioning a new VM.

# File name: Set-EFIFirmware.ps1
#
# Description: This script checks and sets the firmware type to EFI for a particular VM
#
# 26/07/2018 - Version 1.0
# - Initial Release
#
# Author: Roman Dronov (c)
# Clear the screen
Clear-Host
# Set the function
function Set-Firmware {
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true, Position=0)]
[string]$Argument
)
$choice = Read-Host "`n Would you like to set the firmware type to EFI? (Yes/No)"
while ("Yes","No" -notcontains $choice){
$choice = Read-Host "`n Would you like to set the firmware type to EFI? (Yes/No)"
}
switch ($choice){
"Yes" {
Write-Host " Checking the VM power state..." -NoNewline
if ($vm.PowerState -eq 'PoweredOff'){
Write-Host " The VM is powered off." -ForegroundColor Green
Write-Host " Setting the firmware type..." -NoNewline
if ($Argument -eq '0'){
Get-VM -Name $vmName | New-AdvancedSetting -Name 'firmware' -Value 'efi' -Confirm:$false | Out-Null
}
elseif ($Argument -eq '1'){
Get-VM -Name $vmName | Get-AdvancedSetting -Name 'firmware' | Set-AdvancedSetting -Value 'efi' -Confirm:$false | Out-Null
}
Write-Host " Completed successfully!" -ForegroundColor Green
}
else {
Write-Host " $vm is powered on!" -ForegroundColor Red
Write-Host " Please shutdown the VM and run this script again." -ForegroundColor Yellow
}
Write-Host "`n Exiting..."
}
"No" {
Write-Host " Cancelled by the user." -ForegroundColor Yellow
Write-Host "`n Exiting..."
}
}
}
# Promp for the VM name
$vmName = $null
while ($vmName -eq $null){
$vmName = Read-Host -Prompt "`n Input the VM name"
if ($vmName -eq '' -or -not (Get-VM -Name $vmName -ErrorAction SilentlyContinue)){
Write-Host " Invalid VM name, please re-enter." -ForegroundColor Yellow
$vmName = $null
}
}
# Check the firmware advanced setting
$vmName = $vmName.ToUpper()
$vm = Get-VM -Name $vmName
$vmFirmware = Get-AdvancedSetting -Entity $vm -Name 'firmware'
Write-Host "`n Checking the firmware advanced setting for $vmName..."
if (-not $vmFirmware){
Write-Host " Parameter not exist." -ForegroundColor Yellow
# Set the firmware advanced setting to EFI
Set-Firmware -Argument 0
}
elseif ($vmFirmware.Value -ieq 'efi'){
Write-Host " $vmName has the firmware type set to EFI already." -ForegroundColor Green
Write-Host "`n Exiting..."
}
else {
$vmFirmwareValue = $vmFirmware.Value
$vmFirmwareValue = $vmFirmwareValue.ToUpper()
Write-Host " $vmName has the firmware type set to $vmFirmwareValue." -ForegroundColor Yellow
# Set the firmware advanced setting to EFI
Set-Firmware -Argument 1
}
view raw Set-EFIFirmware.ps1 hosted with ❤ by GitHub

 

PowerCLI Tips: Get information about datastores

This script below provides information about the datastore capacity, consumed and provisioned space (all three in GB), as well as the over-provisioning ratio.

# File name: Get-DatastoreUsage.ps1
# Description: This script provides information about datastore usage.
#
# 17/07/2018 - Version 1.0
# - Initial Release (based on https://code.vmware.com/forums/2530/vsphere-powercli#576268)
#
# Author: Roman Dronov (c)
Get-Datastore | select Name,`
@{N='Capacity (GB)';E={[math]::Round($_.ExtensionData.Summary.Capacity/1GB,2)}},`
@{N='Consumed (GB)';E={[math]::Round(($_.ExtensionData.Summary.Capacity - $_.ExtensionData.Summary.FreeSpace)/1GB,2)}},`
@{N='Provisioned (GB)';E={[math]::Round(($_.ExtensionData.Summary.Capacity - $_.ExtensionData.Summary.FreeSpace + $_.ExtensionData.Summary.Uncommitted)/1GB,2)}},`
@{N='Over-Provisioning Ratio';E={[math]::Round((($_.ExtensionData.Summary.Capacity - $_.ExtensionData.Summary.FreeSpace + $_.ExtensionData.Summary.Uncommitted)/$_.ExtensionData.Summary.Capacity),2)}} |`
Format-Table -AutoSize

It can be useful for storage monitoring and reporting.