vSphere 6.7 Design: A new vSphere Upgrade section on vSphere Central

It is great to know that VMware continues to work on improving vSphere Central.

For those who are not familiar with this resource, vSphere Central provides information which compliments an official documentation in terms of the product features and best practices. It also helps with configuring and administering different components of vSphere 6.5 and 6.7.

vSphere Central library is divided into the following categories:

  • vCenter Server,
  • Security,
  • ESXi Host and Virtual Machine,
  • Resource Management and Availability,
  • Operations Management,
  • Developer and Automation Interfaces and the list is growing.

Now it includes a new section dedicated to upgrading to vSphere 6.7.

vSphere-Central-01

As per VMware, this document helps with vSphere upgrade, including pre-upgrade considerations, upgrade process and post-upgrade considerations.

It is worth reading it when planning for an upgrade, preparing for the official exams, or the VCDX defence. As a bonus, all content can be exported to the PDF format if needed with one click!

vSphere 6.5: Additional considerations when migrating to VMFS-6 – Part 1

For those who use the Virtual Machine File System (VMFS) datastores, one of the steps when upgrading to vSphere 6.5 is to migrate them to VMFS-6.

VMFS6-01

VMware provides a detailed overview of VMFS-6 on the StorageHub, as well as an example of how the migration from VMFS-5 can be automated using PowerCLI.

However, there are three edge cases that require extra steps to continue with the migration. They are as follows:

All those objects, if they exist, prevent the ESXi host from unmounting the datastore, and they need to be moved to a new location before migration continues. The required steps to relocate them will be reviewed in the paragraphs below.

Relocating the system swap

The system swap location can be checked and set via vSphere Client in Configure > System > System Swap settings of the ESXi host.

VMFS6-02

Alternatively, the system swap settings can be retrieved via PowerCLI:

# File name: Get-VMHostSystemSwapLocation.ps1
# Description: This script provides information about the system swap location for ESXi hosts.
#
# 08/10/2018 - Version 1.0
# - Initial Release (based on https://code.vmware.com/forums/2530/vsphere-powercli#534332)
#
# Author: Roman Dronov (c)
# Get information about the system swap location for ESXi hosts
Write-Host "`nSystem swap location:`r" -ForegroundColor Green
ForEach ($vmhost in $(Get-VMHost | sort Name)) {
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$esxcli2.sched.swap.system.get.Invoke() | Select `
@{N='Host Name';E={$vmhost.Name}}, `
@{N='Datastore Name';E={$_.DatastoreName}}, `
@{N='Datastore Active';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreActive)}}, `
@{N='Datastore Enabled';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreEnabled)}}
}
Clear-Variable vmhost,esxcli2 -Scope Global

The script above can be modified to create the system swap files on a new datastore:

# File name: New-VMHostSystemSwapLocation.ps1
# Description: This script creates a system swap file for the ESXi host on a desired datastore.
#
# 08/10/2018 - Version 1.0
# - Initial Release (based on https://code.vmware.com/forums/2530/vsphere-powercli#534332)
#
# Author: Roman Dronov (c)
# Get the host name and check it is valid
$vmhosts = Get-VMHost | ? {$_.ConnectionState -eq "Connected" -or $_.ConnectionState -eq "Maintenance"} | ForEach-Object {$_.Name.Split('.')[0]}
$vmhost = Read-Host -Prompt "`n Please type in the ESXi host name"
while ($vmhosts.Contains($vmhost) -ne "True") {
Write-Host "`n Checking the host exists..." -NoNewline
Write-Host " The host is not reachable." -ForegroundColor Yellow
$vmhost = Read-Host -Prompt "`n Please type in the host name correctly"
}
# Get the datastore name and check it is valid
$datastores = $(Get-Datastore | ? {$_.State -eq "Available"}).Name
$datastore = Read-Host -Prompt "`nPlease type in the datastore name"
while ($datastores.Contains($datastore) -ne "True") {
Write-Host "`n Checking the datastore exists..." -NoNewline
Write-Host " The datastore is not reachable." -ForegroundColor Yellow
$datastore = Read-Host -Prompt "`nPlease type in the datastore name correctly"
}
# Check the host has access to the datastore
$vmhost = $vmhost + "*"
$vmhost = $(Get-VMHost | ? {$_.Name -like "$vmhost"}).Name
$lookup = $(Get-VMHost | ? {$_.Name -like $vmhost} | Get-Datastore).Name
if ($lookup.Contains($datastore) -ne "True") {
Write-Host "`n The datastore is not reachable by the host." -ForegroundColor Yellow
}
else {
# Create the system swap file on the provided datastore
$esxcli2 = Get-EsxCli -VMHost $vmhost -V2
$arguments = $esxcli2.sched.swap.system.set.CreateArgs()
$arguments.datastorename = $datastore
$arguments.datastoreenabled = "true"
$esxcli2.sched.swap.system.set.Invoke($arguments) | Out-Null
# Print out the results
Write-Host "`nSystem swap location:`r" -ForegroundColor Green
$esxcli2.sched.swap.system.get.Invoke() | Select `
@{N='Host Name';E={$vmhost}}, `
@{N='Datastore Name';E={$_.DatastoreName}}, `
@{N='Active';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreActive)}}, `
@{N='Enabled';E={(Get-Culture).TextInfo.ToTitleCase($_.DatastoreEnabled)}}
}
Clear-Variable vmhost,vmhosts,datastore,datastores,esxcli2,arguments -Scope Global | Out-Null

Note: The host reboot is not required to apply this change.

Moving the persistent scratch location

A persistent scratch location helps when investigating the host failures. It preserves the host log files on a shared datastore. So they can be reachable for troubleshooting, even if the host experienced the Purple Screen of Death (PSOD) or went down.

To identify the persistent scratch location, filter the key column by the ‘scratch’ word in Settings > System > Advanced System Settings of the ESXi host in vSphere Client.

VMFS6-03

You only need to point the ScratchConfig.ConfiguredScratchLocation setting to a new location and reboot the host for this change to take effect.

Note: Before doing any changes, make sure that the .locker folder (should be unique for each configured host to avoid data mixing or overwrites) has been created on the desired datastore. Otherwise, the persistent scratch location remains the same.

To review and modify advanced host parameters including the persistent scratch location via PowerCLI, look for two cmdlets named Get-AdvancedSetting and Set-AdvancedSetting. This procedure is well-documented in KB 1033696.

An information about how to automate the diagnostic coredump file relocation will be covered in Part 2 or this series later on. Keep you posted!

vSphere 6.5: Failed to deploy OVF package from the Content Library

Working on automating the virtual machine (VM) provisioning from an OVF template in a content library in vSphere 6.5 Update 2, I ran across an interesting behaviour. Some virtual machines were created without any issues, whereas others have been failing with the error message ‘Failed to deploy OVF package.’

OVF-Template-Issue-00

OVF-Template-Issue-01

In the vpxd.log on vCenter Server, this error message looks as follows:

info vpxd[7F83D22C5700] [Originator@6876 sub=Default opID=SelectResourcePageMediator-validate-335519-ngc:70031913-ae-71-7e-92-01] [VpxLRO] — ERROR task-51355 — TEST-VM-01 — ResourcePool.ImportVAppLRO: vim.fault.OvfImportFailed:
–> Result:
–> (vim.fault.OvfImportFailed) {
–> faultCause = (vim.fault.NotFound) {
–> faultCause = (vmodl.MethodFault) null,
–> faultMessage = (vmodl.LocalizableMessage) [
–> (vmodl.LocalizableMessage) {
–> key = “com.vmware.ovfs.ovfs-main.ovfs.object_not_found”,
–> arg = (vmodl.KeyAnyValue) [
–> (vmodl.KeyAnyValue) {
–> key = “0”,
–> value = “/TEST-VM-01/nvram”
–> }
–> ],
–> message = “The specified object /TEST-VM-01/nvram could not be found.”
–> }
–> ]
–> msg = “The specified object /TEST-VM-01/nvram could not be found.”
–> },
–> faultMessage = <unset>
–> msg = “”
–> }
–> Args:
–>

The NVRAM file contains information such as BIOS settings. As VMware states here, a new NVRAM file is created ‘when a virtual machine is migrated, migrated with vMotion, cloned or deployed from a template.’

Looking through the knowledge base articles on the vendor’s website, I have found the following hint in KB 2108718:

Error 2: Could not find the file

This issue occurs if the NVRAM file mentioned in the virtual machines configuration file(*.vmx) is not available anymore.

As you might know, templates are stored in OVF format in the content library. Two files compose a template: an OVF descriptor (.ovf) and a virtual disk file (.vmdk). During the VM provisioning phase, the settings listed in the OVF descriptor form the virtual machine’s configuration file (.vmx). So my assumption was that one of those settings caused a conflict that prevented vSphere from creating those VMs.

When we clone a template to the content library, only minimum required settings are stored in the OVF descriptor. However, there is an option available to include extra configuration.

OVF-Template-Issue-02

I decided to compare a vanilla OVF descriptor with the one with extra settings. While doing this comparison, one particular line in the file with extra configuration caught my eye:

<vmw:ExtraConfig ovf:required=”false” vmw:key=”firmware” vmw:value=”efi”/>

For some reason, the provisioning task is not able to succeed in creating new virtual machines with the EFI firmware from the template in vSphere 6.5. This issue happens regardless of the guest operating system version and content library type (local or subscribed).

Surprisingly, I do not see any problem with deploying VMs with the EFI firmware from the content library in vSphere 6.0. I need some time to test this functionality in vSphere 6.7, before opening a support request with VMware.

Workaround: Feel free to use a PowerCLI script below to set the firmware type to EFI after provisioning a new VM.

# File name: Set-EFIFirmware.ps1
#
# Description: This script checks and sets the firmware type to EFI for a particular VM
#
# 26/07/2018 - Version 1.0
# - Initial Release
#
# Author: Roman Dronov (c)
# Clear the screen
Clear-Host
# Set the function
function Set-Firmware {
[CmdletBinding()]
Param
(
[Parameter(Mandatory=$true, Position=0)]
[string]$Argument
)
$choice = Read-Host "`n Would you like to set the firmware type to EFI? (Yes/No)"
while ("Yes","No" -notcontains $choice){
$choice = Read-Host "`n Would you like to set the firmware type to EFI? (Yes/No)"
}
switch ($choice){
"Yes" {
Write-Host " Checking the VM power state..." -NoNewline
if ($vm.PowerState -eq 'PoweredOff'){
Write-Host " The VM is powered off." -ForegroundColor Green
Write-Host " Setting the firmware type..." -NoNewline
if ($Argument -eq '0'){
Get-VM -Name $vmName | New-AdvancedSetting -Name 'firmware' -Value 'efi' -Confirm:$false | Out-Null
}
elseif ($Argument -eq '1'){
Get-VM -Name $vmName | Get-AdvancedSetting -Name 'firmware' | Set-AdvancedSetting -Value 'efi' -Confirm:$false | Out-Null
}
Write-Host " Completed successfully!" -ForegroundColor Green
}
else {
Write-Host " $vm is powered on!" -ForegroundColor Red
Write-Host " Please shutdown the VM and run this script again." -ForegroundColor Yellow
}
Write-Host "`n Exiting..."
}
"No" {
Write-Host " Cancelled by the user." -ForegroundColor Yellow
Write-Host "`n Exiting..."
}
}
}
# Promp for the VM name
$vmName = $null
while ($vmName -eq $null){
$vmName = Read-Host -Prompt "`n Input the VM name"
if ($vmName -eq '' -or -not (Get-VM -Name $vmName -ErrorAction SilentlyContinue)){
Write-Host " Invalid VM name, please re-enter." -ForegroundColor Yellow
$vmName = $null
}
}
# Check the firmware advanced setting
$vmName = $vmName.ToUpper()
$vm = Get-VM -Name $vmName
$vmFirmware = Get-AdvancedSetting -Entity $vm -Name 'firmware'
Write-Host "`n Checking the firmware advanced setting for $vmName..."
if (-not $vmFirmware){
Write-Host " Parameter not exist." -ForegroundColor Yellow
# Set the firmware advanced setting to EFI
Set-Firmware -Argument 0
}
elseif ($vmFirmware.Value -ieq 'efi'){
Write-Host " $vmName has the firmware type set to EFI already." -ForegroundColor Green
Write-Host "`n Exiting..."
}
else {
$vmFirmwareValue = $vmFirmware.Value
$vmFirmwareValue = $vmFirmwareValue.ToUpper()
Write-Host " $vmName has the firmware type set to $vmFirmwareValue." -ForegroundColor Yellow
# Set the firmware advanced setting to EFI
Set-Firmware -Argument 1
}
view raw Set-EFIFirmware.ps1 hosted with ❤ by GitHub

 

vSphere 6.0: The operation failed due to The operation is not allowed in the current state.

Working on the PowerCLI script to automate VM provisioning in vSphere 6.0, I ran across the following error message:

New-VM-Issue-02

The script was simply creating a new virtual machine from the Content Library template.

New-VM -Name $vmName -VMHost $vmHost -Location $vmLocation -Datastore $vmDatastore -ContentLibraryItem $vmTemplate

Investigating this behaviour further, I have found that it was related to a failing cross vMotion placement. In vSphere Client those errors were looking like this:

New-VM-Issue-01

It was then I concluded it might be related to DRS being disabled for that cluster. Setting DRS to the Manual mode resolved this issue.

The next step was to check whether this problem could be reproduced for other templates. Surprisingly, the VM provisioning from some of them went smoothly, regardless of the DRS status.

Remembering about my previous articles (here and here) related to the Content Library, I decided to have a look into the VM configuration files (*.ovf) for both templates to compare. The only difference was related to the following storage settings which exist for the faulty templates:

<vmw:StorageGroupSection ovf:required=”false” vmw:id=”group1″ vmw:name=”EMC VNX VM Storage Policy”>
<Info>Storage policy for group of disks</Info>
<vmw:Description>The EMC VNX VM Storage Policy storage policy group</vmw:Description>
</vmw:StorageGroupSection>

<vmw:StorageSection ovf:required=”false” vmw:group=”group1″>
<Info>Storage policy group reference</Info>
</vmw:StorageSection>

As expected, removing those block of data from the OVF-file and re-uploading it to the Content Library resolved provisioning issue completely.

A couple of notes:

  • There is no way to identify any problems with the VM configuration using the Content Library configuration page or ‘New VM from Library…’ dialogue in vCenter Web Client.
  • Even if a modified OVF-file has been uploaded to the local Content Library, this change is not going to propagate to the subscribed ones. You need to repeat this process for all subscribed Content Libraries manually.

VMware: StorageHub Portal Refresh

For those of us who have been interested in getting explicit information about VMware vSAN, Site Recovery Manager, and vSphere storage in general, VMware StorageHub was a unique source of technical documentation.

It is great to see the vendor working on improving this portal with the design and user interface refresh.

SorageHub-01

Now it is possible to choose between English (US) and Mandarine languages for some of the articles.

SorageHub-02

All seems quite logical, and I personally like navigation and how fast search works.

SorageHub-03

Well done, VMware!

vSAN 6.5: Virtual Machine with more than 64GB memory fails to Storage vMotion to vSAN cluster

VMware has just posted an article in the Virtual Blocks blog which describes this behaviour. It happens only when trying to Storage vMotion a virtual machine with a swap file larger than 64GB to the vSAN datastore.

The task fails and generates the following error messages:

SvMotion-Issue-01

There are two possible workarounds available: either increase the swap file maximum size on the destination ESXi host or set a reservation of memory on the virtual machine. The former one is more preferable, as it does not require host reboot.

VMware provides a KB 2150316 with “more log samples and specifics for identifying the issue as a cause of a migration failure”.

vSphere Next: A new beta refresh and more

Beta Testing

Less than two months since VMware announced the availability of a vSphere beta and it has been refreshed with the new features and bugfixes. To participate in the program, candidates should indicate their interest by filling out this simple form.

I personally think time around Christmas holidays is the best one for tech geeks to dedicate some of their time and have an understanding of what’s next.

The beta refresh is available as a downloadable media and as a hosted environment in the Hands-on-Labs.

For those folks who need access to the full range of technologies from VMware, the VMware User Group has just announced a 10% discount on the VMUG Advantage subscription. This offer is available until December 31st, 2017.

All this sounds like a great Christmas gift from the vendor. Thank you, VMware!

vSphere 6.0: Templates are shown as ‘Unknown’ in the local Content Library

Another day another case… This time, I was surprised to see an empty list when provisioning a new virtual machine from a Content Library.

CL Issue - 01

I went to check the Content Library status and found all templates were shown as ‘Unknown’ in there.

CL Issue - 02

Funny enough, this behaviour was happening only with the local Content Library. A subscribed one didn’t have any issues at all, and the synchronisation between those two was still working.

CL Issue - 03

More interestingly, the objects of other types were not affected at all.

There is not enough information about how to troubleshoot the Content Library in vSphere 6.0. Some of the diagnostic files can be found in the /var/log/vmware/vdcs directory on vCenter Server Appliance (VCSA). Unfortunately, they are not that informative.

So I opened the case with VMware GSS (SR # 17504701707) and the response was that “this issue is occurring as there is a corrupted or stale PID for the content library service which has not been cleared from the previous running state.”

VMware is working on this to be resolved, but no ETA at the moment.

A workaround provided by VMware:

  1. Connect to the vCenter Server Appliance using SSH and root credentials.
  2. Navigate to /var/log/vmware/vdcs.
  3. Create a new folder to move the PID file to.
  4. Move the vmware-vdcs.pid file to the folder created in step 3.
  5. Reboot the vCenter Server Appliance (In case of external PSC, reboot the PSC first and then the vCenter).

I personally found that restarting VCSA resolves this issue. However, it reappears after some time.

vSphere 6.5 Update 1 has been released!

VMware has just released a major update to vCenter Server 6.5 with a plenty of exciting features including:

  • Ability to run the vCenter Server Appliance GUI and CLI installers on Microsoft Windows 2012 x64 bit, Microsoft Windows 2012 R2 x64 bit, Microsoft Windows 2016 x64 bit, and macOS Sierra
  • vSAN software upgrades through integration with vSphere Update Manager
  • Support for Microsoft SQL Server 2016, Microsoft SQL Server 2016 SP1, and Microsoft SQL Server 2014 SP2 as external databases for vCenter Server
  • Improved HTML5-based vSphere Client
  • Increased configuration maximums for the Linked vCenter Server instances
  • vSphere Replication updates
  • Driver updates and hips of resolved issue.

The following products have been updated:

Updated packages can be found here.

More information about new features is available following those links:

I have a few support requests with VMware GSS open, which should be resolved in this release of the product.

Will keep you posted after upgrading my environment and finishing testing.