‘Response with status: 401 Unauthorized for URL’ while performing Update Manager tasks on Firefox in vSphere 6.5

Updating to vCenter Server 6.5 Update 2d has brought long-awaited support for the vSphere Update Manager (VUM) functionality to vSphere Client.

However, you should be aware of some incompatibility between a VUM plug-in in vSphere Client and Mozilla Firefox.

Update Manager gives a ‘Response with status: 401 Unauthorized for URL‘ error message at random occasions when used with the recent versions of this web-browser (tested with Firefox 63.x and 64.x):

Searching on the VMware’s web-site led me to the KB 59696 that describes the similar issue in vSphere 6.7 and the workaround provided worked for me.

First you need to clear the browser cache:

Then stop Firefox from caching pages by modifying the browser’s default configuration:

According to VMware, they are currently working towards resolving this issue in future releases.

IMPORTANT: Removing a virtual machine folder from inventory deletes the VMs within it from disk when using the vSphere Client

VMware has recently released a new KB 65207 warning about this issue.

Normally, all powered off VMs would be unregistered from vCenter when you remove a parent folder that includes those objects using the vSphere Web Client. The same is expected in vSphere Client.

vCenter logs similar events when the folder object was removed in vSphere Web Client or vSphere Client.

In my case, vCenter generated two events – one for deleting the folder, and another one for removing TEST-VM-02 from the inventory.

The only difference is that vCenter deleted TEST-VM-02 from the corresponding datastore when removing TEST-Folder from the inventory. Moreover, no events are generated during the delete operation!!!

This issue affects both VMware vCenter Server 6.5.x and VMware vCenter Server 6.7.x.

Workaround (provided by the vendor): Use the vSphere Web Client (Flex) for unregistering all virtual machines in a VM folder or move the VMs from that folder before removing it.

22/01/2019 – Update 1: This issue has been resolved in VMware vCenter Server 6.7 Update 1b. Please see the release notes for more details.

PowerCLI: Housekeeping after upgrading to the latest version of PowerCLI

With a new version of VMware PowerCLI 11.1.0 released, it is time to plan for an upgrade.

Even if it is simply a matter of one command in PowerShell, you need to remember that the Update-Module cmdlet does not remove the old modules from the operating system (OS). So over the time, it might look a bit messy:

Following a few threads in regards to this issue (here and here), I wrote a simple script that removes the old versions of VMware.PowerCLI and its dependencies from Windows OS.

A few minutes later, it looks much cleaner with only the latest version of VMware.PowerCLI remaining:

Hope it will be useful for you as well.

vCSA 6.x: WinSCP fails with the error ‘Received too large SFTP packet’

Back to basics… When you try connecting to vCenter Server Virtual Appliance 6.x (vCSA) using WinSCP, the error message ‘Received too large (1433299822 B) SFTP packet‘ might appear.

vCSA6x-WinSCP-01

This is due to the configuration of vCSA when the default shell used for the root account set to the Appliance Shell.

To fix this issue, VMware recommends switching the vCSA 6.x to the Bash Shell. This can be done in the SSH session with the following command:

chsh -s /bin/bash root

Note: You need to log out from the Appliance Shell and log in back again for the changes to take effect.

[URGENT] vSAN 6.6.1: Potential data loss due to resynchronisation mixed with object expansion

Last week VMware released an urgent hotfix to remediate potential data loss in vSAN 6.6.1 due to resynchronisation mixed with object expansion.

This is a known issue affecting earlier versions of ESXi 6.5 Express Patch 9. The vendor states that a sequence of the following operations might cause it:

  1. vSAN initiates resynchronisation to maintain data availability.
  2. You expand a virtual machine disk (VMDK).
  3. vSAN initiates another resync after the VMDK expansion.

Detailed information about this problem is available in KB 60299.

If you are a vSAN customer, additional considerations are required before applying this hotfix:

  • If hosts have already been upgraded to ESXi650-201810001, you can proceed with this upgrade,
  • If hosts have not been upgraded to ESXi650-201810001, and if an expansion of a VMDK is likely, the in-place expansion should be disabled on all of them by setting the VSAN.ClomEnableInplaceExpansion advanced configuration option to ‘0‘.

The VSAN.ClomEnableInplaceExpansion advanced configuration option is not available in vSphere Client. I use the following one-liner scrips to determine and change its value via PowerCLI:

# To check the current status
Get-VMHost | Get-AdvancedSetting -Name “VSAN.ClomEnableInplaceExpansion” | select Entity, Name, Value | Format-Table -AutoSize

# To disable the in-place expansion
Get-VMHost | Get-AdvancedSetting -Name “VSAN.ClomEnableInplaceExpansion” | ? {$_.Value -eq “1”} | Set-AdvancedSetting -Value “0”

Note: No reboot is required after the change.

After hosts were upgraded to ESXi650-201810001 or ESXi650-201811002, you can set VSAN.ClomEnableInplaceExpansion back to ‘1‘ to enable the in-place expansion.

Windows Installer: MSI installation fails with the error status 1603

During the process of distributing an MSI package to the remote Windows Server 2012 R2 hosts via the Start-Process cmdlet, I ran across an interesting behaviour. In some cases, that MSI package was installed without any issues; in others, it was failing silently generating an event ID 10837 in the Application log.

With the verbose logging enabled, the following error message was observed in the MSI log file:

Installation success or error status: 1603.

The error status 1603 is documented on Microsoft Technet. However, none of those scenarios listed in the article applied to my case. I was able to install that MSI package locally with no issues, and the error popped up randomly when doing installation via PowerShell.

With more testing, I have realised the issue was only popping up when the user account, from which the script was running, had never previously log on to the target system.

I asked one of my colleagues, who has a better understanding of how Windows Installer works, to help with this case. After a thorough investigation, he pointed me to the following lines in the MSI log file:

MSI (s) (2C:C4) [02:22:15:584]: SECREPAIR: New Hash Database creation complete.
MSI (s) (2C:C4) [02:22:15:651]: SECREPAIR: CryptAcquireContext: Could not create the default key container
MSI (s) (2C:C4) [02:22:15:651]: SECREPAIR: Crypt Provider not initialized. Error:-2146892987

MSI (s) (2C:C4) [02:22:15:651]: SECUREREPAIR: Failed to CreateContentHash of the file: installer.msi: for computing its hash. Error: -2146892987
MSI (s) (2C:C4) [02:22:15:651]: SECREPAIR: Failed to create hash for the install source files
MSI (s) (2C:C4) [02:22:15:651]: Note: 1: 2262 2: SourceHash 3: -2147287038
MSI (s) (2C:C4) [02:22:15:651]: SECUREREPAIR: SecureRepair Failed. Error code: 8009034524E29A18
Action start 2:22:15: ProcessComponents.
The requested operation cannot be completed. The computer must be trusted for delegation and the current user account must be configured to allow delegation.

Apparently, in 2014 Microsoft released a security bulletin MS14-049 containing a patch to fix a vulnerability in the Windows Installer service. However, after you install this security update it breaks the MSI package installation. This is documented as a ‘Known issue 1’ in the bulletin and explained in more details here.

To resolve this issue, Microsoft recommends installing update 3000988.

Another option, which is documented in the same bulletin under the ‘Known issue 2’ section, is to opt-out the affected programs by using registry settings. However, this workaround implies more manual work and removes the defence-in-depth security feature for those programs.

I have tested those options and can confirm they both working. Hope this article saves you some time with troubleshooting a similar problem.