vCenter 6.0: VMware Common Logging Service Health Alarm [RESOLVED]

A few days ago I noticed a warning message appearing in vCenter Server pointing to some issues with the VMware Common Logging Service.

VCLS Issue - 01

The service status was showing that the disk space had been filling in steadily reaching the 30% warning threshold.

VCLS Issue - 00

VCLS Issue - 02

Considering the infrastructure had not experienced significant changes, I decided to postpone disk space extension and try to find the root cause of this problem.

With the help of the du command, it has become clear some of the subfolders in /var/log/vmware were quite large in size.

VCLS Issue - 03

It shouldn’t be a problem if the log rotation happens and old data is removed from the disk. However, the /storage/log/vmware/cloudvm/cloudvm-ram-size.log file size was 1.4G, and it seemed to be increasing without log rotation.

VCLS Issue - 04

An attempt to find out about the cloudvm-ram-size.log file pointed me to the article which William Lam wrote in early 2015 – apparently it logs activities of a built-in dynamic memory reconfiguration process called cloudvm-ram-size.

The issue is documented in ‘Log rotation of the cloudvm-ram-size.log file is not working (2147261)‘ in VMware Knowledge Base.

As per that article, the problem is fixed in vSphere 6.5. For vSphere 6.0, you need to configure log rotation for the cloudvm-ram-size.log file and run the logrotate command manually to archive it to cloudvm-ram-size.log-xxxxxxxxx-.bz2 file.

VCLS Issue - 05

VMware recommends to do periodic cleanup of older .bz2 files in the /storage/log/vmware/cloudvm location!!! This can be done by adding a rotate parameter to the configuration file as follows:

size 20k
create 0660 root cis
rotate 7

It was a quick fix!

vSAN 6.6.1: vSAN Build Recommendation Engine Health issue [RESOLVED]

In my previous post about vSAN Build Recommendation Engine Health test, I have concluded that it was a bug in vSAN 6.6.1 that prevented vSAN Health service from properly connecting to the Internet via proxy.

With vCenter Server Appliance 6.5 Update 1d release, I have noticed that one of two warning messages disappeared from the vSphere Web Client leaving that task in the ‘Unexpected vSphere Update Manager (VUM) baseline creation failure‘ state.

After checking vSAN configuration one more, I concluded the following:

  • Internet connectivity for automatic updates of the HCL database has been set up properly (vSAN_Cluster > Configure > vSAN > General):


  • The HCL database is up-to-date and CEIP is enabled (vSAN_Cluster > Configure > vSAN > Health and Performance):



  • Update Manager has proxy settings configured and working (vSAN_Cluster > Update Manager > Go to Admin View > Manage > Settings > Download Settings):



At the same time, the proxy server replaces SSL certificates with its own one signed by the corporate CA when establishing HTTPS connection with the remote peer.

As a result, it causes an error message for the vSAN Build Recommendation Engine Health task as follows (extract from vmware-vsan-health-service.log):

INFO vsan-health[Thread-49] [VsanVumConnection::RemediateVsanClusterInVum] build = {u’release’: {u’baselineName’: u’VMware ESXi 6.5.0 U1 (build 5969303)’, u’isoDisplayName’: u’VMware ESXi Release 6.5.0, Build 5969303′, u’bldnum’: 5969303, u’vcVersion’: [u’6.5.0′], u’patchids’: [u’ESXi650-Update01′], u’patchDisplayName’: u’VMware ESXi 6.5.0 U1 (vSAN 6.6.1, build 5969303)’}}

INFO vsan-health[Thread-49] [VsanVumConnection::_LookupPatchBaseline] Looking up baseline for patch VMware ESXi 6.5.0 U1 (vSAN 6.6.1, build 5969303) (keys: [40], hash: None)…

INFO vsan-health[Thread-49] [VsanVumConnection::_LookupPatchBaseline] Looking up baseline for patch vSAN recommended patch to be applied on top of ESXi 6.5 U1: ESXi650-201712401-BG (keys: [], hash: None)…

ERROR vsan-health[Thread-49] [VsanVumConnection::RemediateAllClusters] Failed to remediate cluster ‘vim.ClusterComputeResource:domain-c61’

Traceback (most recent call last):
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 1061, in RemediateAllClusters
performScan = performScan)
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 876, in RemediateVsanClusterInVum
patchName, patchMap[chosenRelease])
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 373, in CreateBaselineFromOfficialPatches
baseline = self._LookupPatchBaseline(name, keys)
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 411, in _LookupPatchBaseline
result = bm.QueryBaselinesForUpdate(update = updateKeys)
File “/usr/lib/vmware-vpx/pyJack/pyVmomi/”, line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File “/usr/lib/vmware-vpx/pyJack/pyVmomi/”, line 362, in _InvokeMethod
list(map(CheckField, info.params, args))
File “/usr/lib/vmware-vpx/pyJack/pyVmomi/”, line 883, in CheckField
raise TypeError(‘Required field “%s” not provided (not @optional)’ %
TypeError: Required field “update” not provided (not @optional)

INFO vsan-health[Thread-49] [VsanVumSystemUtil::AddConfigIssue] Add config issue createBaselineFailed

INFO vsan-health[Thread-49] [VsanVumConnection::_DeleteUnusedBaselines] Deleting baseline VMware ESXi 6.5.0 U1 (vSAN 6.6.1, build 5969303) (id 424) because it is unused

INFO vsan-health[Thread-49] [VsanVumSystemUtil::VumRemediateAllClusters_DoWork] Complete VUM check for clusters [‘vim.ClusterComputeResource:domain-c61’]

ERROR vsan-health[Thread-49] [VsanVumConnection::RemediateAllClusters] Failed to remediate cluster

Following the community advice, I decided to add Root CA and subordinate CA certificates (in *.pem format) to the local keystore on vCenter Server Appliance. After copying certificates to /etc/ssl/certs and running the c_rehash command, I added proxy servers to /etc/sysconfig/proxy and rebooted the server.


To test that new configuration works, I used the wget command, and it all seemed to work smoothly.


Regardless of all that changes, I still got error messages with the vSAN Build Recommendation Engine Health test, but this time they looked a bit different:

INFO vsan-health[Thread-11125] [VsanPyVmomiProfiler::InvokeAccessor] Invoke: mo=ServiceInstance, info=content

WARNING vsan-health[Thread-11125] [VsanPhoneHomeWrapperImpl::_try_connect] Cannot connect to VUM. Will retry connection

INFO vsan-health[Thread-11125] [VsanPyVmomiProfiler::InvokeAccessor] Invoke: mo=group-d1, info=name

INFO vsan-health[Thread-11125] [VsanPyVmomiProfiler::InvokeAccessor] Invoke: mo=group-d1, info=name

ERROR vsan-health[Thread-11125] [VsanCloudHealthDaemon::run] VsanCloudHealthSenderThread exception: Exception: HTTP Error 411: Length Required, Url:<Support_Tag&gt;, Traceback: Traceback (most recent call last):
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 511, in getResponse
resp =*args, **kwargs)
File “/usr/lib/python2.7/”, line 435, in open
response = meth(req, response)
File “/usr/lib/python2.7/”, line 548, in http_response
‘http’, request, response, code, msg, hdrs)
File “/usr/lib/python2.7/”, line 473, in error
return self._call_chain(*args)
File “/usr/lib/python2.7/”, line 407, in _call_chain
result = func(*args)
File “/usr/lib/python2.7/”, line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 411: Length Required
Traceback (most recent call last):
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 353, in run
self._sendCloudHealthData(clusterUuid, data=data)
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 321, in _sendCloudHealthData
objectId=clusterUuid, additionalUrlParams=additionalUrlParams)
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 156, in send
dataType=dataType, pluginType=pluginType, url=postUrl)
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 139, in sendRawData
raise ex
VsanCloudHealthHTTPException: Exception: HTTP Error 411: Length Required, Url:<Support_Tag&gt;, Traceback: Traceback (most recent call last):
File “/usr/lib/vmware-vpx/vsan-health/pyMoVsan/”, line 511, in getResponse
resp =*args, **kwargs)
File “/usr/lib/python2.7/”, line 435, in open
response = meth(req, response)
File “/usr/lib/python2.7/”, line 548, in http_response
‘http’, request, response, code, msg, hdrs)
File “/usr/lib/python2.7/”, line 473, in error
return self._call_chain(*args)
File “/usr/lib/python2.7/”, line 407, in _call_chain
result = func(*args)
File “/usr/lib/python2.7/”, line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 411: Length Required

INFO vsan-health[Thread-11125] [VsanCloudHealthDaemon::run] VsanCloudHealthSenderThread done.

vsan-health[Thread-9] [VsanCloudHealthDaemon::_sendExceptionsToPhoneHome] Exceptions for collection/sending exceptions

I thought that the vSAN Health service might try to contact vSphere Update Manager directly, and the proxy settings set on the OS level redirected this request to the Internet proxy instead.

I have added the local domain to the exception list in /etc/sysconfig/proxy and rebooted the server again.


After reading about ‘HTTP Error 411’, the only idea was to add a domain service account and its password to HTTP_PROXY and HTTPS_PROXY lines in /etc/sysconfig/proxy. If the password has special characters, they should be added in ASCII encoding to work correctly.

To my great surprise, all communication issues have been resolved, and the vSAN Health service was able to synchronise data with vSphere Update Manager and online services correctly.



A few minutes later vSAN system baselines and baseline groups appeared in vSphere Update Manager.


Of cause, those modifications in Photon OS configuration files are not supported by VMware and could be overwritten by future updates. Yet I hope engineers and developers are working on better integration between vSAN Health and vSphere Update Manager when vCenter resides behind the proxy.

23/02/2018 – Update 1: Per VMware documentation, a starting point to troubleshoot connectivity to the CEIP web server is to make sure the following prerequisites are met:

vSphere 6.0: Templates are shown as ‘Unknown’ in the local Content Library

Another day another case… This time, I was surprised to see an empty list when provisioning a new virtual machine from a Content Library.

CL Issue - 01

I went to check the Content Library status and found all templates were shown as ‘Unknown’ in there.

CL Issue - 02

Funny enough, this behaviour was happening only with the local Content Library. A subscribed one didn’t have any issues at all, and the synchronisation between those two was still working.

CL Issue - 03

More interestingly, the objects of other types were not affected at all.

There is not enough information about how to troubleshoot the Content Library in vSphere 6.0. Some of the diagnostic files can be found in the /var/log/vmware/vdcs directory on vCenter Server Appliance (VCSA). Unfortunately, they are not that informative.

So I opened the case with VMware GSS (SR # 17504701707) and the response was that “this issue is occurring as there is a corrupted or stale PID for the content library service which has not been cleared from the previous running state.”

VMware is working on this to be resolved, but no ETA at the moment.

A workaround provided by VMware:

  1. Connect to the vCenter Server Appliance using SSH and root credentials.
  2. Navigate to /var/log/vmware/vdcs.
  3. Create a new folder to move the PID file to.
  4. Move the file to the folder created in step 3.
  5. Reboot the vCenter Server Appliance (In case of external PSC, reboot the PSC first and then the vCenter).

I personally found that restarting VCSA resolves this issue. However, it reappears after some time.

VCSA 6.5: The mysterious dependency on the IPv6 protocol – Part 2

In Part 1 of this mini-series, I was writing about the issue with the Appliance Management User Interface. However, a dependency on the IPv6 protocol in VCSA 6.5 can cause an unexpected behaviour with the vSphere ESXi Dump Collector service as well. Let’s look into this one now.

In the environment with many ESXi hosts, it is vital to have their logs available for troubleshooting. By default, each host has a diagnostic coredump partition available on the local storage. The hypervisor can preserve diagnostic information in one or more pre-configured locations such as the local partition, a file located on VMFS datastore, or a network dump server on vCenter Server.


In a case of the critical failure with the host, when the system gets into the Purple Screen of Death (PSOD) state, the hypervisor generates a set of diagnostic data archived in a coredump. In my opinion, it is more efficient to have this information stored in the centralised location, and this is where vSphere ESXi Dump Collector service can be useful.

Initially, the vSphere ESXi Dump Collector service is disabled on the vCenter Server Appliance.


The setup process is straightforward: you should select a startup type of this service (by default, it is set to Manual) and click on a Start button to enable it.


Depending on the network requirements and the number of ESXi hosts, you might need changing the Coredump Server UDP Port (6500) and increasing the Repository max size (2GB). Both settings require restarting the vSphere ESXi Dump Collector service.

This process becomes a little bit complicated when IPv6 is disabled on VCSA. An attempt to start the vSphere ESXi Dump Collector service generates an error message in vSphere Web Client as follows:


If we remote to the virtual appliance and run the netdumper service from the console session, it will show us more information:

root@n-vcsa-01 [ ~ ]# service-control –start netdumper
Perform start operation. vmon_profile=None, svc_names=[‘netdumper’], include_coreossvcs=False, include_leafossvcs=False
2017-07-04T10:15:32.179Z Service netdumper state STOPPED
Error executing start on service netdumper. Details {
“resolution”: null,
“detail”: [
“args”: [
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘netdumper'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
“componentKey”: null,
“problemId”: null
Service-control failed. Error {
“resolution”: null,
“detail”: [
“args”: [
“id”: “install.ciscommon.service.failstart”,
“localized”: “An error occurred while starting service ‘netdumper'”,
“translatable”: “An error occurred while starting service ‘%(0)s'”
“componentKey”: null,
“problemId”: null

The next step to troubleshoot this issue is to look into the vSphere ESXi Dump Collector service log file (/var/log/vmware/netdumper/netdumper.log). It reports that the address is already in use:

root@n-vcsa-01 [ ~ ]# cat /var/log/vmware/netdumper/netdumper.log
2017-07-04T10:19:32.121Z| netdumper| I125: Log for vmware-netdumper pid=8347 version=XXX build=build-5318154 option=Release
2017-07-04T10:19:32.121Z| netdumper| I125: The process is 64-bit.
2017-07-04T10:19:32.121Z| netdumper| I125: Host codepage=UTF-8 encoding=UTF-8
2017-07-04T10:19:32.121Z| netdumper| I125: Host is Linux 4.4.8 VMware Photon 1.0 Photon VMware Photon 1.0

2017-07-04T10:19:32.123Z| netdumper| I125: Configured to handle 1024 clients in parallel.
2017-07-04T10:19:32.123Z| netdumper| I125: Configuring /var/core/netdumps as the directory to store the cores
2017-07-04T10:19:32.123Z| netdumper| I125: Configured to use wildcard [::0/]:6500 as IP address:port
2017-07-04T10:19:32.123Z| netdumper| I125: Using /var/log/vmware/netdumper/netdumper.log as the logfile.
2017-07-04T10:19:32.123Z| netdumper| I125: Nothing to post process
2017-07-04T10:19:32.123Z| netdumper| I125: Couldn’t bind socket to port 6500: 98 Address already in use
2017-07-04T10:19:32.123Z| netdumper| I125:

Playing a bit with the Linux commands gave me some clues:

root@n-vcsa-01 [ ~ ]# netstat -lup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
udp 0 0 *:kerberos *:* 1489/vmdird
udp 0 0 *:sunrpc *:* 1062/rpcbind
udp 0 0 n-vcsa-01.testorg.l:ntp *:* 1249/ntpd
udp 0 0 photon-machine:ntp *:* 1249/ntpd
udp 0 0 *:ntp *:* 1249/ntpd
udp 0 0 *:epmap *:* 1388/dcerpcd
udp 0 0 *:syslog *:* 2229/rsyslogd
udp 0 0 *:794 *:* 1062/rpcbind
udp 0 0 *:ideafarm-door *:* 3905/vpxd
udp 0 0 *:llmnr *:* 1223/systemd-resolv
udp6 0 0 [::]:tftp [::]:* 1/systemd
udp6 0 0 [::]:sunrpc [::]:* 1062/rpcbind
udp6 0 0 [::]:ntp [::]:* 1249/ntpd
udp6 0 0 [::]:syslog [::]:* 2229/rsyslogd
udp6 0 0 [::]:794 [::]:* 1062/rpcbind
udp6 0 0 [::]:boks [::]:* 17377/vmware-netdum

root@n-vcsa-01 [ ~ ]# ps -p 17377
17377 ? 00:00:00 vmware-netdumpe

root@n-vcsa-01 [ ~ ]# cat /proc/17377/cmdline

Even if it reports an error at startup, the vSphere ESXi Dump Collector service is running (partially) on the virtual appliance.

Thanks to Michael (for sharing a detailed guide), I was able to test this assumption quickly.



The coredump was successfully transferred from the ESXi host to the /var/core/netdumps/ folder on the VCSA appliance. However, there were no records about this operation in the netdumper.log.

This issue has been reported to VMware GSS (SR # 17385781602) and should be resolved in the future updates to VCSA 6.5.

VCSA 6.5: The mysterious dependency on the IPv6 protocol – Part 1

Starting from vSphere 4.1, IPv6 support has been introduced to the virtual platform from VMware. It is enabled in the vCenter Server Appliance by default and can be controlled in VCSA 6.0 and 6.5 from the Direct Console User Interface (Customize System > Configure Management Network > IPv6 Configuration).


To my surprise, disabling IPv6 can cause some problems with the VCSA updates. I will explain this statement and provide a workaround in the paragraphs below.

Imagine your security team requires IPv6 to be turned off on vCenter Server. Following this call, you proceeded with the configuration change in DCUI.


After rebooting the virtual machine, it all should work fine. Now, it is time to update the virtual appliance to a newer version. You downloaded a patch file, attached it to the VM, and started the update process from the VMware vSphere Appliance Management Interface.

When the server reboots, you will notice the Appliance Management User Interface is not accessible anymore. To troubleshoot this issue further, we need to open SSH session with the appliance and enable Shell mode.

Firstly, we need to netstat command to see if any service is listening on TCP port 5480. The command output does not show anything.


The next step is to identify the service which provides the Appliance MUI and its current status. Fortunately, I have noticed an error message which is related to the problem when the operating system is booting up.


Querying the vami-lighttp.service status shows the following results.


So it is a duplicate parameter server.use-ipv6 in the configuration file which was causing this behaviour. To find this file, I was using a combination of rpm and egrep commands to filter the output.


A quick search in /opt/vmware/etc/lighttpd/lighttpd.conf shows that there are two identical lines with IPv6 settings as follows:


To fix this issue, I removed one of the lines, started the vami-lighttp.service and checked that the service works as expected.


To be continued…

vSphere 6.5 GA: VMware-VMRC.exe – Failed to install hcmon driver.

After upgrading the vCenter Server Appliance to version 6.5, I needed to install a new version of VMware Remote Console 9.0 on my Windows 10 machine.

VMware-VMRC.msi was downloaded from the vCenter Server, and I initiated its installation.


To my surprise, this task ended up with an error message below.


I immediately searched on VMware for any explanation and found KB # 2130850. Despite the workaround provided, I haven’t had vSphere Client installed on the computer.

Quickly checking the list of VMware products available, I was able to identify the package which caused the problem. It was a VMware Remote Console Plug-in 5.1 from the previous version of vSphere which prevented the installer from doing its job. Removing the old piece of software completely resolved the obstacle for my environment. Easy-peasy!

vCenter Server 6.0: vmware-dataservice-sca and vsphere-client status change from green to yellow


Some of you might be aware of the behaviour in vCenter Server 6.0 when the vmware-dataservice-sca and vsphere-client status change from green to yellow continually, and VMware KB # 2144950 provided as a workaround for this one.

However, two questions should be explained in the knowledge base above but seem to be missing.

The first question is current memory usage by the vsphere-client service. Knowing this data will help to choose the correct value for the vSphere Web Client’s maximum heap size when setting it manually. Fortunately, William Lam has a great article that explains a dynamic memory reconfiguration process when vCenter Server is booting up. One of the suggestions that William had was to use a CLI utility cloudvm-ram-size to monitor the memory usage.

cloudvm-ram-size -S | grep -e Service-* -e Linux* -e OS -e vsphere-client -e TOTAL

In the picture below, an output of the command shows memory usage in MB for the vCenter Server with external PSC in a small environment.


As a rule of thumb, to determine the maximum heap size for the vsphere-client service, I usually round the MaxMB value to the nearest gigabytes and add extra 512 MB as a reserve. In this example, AllocatedMB is 2,048 Mb + 512 MB = 2,560 MB.

Then, I set this parameter manually and restarted the vSphere Web Client service using the commands below.

cloudvm-ram-size -C 2560 vsphere-client | service vsphere-client restart

The dynamic memory algorithm adjusts the value of this setting automatically. After few minutes the service reinitialises, and memory allocation looks much better.


On rare occasions, you might notice that vAPI Endpoint service generate error messages after restarting vsphere-client. Restarting this service helps to resolve the problem.

service vmware-vapi-endpoint restart

Now we come to the second question: does this setting change survive the vCenter Server reboots? The answer is yes! And this is great news.

08/12/2016 – Update 1: You should reapply this setting after VCSA has been updated.

19/07/2017 – Update 2: VMware released a KB 2150757 to guide through the process of manually changing the heap memory on vCenter Server components in vCenter 6.x.