For those who have vCenter Server Appliance with an External Platform Services Controller, you might notice a warning message in Services Health area in Administration -> System Configuration -> Summary tab.
VMware Syslog Service reports a warning message as soon as /storage/log has less than 30 percent of free space, similar to what is in the picture below.
The problem appears to be with the VMDK disk for a /storage/log mount point. On PSC, it has a default size of 5 GB and is quickly filling in with the SSO log files.
VMware has two possible solutions to resolve this issue, as follows:
- Decrease the maximum backup size and the maximum backup index in the log4j.properties file – https://kb.vmware.com/kb/2143565, or
- Increase the disk space of a specific VMDK for the vCenter Server Appliance 6.x – https://kb.vmware.com/kb/2126276.
The second option sounds more preferable, as it eliminates the need to monitor changes in the log4j.properties file after a system update. However, the commands in the VMware KB 2126276 do not apply to the Platform Services Controller appliance. It doesn’t have a vpxd_servicecfg script to automate the volume extension.
Fortunately, Florian Grehl has documented a workaround for PSC, which requires us to extend the VMDK5 using the vSphere Web Client and execute the following commands in an SSH session on the affected server:
1. Rescan the SCSI Bus to make Linux aware of the resized virtual disk
# rescan-scsi-bus.sh -w –forcerescan
2. Change the size of the Volume Group by using the Disk Device from the table above
# pvresize /dev/sde
3. Resize the Logical Volume by using the name from the table above
# lvresize –resizefs -l +100%FREE /dev/log_vg/log
After completing the commands and verifying the volume size, we should restart VMware Syslog Service to refresh its state. It can be done from the same SSH session or using vSphere Web Client.
And this is how things are back to normal 🙂
VCSA 6.0 U2a – build 4541947 – Nov 22nd, 2016 finally added an official way /usr/lib/applmgmt/support/scripts/lvm_cfg.sh
https://vmscribble.com/vcenter/6-0-external-vcsa-vm_cfg-sh-permission-denied-vpxd_servicecfg-command-not-found/
LikeLike
Great! Thank you for sharing this information. Have you checked whether this script available on PSC node?
LikeLike
They ‘fixed’ this problem and replaced it with a brand new one. Syslog doesn’t exist in VCSA 6.5, so they fill up that mount with java dump files instead.
Large Java dump files created in the location /storage/log/vmware/perfcharts (2151394)
https://kb.vmware.com/s/article/2151394
LikeLiked by 1 person
Thank you mate! Interesting article.
LikeLike