Although you´ve done as much as you can do (or afford) to avoid a virtual machine crash, it can always happen. Sometimes you´re storage system goes down, sometime your HBA breaks or the fabric in between. Many misconfigurations lead to a non-redundant situation you find out about when it goes down. Therefore you should run an vSphere Health Analyzer Test to make sure.
If your VMware vCenter Server Appliance crashes and the VCSA filesystem is affected (especially VCSA 6, VMware vSphere 6) , it might happen that the system is not recovering anymore. You end up with a nice console message telling you that the filesystem is corrupt and you can try to recover it manually. You need to know that VCSA 6 is using a LVM (logical volume manager), so for most system administrators with little linux experience it can be pretty hard to troubleshoot.
You can get a better understanding when reading Cormac Hogan’s Post about fsck on VCSA 6.0 and the lvm partitions.
fsck failed. Please repair manually and reboot. The root**file system is currently mounted read-only. To remount it read-write do:**
bash# mount -n -o remount,rw
Actually that is not so simple when using VCSA as the shell (bash) is not enabled by default and you don’t have a chance to do so without changing the grub boot entry.
Within the grub boot menu you need to press p to get into the boot options, using vmware as the default password or your own password configured for the root account.
Select VMware vCenter Server Appliance and hit e to edit the grub entry.
Select the second entry that starts with kernel and hit e again to edit that.
Change the line and append init=/bin/bash and hit enter to enable the bash shell after the OS start.
Press b to boot the changed grub entry
You should see a command prompt to enter
mount -n -o remount,rw
fsck -f -c -y
That starts fixing all filesystem errors. After a while you should get some messages about VCSA filesystem issues you need to confirm to fix them. Therefore it always makes sense to create a VM snapshot before starting the filesystem fix.
When all is done, the command reboot is all you need and VCSA should start without any issues.
You can read more about VMware vCenter Server Appliance (VCSA) filesystem issues here:
