Issue with unresponsive ESX host (Running v7.0.2) in vCenter (Shows disconnected), all VMs are still running on the host and can access ESX UI directly (IP/DNS).

Trying to resolve I complete the following:

• iDRAC to ESX direct console
  • Enable SSH (This provides a backup to access with CLI)

SSH to ESX console and restart management agents

/etc/init.d/hostd restart

/etc/init.d/vpxa restart

After restarting the management agents the ESX GUI becomes inaccessible with the following error: 

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000eb77f04cd0] _serverNamespace = / action = Allow _port = 8309)

After some searching on the internet I find this vmware KB: 2144962 which looks promising however… there is no “/ui local 8308 redirect allow” in the endpoint.conf file so this doesn’t apply. I also compare the contents of “endpoints.conf” files with another ESX host and they are identical.

cat /etc/vmware/rhttpproxy/endpoints.conf

Double check that all services are running on the ESX host:

/etc/init.d/hostd status

/etc/init.d/vpxa status

/etc/init.d/rhttpproxy status

Both hostd & vpxa status is “stopped” after starting this again and waiting the GUI finally displays (After a few minutes….)

/etc/init.d/hostd start

/etc/init.d/vpxa start

Investigating further with the use of “esxtop” it indicates that the CPU & memory is heavily over commited which is what possibly prevented the “restart” services command from working correctly. The host is currently used at a branch office with only 3 low resource required VMs running. I’ve also seen this same behavior on other high performance ESX hosts which I believe is either related to the build “VMware ESXi, 7.0.2, 17867351” or potentialy the “Dell EMC iDRAC service module v4.2.0.0”

After finding a patching window to reboot the host and apply a later build “VMware ESXi, 7.0.2, 18538813” I have yet to experience the same issues.

Note: I’ve also had the same issue whereby I was unable to SSH or GUI into the ESX. In this scenario I ended up having to reboot the host from the iDRAC. Fortunately this was at smaller site locations and RDP to the VMs was still available so they could be cleanly powered down through Windows before a hard reset of ESX.