{"id":3756,"date":"2022-03-15T12:05:56","date_gmt":"2022-03-15T12:05:56","guid":{"rendered":"https:\/\/www.edwardsd.co.uk\/work\/?p=3756"},"modified":"2022-03-15T12:09:34","modified_gmt":"2022-03-15T12:09:34","slug":"esx-7-0-2-503-service-unavailable","status":"publish","type":"post","link":"https:\/\/www.edwardsd.co.uk\/work\/2022\/03\/esx-7-0-2-503-service-unavailable\/","title":{"rendered":"VMware ESX 7.0.2 | 503 Service Unavailable"},"content":{"rendered":"<p>Issue with unresponsive ESX host (Running v7.0.2) in vCenter (Shows disconnected), all VMs are still running on the host and can access ESX UI directly (IP\/DNS).<\/p>\n<p>Trying to resolve I complete the following:<\/p>\n<ul>\n<li>iDRAC to ESX direct console\n<ul>\n<li>Enable SSH (This provides a backup to access with CLI)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>SSH to ESX console and restart management agents<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/hostd restart<\/pre>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/vpxa restart<\/pre>\n<p>After restarting the management agents the ESX GUI becomes inaccessible with the following error:&nbsp;<\/p>\n<p style=\"color: red;\"><em>503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000eb77f04cd0] _serverNamespace = \/ action = Allow _port = 8309)<\/em><\/p>\n<p>After some searching on the internet I find this vmware KB: <a href=\"https:\/\/kb.vmware.com\/s\/article\/2144962\">2144962<\/a> which looks promising however&#8230; there is <span style=\"text-decoration: underline;\"><strong>no<\/strong><\/span><span style=\"font-size: 1rem;\"> &#8220;<\/span><strong style=\"font-size: 1rem;\"><em>\/ui local 8308 redirect allow<\/em><\/strong><span style=\"font-size: 1rem;\">&#8221; in the endpoint.conf file so this doesn&#8217;t apply. I also compare the contents of &#8220;endpoints.conf&#8221; files with another ESX host and they are identical.<br \/>\n<\/span><\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">cat \/etc\/vmware\/rhttpproxy\/endpoints.conf<\/pre>\n<p><span style=\"font-size: 1rem;\"><strong>Double check that all services are running on the ESX host:<\/strong><\/span><\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/hostd status<\/pre>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/vpxa status<\/pre>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/rhttpproxy status<\/pre>\n<p>Both hostd &amp; vpxa status is &#8220;stopped&#8221; after starting this again and waiting the GUI finally displays (After a few minutes&#8230;.)<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/hostd start<\/pre>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\/etc\/init.d\/vpxa start<\/pre>\n<p>Investigating further with the use of &#8220;<a href=\"https:\/\/docs.vmware.com\/en\/VMware-vSphere\/7.0\/com.vmware.vsphere.monitoring.doc\/GUID-D89E8267-C74A-496F-B58E-19672CAB5A53.html\">esxtop<\/a>&#8221; it indicates that the CPU &amp; memory is heavily over commited which is what possibly prevented the &#8220;restart&#8221; services command from working correctly. The host is currently used at a branch office with only 3 low resource required VMs running. I&#8217;ve also seen this same behavior on other high performance ESX hosts which I believe is either related to the build &#8220;<a href=\"https:\/\/docs.vmware.com\/en\/VMware-vSphere\/7.0\/rn\/vsphere-esxi-70u2a-release-notes.html\">VMware ESXi, 7.0.2, 17867351<\/a>&#8221; or potentialy the &#8220;<a href=\"https:\/\/www.dell.com\/support\/home\/en-uk\/drivers\/driversdetails?driverid=74fgk&amp;oscode=xi70&amp;productcode=poweredge-r730\">Dell EMC iDRAC service module v4.2.0.0<\/a>&#8221;<\/p>\n<p><span style=\"font-size: 1rem;\">After finding a patching window to reboot the host and apply a later build &#8220;<a href=\"https:\/\/docs.vmware.com\/en\/VMware-vSphere\/7.0\/rn\/vsphere-esxi-70u2d-release-notes.html\">VMware ESXi, 7.0.2, 18538813<\/a>&#8221; I have yet to experience the same issues.<\/span><\/p>\n<p><strong>Note:<\/strong> I&#8217;ve also had the same issue whereby I was unable to SSH or GUI into the ESX. In this scenario I ended up having to reboot the host from the iDRAC. Fortunately this was at smaller site locations and RDP to the VMs was still available so they could be cleanly powered down through Windows before a hard reset of ESX.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Issue with unresponsive ESX host (Running v7.0.2) in vCenter (Shows disconnected), all VMs are still running on the host and can access ESX UI directly (IP\/DNS). Trying to resolve I complete the following: iDRAC to ESX direct console Enable SSH (This provides a backup to access with CLI) SSH to ESX console and restart management [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[58,33],"tags":[],"class_list":["post-3756","post","type-post","status-publish","format-standard","hentry","category-esx","category-vmware"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/pOPt8-YA","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/posts\/3756","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/comments?post=3756"}],"version-history":[{"count":6,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/posts\/3756\/revisions"}],"predecessor-version":[{"id":3762,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/posts\/3756\/revisions\/3762"}],"wp:attachment":[{"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/media?parent=3756"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/categories?post=3756"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.edwardsd.co.uk\/work\/wp-json\/wp\/v2\/tags?post=3756"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}