Info2soft use cookies to help you have a superior and more admissible browsing experience on our website. Privacy Policy
Loading...
By:
A 503 Service Unavailable error in VMware does not usually mean your virtual machines are down. In most cases, the VMs keep running while the management layer stops responding. The real problem is the lockout: without vCenter Server or the ESXi Host Client, you lose the tools you need to migrate VMs, apply patches, or run backup jobs.
The message itself varies, from “Failed to connect to endpoint” in vCenter to a bare 503 page in the ESXi Host Client. Behind it, the cause is often a stopped service, a full partition, an expired certificate, DNS or time drift, or a change made during an upgrade. Since each cause needs a different fix, it is best to identify the failing layer first.
A 503 Service Unavailable response means the server is temporarily unable to handle a request. In vCenter Server, this usually happens when the reverse proxy cannot reach a required backend service, such as vmware-vpxd. The exact message shows which layer failed.
The proxy layer also differs by product and version. vCenter Server 7.x and 8.x use the rhttpproxy service together with Envoy, and ESXi 8 uses Envoy as well. This is why the same type of failure can appear under different messages.
Each message points to a different failure point, so match the message and its location before restarting anything.
| Error message | Where it appears | Likely cause |
|---|---|---|
Failed to connect to endpoint ... vpxd-webserver-pipe |
vCenter UI | Backend service stopped |
no healthy upstream |
vCenter or ESXi 8 UI | Backend service or certificate failure |
_port = 8309 |
ESXi Host Client | hostd not responding, often after certificate changes |
/sdk endpoint 503 |
Backup jobs | vpxd not responding or overloaded |
The vpxd-webserver-pipe message usually means a backend service behind the proxy is down. The vCenter no healthy upstream error is an alternate 503 message, and Broadcom documents expired certificates as a cause of both.
For ESXi, a 503 containing _port = 8309 usually means hostd is not running or not responding. Invalid certificate files are a documented cause, but an overloaded host or a storage problem can produce the same result. On ESXi 8, the Host Client may show no healthy upstream instead.
When the error appeared, and how widely, shows where to start.
| Symptom | Likely cause | Where to start |
|---|---|---|
| 503 after reboot | Services still starting | Wait and Retry |
| 503 after certificate change | Expired or invalid certificate | Check Certificates for vCenter, or Regenerate ESXi Host Certificates for ESXi |
| 503 after upgrade | Stopped service or crashing hostd |
Check Service Status for vCenter, or If hostd Crashes After an ESXi Upgrade for ESXi |
| One interface only | One service or endpoint affected | Determine Whether the Problem Is vCenter or the Client |
| Backup job returns 503 | vpxd not responding or overloaded |
Check the vCenter SDK Endpoint |
Restarting services alone will not fix a full partition, an expired certificate, or a DNS problem, so match the symptom first. On ESXi, a Host Client 503 points to the management layer rather than the VMs, and running VMs are often unaffected. Storage problems are the main exception, because they can affect hostd and the VMs at the same time.
Start with the least invasive checks and work toward certificates, DNS, and time settings. The goal is to find the failed layer before restarting anything else.
A vCenter Server Appliance (VCSA) can take several minutes to bring all services online after a reboot. If the 503 appeared right after startup, wait 10 to 20 minutes and test the vSphere Client again.
Try a different browser or client as well. A 503 is a server-side error, so this mainly rules out a cached page or a stale session.
If the 503 persists, check service state from the VCSA shell. If the VAMI on port 5480 loads, its Services page shows the same information.
From the VCSA shell, run:
service-control --status --all
The output lists services under Running and Stopped. Focus on core services such as vmware-vpxd, vmware-stsd, vmware-rhttpproxy, and vsphere-ui. If one is stopped, start that service rather than restarting everything.
To start a stopped service:
service-control --start
If vmware-vpxd fails to start, check that the services it relies on, such as vmware-vpostgres and vmware-stsd, are running. Then read /var/log/vmware/vpxd/vpxd.log for the cause. Repeated restarts without a known cause often end in the same failure.
When several core services are stopped, restarting all of them can be appropriate once the underlying cause is fixed. Broadcom documents this command:
service-control --stop --all && service-control --start --all
A full partition can stop services from starting. Broadcom notes that disk space problems do not cause the 503 directly, but the services that go down as a result do.
From the VCSA shell, run:
df -h
Look for partitions at or near 100% usage. Logs and database growth are common causes, and the fix depends on which partition is full. For log-related cases, see log disk exhaustion on vCenter.
CPU and memory come second. Broadcom lists very high CPU contention and very high storage latency among the causes of stopped core services. Also confirm that the VCSA still has the memory its deployment size requires. For example, a Tiny deployment needs 12 GB in vCenter 7.0 and 14 GB in vCenter 8.0.
Expired vCenter certificates can make several services fail at once and produce both 503 Service Unavailable and no healthy upstream errors. Broadcom KB 344201 documents this for vCenter Server 6.x, 7.x, 8.x, and 9.x.
Before changing certificates, take a backup or a powered-off snapshot of the vCenter VM. A failed replacement can leave vCenter inaccessible.
From the VCSA shell, check certificate expiration:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do
echo "[*] Store :" $store
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After"
done
If any certificate has expired, Broadcom points to the vCert tool. Option 1 identifies expired certificates, and Option 6 resets all of them when several stores are affected. Restart the services afterward.
Name resolution and clock drift can stop vCenter services from starting or break trust between them. Broadcom lists time synchronization problems among the causes of stopped core services, and a modified /etc/hosts file can also prevent services from starting.
Check these items from the VCSA shell:
nslookup./etc/hosts for unexpected changes.date with the ESXi host running the VCSA and with your NTP source.Time drift between vCenter and ESXi hosts also causes the VMware cannot synchronize host error.
The VAMI runs on port 5480 and depends on the applmgmt service. Broadcom KB 378577 identifies a disabled or stopped applmgmt as a common cause of a VAMI 503 on vCenter Server 6.7, 7.0, and 8.0. If the vSphere Client loads but the VAMI does not, check this service first.
From the VCSA shell, check its status:
service-control --status applmgmt
If the service is stopped, start it and reload the VAMI:
service-control --start applmgmt
If applmgmt fails to start, check certificate expiration before anything else, because expired certificates can block service startup. Then review the vCenter logs for errors related to the service.
An ESXi 503 usually points to the host management plane, not the virtual machines. Start with hostd and vpxa, then check certificates if the error appeared after a certificate change. After an ESXi upgrade, a persistent hostd crash can also point to stale DVS configuration.
hostd handles core ESXi management operations, while vpxa handles communication between the host and vCenter Server. If either service is stopped or unresponsive, the Host Client or the vCenter connection can fail while the running VMs keep working.
Check the services before restarting them. This shows whether the 503 comes from a stopped service or a deeper configuration problem.
From ESXi SSH, check hostd:
/etc/init.d/hostd status
From ESXi SSH, check vpxa:
/etc/init.d/vpxa status
If hostd is stopped or unresponsive, restarting it often restores access to the Host Client. If the host is managed by vCenter, check vpxa too.
From ESXi SSH, restart the management services:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
Restarting these services does not power off running VMs, but it briefly interrupts management operations and communication with vCenter. If they stop again right away, do not keep restarting them. Check /var/run/log/hostd.log and /var/run/log/vpxa.log for the underlying cause instead.
_port = 8309An ESXi Host Client 503 containing _port = 8309 after a certificate update usually points to a certificate or SSL key problem. Broadcom KB 376026 documents this: an invalid certificate and key pair uploaded through the vSphere Client or the CLI gets applied even though it should be refused, and this crashes hostd and vpxa.
Check the certificate files before replacing them. The relevant files are /etc/vmware/ssl/rui.crt and /etc/vmware/ssl/rui.key.
If the hostd log shows a certificate parsing error or a key and certificate mismatch, follow the certificate recovery procedure below rather than repeatedly restarting the management agents.
When the existing certificate files are invalid or mismatched, Broadcom KB 376026 covers moving the affected files to a backup directory and generating a new self-signed certificate.
Before changing the certificate files, confirm the host is in an appropriate maintenance state and that the change will not disrupt required management connections. If the host is managed by vCenter, coordinate the change with the vSphere administrator.
From ESXi SSH, back up the existing certificate files:
cd /etc/vmware/ssl/
mkdir backup
mv rui.crt ./backup/
mv rui.key ./backup/
From ESXi SSH, generate a new certificate:
/sbin/generate-certificates
Then restart the management agents:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
The generated certificate is self-signed. If the host previously used a CA-signed certificate, reconfigure it once management access is restored.
hostd Crashes After an ESXi UpgradeA persistent hostd crash after an upgrade can have a different cause than a certificate failure. Broadcom KB 412202 documents this for hosts upgraded from ESXi 7.0 Update 3 to ESXi 8.0 Update 3: leftover references to physically removed NICs in the host config store no longer match the active DVS configuration, and this crashes hostd and leaves the Host Client showing HTTP 503 - no healthy upstream.
The affected host can keep crashing hostd even after a manual restart.
Check the hostd log for DVS-related errors before changing the network configuration. If the log points to stale vmnic entries or a config store mismatch, follow the procedure in Broadcom KB 412202.
Do not remove DVS configuration or physical uplinks as a first response to every ESXi 503. Incorrect network changes can affect management connectivity and VM networking, so this fix is best handled by a vSphere administrator after the DVS mismatch is confirmed.
Most VMware 503 errors are easier to prevent than to recover from once the management plane goes down. The same dependencies show up again and again: service health, disk capacity, certificates, and time synchronization. Broadcom lists full disk space, resource contention, and service dependencies among the common causes of vCenter service startup failures.
Regular checks can catch most of these conditions before they turn into a management outage.
503 Service Unavailable and no healthy upstream errors and prevent required services from starting.A 503 error does not put running VMs at risk by itself, but the troubleshooting steps above do. Restarting services, replacing certificates, and reconfiguring a DVS all touch systems that keep production VMs running. A backup taken before these changes gives you a way back if something goes wrong.
i2Backup is built for this kind of protection in VMware environments:
hostd or vpxa.For businesses running vSphere clusters at scale, pairing i2Backup with i2CDM adds fast disaster recovery drills, so a management-plane failure can be tested and rehearsed before it happens for real.
A VMware 503 error almost always traces back to one of a few places: a stopped service, a full partition, an expired certificate, a DNS or time mismatch, or a configuration issue left over from an upgrade. Matching the exact error message and the symptom to the right section above is the fastest way to narrow down which one applies.
Once the immediate error is fixed, the more useful step is making sure the same failure does not repeat. Regular checks on service health, disk space, certificates, and time sync catch most of these problems before they turn into a 503 page.
Info2soft builds data protection tools for VMware and other enterprise environments, including the backup and recovery options covered above.
· Enterprise & Mid-market Customers Worldwide
· Support team available to assist you throughout your trial
· Start a 60-day free trial or view demo to see how Info2Soft protects enterprise data.
