vCenter 5.5 Resource Exhaustion Detected

Following an upgrade of vCenter server from 5.0 to 5.5 the vCenter service intermittently stopped and we began to see a number of resource exhaustion events:

Event ID: 2004
Source: Resource-Exhaustion-Detect
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: java.exe (2116) consumed 10149273600 bytes, java.exe (2088) consumed 4624416768 bytes, and vpxd.exe (14440) consumed 4113379328 bytes.

We increased the allocated memory (from 12GB to 24GB) and page file (from 4Gb to 6GB) but continued to experience problems. I came across a similar issue in the VMware Communities in which user Sateesh_vcloud documented the standard JVM Heap settings:

Default values for vCenter server installation:

(null)

Our vCenter inventory was approx 100 hosts and 2000 virtual machines but we had selected large inventory for all services during the upgrade. Therefore the JVM heap allocated for each service was likely larger than we required. Sateesh_vcloud also documented the locations of the configuration files for each service:

Single Sign On:
C:\Program Files\VMware\Infrastructure\SSOServer\conf\wrapper.conf
Set wrapper.java.additional.9=”-Xmx” (default: “1024M”) to “256M”
Set wrapper.java.additional.14=”-XX:MaxPermSize=” (default: “512M”) to “128M” (or half of the Xmx value chosen before)

Inventory Service:
C:\Program Files\VMware\Infrastructure\Inventory Service\conf\wrapper.conf
Set wrapper.java.maxmemory (default: “3072”) to “384” (MB)

Tomcat:
C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf
Set wrapper.java.additional.9=”-Xmx” (default: “1024M”) to “512M” – “768M”
Set wrapper.java.additional.14=”-XX:MaxPermSize” (default: “256M”) to half of the Xmx value chosen before

Web Client:
C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
Set wrapper.java.initmemory (default: “1024m”) to “256m”
Set wrapper.java.maxmemory (default: “1024m”) to “384m”

Log Browser:
C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf
Set wrapper.java.maxmemory (default: “512”) to “256” (MB)

Profile Driven Storage:
C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf\wrapper.conf
Set wrapper.java.initmemory (default: “256”) to “128” (MB)
Set wrapper.java.maxmemory (default: “1024”) to “384” (MB)

Orchestrator:
C:\Program Files\VMware\Infrastructure\Orchestrator\app-server\bin\wrapper.conf
Set wrapper.java.additional.3=-Xmn (default: “768m”) to “256m”
Set wrapper.java.initmemory (default: “2048”) to “384” (MB)
Set wrapper.java.maxmemory (default: “2048”) to “512” (MB)

I updated the Inventory Service configuration file to 6144MB (previously 1288MB) and restarted the service. We have not had a reoccurrence of the resource exhaustion and the vCenter service has been stable.

Leave a Reply

Your email address will not be published. Required fields are marked *