Test port connectivity in VMware Virtual Appliances

When troubleshooting connectivity for VMware virtual appliances (for example Log Insight, vRealize Operations Manager, vRealize Orchestrator etc) it can be useful to have a simple way to test that ports are open. Especially in DMZ environments with firewall devices.

In the past I have been asked by VMware support to manually install netcat but I recently came across a VMware KB suggesting curl could be used. This is a simpler solution than installing additional software and while the KB references the vCenter server Appliance I have confirmed the curl is available in the vROps appliance as well.

https://kb.vmware.com/kb/2097039

Expanding an Eager Zero Thick VMDK

Expanding an Eager Zero Thick VMDK via the vsphere client will result in the disk being labelled Lazy Zero Thick. I was not expecting this behaviour but it is described in an article by Cormac Hogan (https://blogs.vmware.com/vsphere/2012/06/extending-an-eagerzeroedthick-disk.html). A user in the comments section of that article suggests the issue has been fixed in vSphere 6 but I have not been able to test it for myself.

PowerCLI for Host Storage

Thanks to these excellent blogs:

https://www.virten.net/2014/02/howto-use-esxcli-in-powercli/

http://rvdnieuwendijk.com/2012/08/19/how-to-list-all-the-powercli-esxcli-commands/

 

Following the information presented one can easily perform operations such as listing detached storage devices:

$hosts = Get-Cluster [cluster name] | Get-VMHost

Foreach ($vihost in $hosts) {

Write-Host $vihost

$esxcli = get-vmhost $vihost | Get-EsxCli

$esxcli.storage.core.device.detached.list() | Select DeviceUID | Format-List

}

 

Or you can list the naa ids for all mounted vmfs datastores:

$hosts = Get-Cluster [cluster name] | Get-VMHost

Foreach ($vihost in $hosts) {

Write-Host $vihost

$esxcli = get-vmhost $vihost | Get-EsxCli

$esxcli.storage.vmfs.extent.list() | Select DeviceName | Format-List

}

VMworld 2015

VMware’s annual conference, VMworld, kicked off in San Francisco last week a number of interesting announcements came out of it. A few that jumped out at me:

VMware EVO SDDC:

A pre-configured suite of existing products (including vSphere, VSAN and NSX) bundled with a new automation engine (VMware EVO SDDC Manager) and a new service responsible for physical hardware (Hardware Management Services). It will be sold as a converged platform from partners such as Dell.

The aim is to provide cloud capabilities for private infrastructure with simplified configuration/deployment.

 

Project Skyscraper:

A hybrid cloud development, two features stand out to me –

“Cross-Cloud vMotion” – Live migration between private cloud to vCloud Air and back

“Content Library Sync” – Sync your content library (VMs, OVFs, ISOs) with vCloud Air

Site Recovery Manager 6.1:

New features – Stretched storage support, Storage Policy Protection Groups, Integration with NSX 6.2.

VMware What’s New

VSAN 6.1:

New features – Stretched clusters for high availability across data centers. A new two node cluster deployment (ideal for for branch offices).

VMware Blog

vCloud Air:

Announcement of a SQL service and an object storage service.

SQL datasheet

NSX 6.2 Released

I had though NSX 6.2 would be held for an announcement at VMworld next week but it turns out I was wrong. It’s been released already and brings with it a few nice features. Stand outs include:

  • NSX 6.2 with vSphere 6.0 supports Cross vCenter NSX where logical switches (LS), distributed logical routers (DLR) and distributed firewalls (DFW) can be deployed across multiple vCenters, thereby enabling logical networking and security for applications with workloads (VMs) that span multiple vCenters or multiple physical locations.
  • Consistent firewall policy across multiple vCenters: Firewall Rule Sections in NSX can now be marked as “Universal” whereby the rules defined in these sections get replicated across multiple NSX managers. This simplifies the workflows involving defining consistent firewall policy spanning multiple NSX installations
  • Cross vCenter vMotion with DFW: Virtual Machines that have policies defined in the “Universal” sections can be moved across hosts that belong to different vCenters with consistent security policy enforcement.

These are going to very useful for larger enterprises running multiple vCenter instances. The full release notes may be found on the VMware site.

Which vmdk was deleted?

An admin has deleted a disk from a VM. The only task you see in vCenter simply states “Reconfigure virtual machine”. So which disk was deleted? Where was it located? You can check this in the hostd log.

 

Enable SSH on the host that the VM was running on at the time of the deletion, log on, run:

cat /var/log/hostd.log | grep [VM name]

 

You will see entries like this:

2015-06-17T02:30:57.351Z [FFD9C920 verbose ‘Vmsvc.vm:/vmfs/volumes/550c429f-8eb29ae1-4ebb-b82a71234f1d/[VM name]/ [VM name].vmx’ opID=F669BF69-00005F62-4c-70 user=vpxuser] Adjusting tracking state for disk /vmfs/volumes/550c429f-8eb29ae1-4ebb-b82a71234f1d /[VM name]/ [VM name].vmdk to state disabled.

2015-06-17T02:30:57.355Z [FFD9C920 verbose ‘Vmsvc.vm:/vmfs/volumes/550c429f-8eb29ae1-4ebb-b82a71234f1d/[VM name]/[VM name].vmx’ opID=F669BF69-00005F62-4c-70 user=vpxuser] Disk /vmfs/volumes/550c429f-8eb29ae1-4ebb-b82a71234f1d/[VM name]/[VM name].vmdk: Tracking is already deactivated, skipping.

 

There you can see the location and name of vmdk that was deleted.

PowerCLI One Liners

I have come across various powercli one liners as I work with vSphere that are helpful for quickly collecting information. Some of these come from blogs or the VMware communities pages, thanks to all those contributors! 

This will list the details of RDMs for a VM (very useful if you have many gatekeepers for example):

get-vm -name [VM Name] | get-harddisk -disktype “RawPhysical”,”RawVirtual” | select Parent,Name,DiskType,ScsiCanonicalName,DeviceName | fl

 

This will list VM mapping to physical vmnic:

Connect-viserver [hostname]

Get-EsxTop -CounterName NetPort | Select PortID, ClientName, TeamUplink

 

This will get your host DNS settings (Primary / Secondary DNS servers):

Get-VMHost [hostname] | Get-VMHostNetwork | Select Hostname,DnsAddress

 

This will set your host DNS settings (Primary / Secondary DNS servers)::

Get-VMHostNetwork -VMHost (Get-VMHost -Name [hostname]) | Set-VMHostNetwork -DnsAddress [Primary DNS],[Secondary DNS]

 

This will list the VMs connected to a specific port group:

Get-VM | where { ($_ | Get-NetworkAdapter | where {$_.networkname -match “portgroupname”})}

 

This will start the SSH service on all hosts in a cluster:

Get-Datacenter “[datacenter object name]” | Get-Cluster “[cluster name]” | Get-VMHost | Sort Name | Get-VMHostService | Where { $_.Key -eq “TSM-SSH”} | Start-VMHostService -confirm:$false

 

Stop SSH:

Get-Datacenter “[datacenter object name]” | Get-Cluster “[cluster name]” | Get-VMHost | Sort Name | Get-VMHostService | Where { $_.Key -eq “TSM-SSH”} | Stop-VMHostService -confirm:$false

vSphere 6.0 – The Future Is Here!

vSphere 6.0 has been officially announced. See the VMware vSphere Blog article and the What’s New PDF.

New features that immediately stand out to me include:
· Virtual Volumes
· Long Distance vMotion
· Cross vSwitch and vCenter vMotion
· Content Library
· Storage IOPS reservations
· Virtual SAN 6

The VMTN Blog has posted a page linking to over 80 articles posted by vExperts on the vSphere 6.0 announcement, it’s new features, and it’s deployment. Until I sit down and spend some quality time with the release I can’t add a great deal to this wealth of information. My suggestion is to pick through these articles looking at the features and changes of particular interest to you.

vCenter 5.5 Resource Exhaustion Detected

Following an upgrade of vCenter server from 5.0 to 5.5 the vCenter service intermittently stopped and we began to see a number of resource exhaustion events:

Event ID: 2004
Source: Resource-Exhaustion-Detect
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: java.exe (2116) consumed 10149273600 bytes, java.exe (2088) consumed 4624416768 bytes, and vpxd.exe (14440) consumed 4113379328 bytes.

We increased the allocated memory (from 12GB to 24GB) and page file (from 4Gb to 6GB) but continued to experience problems. I came across a similar issue in the VMware Communities in which user Sateesh_vcloud documented the standard JVM Heap settings:

Default values for vCenter server installation:

(null)

Our vCenter inventory was approx 100 hosts and 2000 virtual machines but we had selected large inventory for all services during the upgrade. Therefore the JVM heap allocated for each service was likely larger than we required. Sateesh_vcloud also documented the locations of the configuration files for each service:

Single Sign On:
C:\Program Files\VMware\Infrastructure\SSOServer\conf\wrapper.conf
Set wrapper.java.additional.9=”-Xmx” (default: “1024M”) to “256M”
Set wrapper.java.additional.14=”-XX:MaxPermSize=” (default: “512M”) to “128M” (or half of the Xmx value chosen before)

Inventory Service:
C:\Program Files\VMware\Infrastructure\Inventory Service\conf\wrapper.conf
Set wrapper.java.maxmemory (default: “3072”) to “384” (MB)

Tomcat:
C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf
Set wrapper.java.additional.9=”-Xmx” (default: “1024M”) to “512M” – “768M”
Set wrapper.java.additional.14=”-XX:MaxPermSize” (default: “256M”) to half of the Xmx value chosen before

Web Client:
C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
Set wrapper.java.initmemory (default: “1024m”) to “256m”
Set wrapper.java.maxmemory (default: “1024m”) to “384m”

Log Browser:
C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf
Set wrapper.java.maxmemory (default: “512”) to “256” (MB)

Profile Driven Storage:
C:\Program Files\VMware\Infrastructure\Profile-Driven Storage\conf\wrapper.conf
Set wrapper.java.initmemory (default: “256”) to “128” (MB)
Set wrapper.java.maxmemory (default: “1024”) to “384” (MB)

Orchestrator:
C:\Program Files\VMware\Infrastructure\Orchestrator\app-server\bin\wrapper.conf
Set wrapper.java.additional.3=-Xmn (default: “768m”) to “256m”
Set wrapper.java.initmemory (default: “2048”) to “384” (MB)
Set wrapper.java.maxmemory (default: “2048”) to “512” (MB)

I updated the Inventory Service configuration file to 6144MB (previously 1288MB) and restarted the service. We have not had a reoccurrence of the resource exhaustion and the vCenter service has been stable.

vCenter Service Failing to Start

I recently upgraded vCenter from 5.0 U3 to 5.5 U2 which went smoothly and ran fine until our standard monthly windows patch window when we found the primary vCenter service would not start.

I initially flagged the issue with our database operations team and asked them to health check the SQL database for vCenter.

However I continued investigating and upon checking the vpxd.log file I found:

[VpxdReverseProxy] Failed to create http proxy: An attempt was made to access a socket in a way forbidden by its access permissions.

This lead me to a VMware knowledge base article listing troubleshooting steps for the vCenter service. Step four of this article suggested verification of the ports required by vCenter. Running ‘netstat –bano’ I found port 80 appeared to be in use by process id 4. Via Task Manager I found process ID 4 owned by the System which was not a conclusive identifier however it ruled out some potential suspects.

Looking at the knowledge base article again, it lists some services to specifically check for –

‘If another application, such as Microsoft Internet Information Server (IIS) (also known as Web Server (IIS) on Windows 2008 Enterprise), Routing and Remote Access Service (RAS), World Wide Web Publishing Services (W3SVC), Windows Remote Management service (WS-Management) or the Citrix Licensing Support service are utilizing any of the ports, vCenter Server cannot start.’

Reviewing the services running on the server I found the Window Remote Management service. I stopped the service and then retried vCenter. It was successful. I was then able to restart the Windows Remote Management service and vCenter continued to run.

I subsequently found a blog called The World According to Gabe that detailed a permanent solution.

Recording the key steps here for my own future reference:

If when you run winrm get winrm/config | find /I “http” you find that WinRM is listening on port 80 by default, run the following command:

winrm set winrm/config/listener?Address=*+Transport=HTTP @{Port=”8888”}

If you want WinRM to listen on a different port, just change the “8888” to whatever port you wish, without breaking the formatting.

If you find that WinRM is not listening on port 80 by default, but is still grabbing the port, run the following command:

winrm set winrm/config/service @{EnableCompatibilityHttpListener=”false”}

Later still I found another VMware knowledge base article specific to the Window Remote Management service.