vMotion stuck at 7%

After logging into vCenter (6.7 U2) I noticed several vMotions stuck at 7%. I also found several errors in tasks – “A general system error occurred: Operation timed out”. When I tried to vMotion a VM manually I got and error. vCenter failed to validate the target. After some investigation in different log files I decided to restart vCenter. After vCenter restart all the stuck vMotions were gone and everything was working again.

Update to “CROSS SITE VMOTION VALITATION ERROR” issue.

Last year in June I wrote about an issue which we encountered while trying to do cross-site vMotion in vCenter 6.5 – Cross Site vMotion valitation error: Cannot complete login due to an incorrect user name or password.

After upgrading vCenter to vCenter Server 6.5 Update 2d (build 10964411) the issue has disappeared. Seems VMware has fixed the issue.

vSphere vCenter HTML5 Web Client buttons and icons all wrong after patching

Recently we patched our vSphere vCenter 6.5  to latest version and we noticed that after upgrade most of icons were weird and some of the buttons were wrong.

At first my heart missed a beat since all green icons were suddenly red. After closer look we saw that the status was OK but icon was wrong. I’m not gonna list all the things we tried to fix the issue but eventually we figured out the solution – clear Chrome cache. After clearing the cache the HTML5 GUI was “green” again.

VMware ESXi HA State Uninitialized after exiting maintenance mode

Some time ago I had issue with on ESXi 6.5 host. After exiting ESXi 6.5 hosts from maintenance mode HA state was “Uninitialized”. It was OK before entering maintenance mode.  That host was running build 10884925. All the other hosts including HA master were running build 11925212. I tried to “Reconfigure for VMware HA” – didn’t help. After upgrading the host to build 11925212 HA worked. My colleague also saw similar issue on another cluster with mixed ESXi builds.

Dell software repository for VMware Update Manager

I have written in the past about HPE software and driver repositories – Automatically download HP drivers to VMware Update Manager and New URLs for HP(E) Online Depot for VMWare

There is also similar repository for Dell – https://vmwaredepot.dell.com/index.xml. By adding this to Update Manager it will download additional Dell software – Dell EMC OpenManage Server Administrator and Dell EMC iDRAC Service Module.

We have found the Dell EMC iDRAC Service Module very useful. It provides OS information to iDRAC and also installs binaries which you can use to reset iDRAC from ESXi if it becomes unresponsive (command: /opt/dell/srvadmin/iSM/bin/Invoke-iDRACHardReset -f )

VMWare Workstation VMs does not work with Windows Server 2019 deduplicated volumes.

I have been running some of my home lab VMs on top of Windows Server 2012 R2 using VMware Workstation for years now. Over the holidays I decided to setup a test system with Windows Server 2019 and VMware Workstation 15. During the testing I discovered that some the VMs were randomly dying with a crash message in the eventlog.

Faulting application name: vmware-vmx.exe, version: 15.0.2.40550, time stamp: 0x5bf5251a
Faulting module name: vmware-vmx.exe, version: 15.0.2.40550, time stamp: 0x5bf5251a
Exception code: 0xc0000005
Fault offset: 0x000000000049db36
Faulting process id: 0x2a9c
Faulting application start time: 0x01d48a67966e2ea4
Faulting application path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Faulting module path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Report Id: 7dd5aa8c-2f51-471d-b9e4-e04d0caf099b
Faulting package full name:
Faulting package-relative application ID: 

All these VMs were located on disks with data deduplication enabled. One the disks was ReFS and other was NTFS file system. Deduplication settings on Windows Server 2019 and Windows Server 2012 R2 are the same. The issue does not occur on Windows Server 2012 R2.

I also tested other deduplication settings (General purpose file server and Virtualized Backup Server) and the result was the same. VMs on those deduplicated volumes crashed.

 

Nested ESXi 6.7 fails to boot

Recently I moved some of my nested ESXi 6.7 VMs from standalone host to a 3 node vSAN cluster. Standalone host is running ESXi 6.7 on HPE DL380 G7 with Xeon 5600 series CPU. vSAN cluster is running similar hardware. I moved the ESXi 6.7 VMs while they were offline. When I powered on the VMs they failed to boot. I saw following error message:

VMB: 553:
Unsupported CPU: Intel family 0x06, model 0x25, stepping 0x1
Intel(R) Xeon(R) CPU X5670 @ 2.93GHz
See http://www.vmware.com/resources/compatibility

CPU unsupported

Only difference between standalone host and cluster hosts was that I had EVC mode enabled on the cluster – Intel® “Westmere” Generation. After I disabled EVC mode the ESXi 6.7 VMs booted without any issues.