Nested ESXi 6.7 fails to boot

Recently I moved some of my nested ESXi 6.7 VMs from standalone host to a 3 node vSAN cluster. Standalone host is running ESXi 6.7 on HPE DL380 G7 with Xeon 5600 series CPU. vSAN cluster is running similar hardware. I moved the ESXi 6.7 VMs while they were offline. When I powered on the VMs they failed to boot. I saw following error message:

VMB: 553:
Unsupported CPU: Intel family 0x06, model 0x25, stepping 0x1
Intel(R) Xeon(R) CPU X5670 @ 2.93GHz
See http://www.vmware.com/resources/compatibility

CPU unsupported

Only difference between standalone host and cluster hosts was that I had EVC mode enabled on the cluster – Intel® “Westmere” Generation. After I disabled EVC mode the ESXi 6.7 VMs booted without any issues.

 

Home lab update – 2018

Past couple of months I have been working on to update and upgrade my home lab.

My LAB now includes:

3 node VSAN cluster with HPE DL360 G7 SFF, HPE DL380 G7 SFF and HPE DL380 G7 LFF.
A standalone ESXi running on HPE DL380 G7 for running vCenter 6.7 U1 and other supporting services.
A standalone Windows Server HPE DL380 Gen8 to run VMs in WMware Workstation and file server service.

Currently network is 1G. Planning to upgrade to 10G in the future.

Some things I discovered during building the lab.
HPE DL380 G7 LFF with HP P410i also accepts 8TB disks. HPE quick specs only include disks as large as 4TB.
HPE DL380 Gen8 also works with DDR3 16GB 1067Mhz Quad Rank RDIMM memory modules. I was able to install 128GB per CPU. The operating frequency was reduced to 800Mhz.

Excessive amount of “addVob” logs in syslog.log on ESXi 6.5 (Updated)

Last week we noticed that the amount of logs that our ESXi 6.5 servers are sending to syslog server have increased significantly. After some analysis we say that almost 60% of logs are from syslog.log with lines like this – “2018-11-15T15:37:18Z addVob[297777]:<message>” 

We have opened a case in VMware Support but no help so far. I will update the post as the support case progresses.

Some one else had/has a similar issue: https://communities.vmware.com/message/2817143

Update 18.11.2018 These following lines repeat in syslog.log every 5 seconds.

2018-11-15T21:09:42Z addVob[316208]: Could not expand environment variable HOME.
2018-11-15T21:09:42Z addVob[316208]: Could not expand environment variable HOME.
2018-11-15T21:09:42Z addVob[316208]: DictionaryLoad: Cannot open file “/usr/lib/vmware/config”: No such file or directory.
2018-11-15T21:09:42Z addVob[316208]: DictionaryLoad: Cannot open file “~/.vmware/config”: No such file or directory.
2018-11-15T21:09:42Z addVob[316208]: DictionaryLoad: Cannot open file “~/.vmware/preferences”: No such file or directory.
2018-11-15T21:09:42Z addVob[316208]: Wrong number of arguments for vob vob.user.vmsyslogd.remote.failure (got 2; expected 1)
2018-11-15T21:09:42Z addVob[316208]: Usage: /usr/lib/vmware/vob/bin/addvob vob-id [vob-args]
2018-11-15T21:09:42Z addVob[316208]: Recognized vobs:
2018-11-15T21:09:42Z addVob[316208]: vob.user.shell.enabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.shell.disabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.ssh.enabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.ssh.disabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.dcui.enabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.dcui.disabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.lockdownmode.enabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.lockdownmode.disabled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.account.loginfailures (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.account.locked (3 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.maintenancemode.entering (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.maintenancemode.canceled (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.maintenancemode.entered (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.maintenancemode.exited (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.maintenancemode.failed (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.vmsyslogd.remote.failure (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.vmsyslogd.storage.failure (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.vmsyslogd.unexpected (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.vmsyslogd.storage.logdir.invalid (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.hostacceptance.changed (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.install.invalidhardware (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.install.securityalert (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.install.stage.error (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.install.error (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.vib.install.successful (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.vib.remove.successful (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.install.novalidation (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.profile.install.successful (3 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esximage.profile.update.successful (3 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.dhclient.lease.none (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.dhclient.lease.offered.noexpiry (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.ntpd.clock.correction.error (2 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.host.boot (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.host.stop.reboot (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.host.stop.shutdown (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esxcli.host.reboot (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.esxcli.host.poweroff (1 arg)
2018-11-15T21:09:42Z addVob[316208]: vob.user.host.coredump (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.coredump.unconfigured (0 args)
2018-11-15T21:09:42Z addVob[316208]: vob.user.coredump.copyspace (1 arg)

VMware Support claims this is normal.

Update 19.11.2018

In the repeating logs entries there is a message “2018-11-15T21:09:42Z addVob[316208]: Wrong number of arguments for vob vob.user.vmsyslogd.remote.failure (got 2; expected 1)”. This line got me thinking. We have configured two syslog targets. I decided to remove one them and as soon as I did that the spamming in syslog.log stopped. As soon as I added second syslog target back it started again.

vCenter 6.7 update failed – Update installation failed, vCenter is non-operational

I tried to update vCenter 6.7 to vCenter 6.7 Update 1 through Appliance Management web page and as soon as I pressed “Finish” on the update wizard I got an error – Update installation failed, vCenter is non-operational.

I checked the software-packages.log in /var/log/vmware/applmgmt folder but did not notice anything.

Tried running the update from command line: software-packages install –url and the upgrade worked without any issues.

vCenter Appliance update fails – “Update failed. Fix and reset banner.”

I recently tried to patch my vCenter 6.7 to latest patch level and it failed with a weird message on the appliance console – “Update failed. Fix and reset banner.”

Tried to update from console using “software-package” tool – same result. I took a look into software-packages.log file in /var/log/vmware/applmgmt folder and noticed a line “You are required to change your password immediately (root enforced)”. Used passwd to change the root password and restarted the upgrade – Success!.

Warning “esx.problem.hyperthreading.unmitigated” after installing ESXi patches.

This warning may appear after installing patches contained in release ESXi650-201808001 (14 Aug 2018) and you have not updated your vCenter Server. After upgrade of vCenter Server you will see a following warning – “This host is potentially vulnerable to issues described in CVE-2018-3646, please refer to https://kb.vmware.com/s/article/55636 for details and VMware recommendations.”

Link to KB55636
Link to CVE-2018-3646

The correct order to apply these patches is:

1) vCenter patches
2) ESXi patches
3) Evaluate and set “VMkernel.Boot.hyperthreadingMitigation” to “true” if you want to enable the patch.

Some more links to check: KB55806

VMWare vCenter crashing when deploying a VM from a template.

Recently we encountered an issue when we tried to deploy VMs from templates it crashed vCenter Server. VM deployment got stuck between 70%-95% and after a few minutes vCenter Web Clients were not responding and we had to restart vCenter services to get it back up and running.

It seems that templates were missing some info – check https://kb.vmware.com/s/article/2056542. After converting templates back to VMs and again back to templates the deployments were successful and vCenter Server no longer crashed.

Update (13.07.2018) – Same issue affects Veritas NetBackup to get an inventory from a vCenter Server – error “Validation constraint violation”. More info – https://www.veritas.com/support/en_US/article.100033934