Enabled CPU hot-add setting in vSphere disables vNUMA

Few days ago I was helping a database admin – he was complaining that his large MS SQL server is seeing only one 20 vCPU vNUMA node and this is hindering performance.

So I did some research …

  • It’s better to increase socket count rather then core count – this allows more flexible vNUMA configurations
  • vNUMA will be automatically enabled when VM has more than 8 vCPUs
  • vNUMA-aware VM must be hardware version 8
  • Enabling CPU hot-add disables vNUMA (VMware KB2040375)

All four points were valid for this database server – 20 vCPUs (20 sockets * 1 core), VM hardware is version 8 and CPU hot-add was enabled. Enabled CPU hot-add setting disabled vNUMA and forced VM to use UMA (Uniform memory access). When CPU hot-add was disabled vNUMA kicked in and created 2 vNUMA nodes with 10 vCPU each.

To view how many vNUMA nodes server has following software can be used – Windows: Coreinfo from Sysinternals (command: coreinfo -n) and Linux: numactl (command: numactl  –hardware).

There is a good blog post about effects of wrongly configured vNUMA on VMware vSphere Blog titled – Does corespersocket Affect Performance?

It is highly probable that miss-configured vNUMA can have performance impact. I will disable CPU hot-add on all of the large VMs and in the future I will not enable CPU hot-add on large VMs to get all the benefits of enabled vNUMA.

More information about vNUMA:
VMware CPU Hot Plug vNUMA Effects on SQL Server
vNUMA: What it is and why it matters
Checking the vNUMA Topology

IO performance issues with NOS 4.1.1 on NX-8K and NX-9K series nodes

Nutanix has reported that there is a performance issue with NX-8K and NX-9K series nodes after upgrading to NOX 4.1.1 (Nutanix Field Advisory #27). Nutanix engineering is working on the problem. Mean while the customers are advised not to upgrade to NOS 4.1.1 when running on NX-8K and NX-9K nodes. More information about the problem will be published on 9th of February.