Step by step how to install Nutanix CE nested on VMware ESXi

I know that it is not ideal to run Nutanix CE in nested configuration as performance will be affected, but if you don’t have separate compatible hardware available you are still able to install it and play with it. Since version ce-2015.06.08-beta Nutanix Community Edition can run nested on different VMware products. This post will describe how to install Nutanix CE nested on ESXi.

ESXi host configuration

2 x 8 core Intel Xeon E5-2680
384GB RAM
500GB SSD
1024GB HDD
VMWare ESXi 5,5 U2

Nutanix CE VM configuration

12 vCPUs ( 2 sockets x 6 cores)
128GB RAM
400GB vmdk on SSD
900GB vmdk on HDD
8GB Nutanix CE image as vmdk
1 x Intel e1000 virtual nic

Preparations

I’m assuming that registration and downloading Nutanix CE is already done.

Step 1 – prepare the boot disk

  1. extract the ce-2015.06.08-beta.img from the archive
  2. rename the ce-2015.06.08-beta.img to ce-flat.vmdk
  3. create disk descriptor file or download it from Joep Piscaer (http://www.virtuallifestyle.nl) blog here -> https://www.virtuallifestyle.nl/2015/06/nextconf-running-nutanix-community-edition-nested-on-fusion/.
  4. Rename ce.txt to ce.vmdk.

Step 2 – prepare the VM

  1. create a VM
    1. create new Centos 4/5/6/7 (64bit) VM
    2. 12vCPUs (minimum of 4 vCPU)
    3. 128GB RAM (minimum of 16GB RAM)
    4. 900GB vmdk for HDD – map as SCSI0:0 (HDD has to be atleast 500+ GB)
    5. 350GB vmdk for SSD – map as SCSI0:1 (SSD has to be atleast 200+ GB)
    6. Intel e1000 NIC
  2. browse the datastore where you stored the VM and upload ce-flat.vmdk and ce.vmdk
  3. edit VM
    1. Enable virtualization for VM – select CPU -> Hardware virtualization -> Expose hardware assisted virtualization to the guest OS.Nutanix CE VM CPU settings
    2. add new SATA controller
    3. attach existing disk ce.vmdk as SATA disk

Step 3 – installation

  1. Start the VM -> press F2 to enter BIOS -> setup SATA disk as default boot disk
  2. boot the server from SATA disk
  3. Optional: Modify SSD IOps requirements
    1. login as root (password nutanix/4u)
    2. edit /home/install/phx_iso/phoenix/sysUtil.py lines “SSD_rdIOPS_thresh = 5000” and “SSD_wrIOPS_thresh = 5000”
  4. Optional: Modify Controller VM (CVM) memory
    1. login as root (password nutanix/4u)
    2. edit /home/install/phx_iso/phoenix/sysUtil.py line “SVM_GB_RAM = 16”
  5. to start installation login as user “install”

Step 4 – enable “Promiscuous mode”

  1. enable “Promiscuous mode” on the vSwitch or virtual machine network where you have nutanix-ce VM connected. Otherwise controller VM is not accessible.
  2. Update 03.08.2016 – you may also need to enable “Forged Transmits”. Thanks to Thiago to pointing this out.

Conclusion

I used these steps to successfully install Nutanix CE on top of VMware ESXi 5.5 U2 running on HP server which is not in the Nutanix HCL.

More information

Joep Piscaer describes how to install Nutanix CE to VMware Fusion which also works for VMware Workstation – https://www.virtuallifestyle.nl/2015/06/nextconf-running-nutanix-community-edition-nested-on-fusion/

Running Nutanix Community Edition nested on VMware Workstation

It is now possible to install and run Nutanix Community Edition (Version 2015.06.06) nested on VMware Fusion and VMware Workstation. 

Joep Piscaer has detailed install guide for Fusion that also works for Workstation – http://www.virtuallifestyle.nl/2015/06/nextconf-running-nutanix-community-edition-nested-on-fusion/

UPDATE – This issue occurred with version ce-2015.06.06-beta. This issue no longer exist with version ce-2015.06.08-beta.

Only problem I ran into was that I was getting following error from the installer:

INFO: Nutanix Community Edition cannot be installed on this hardware. Reason:
INFO: A set of disks matched the minimum requirements was not found.

After some digging I found that both of the added SCSI drives were identified as spinning disks although they both were stored on SSD. To fix this issue:

  • Login as root. Password – “nutanix/4u”
  • Identify your “SSD” and “HDD” -> “fdisk -l”. In my setup “SSD” is “/dev/sda”
  • Run “cat /sys/block/sda/queue/rotational”. 0 means SSD and 1 means HDD.
  • Run “echo 0 > /sys/block/sda/queue/rotational” to change the value from 1 to 0.
  • Rerun the installer

Now the installer was happy with my setup and after few minutes I had Nutanix Community Edition up and running on VMware Workstation.

Nutanix – No rebuild capacity available

Nutanix resilency status

“No rebuild capacity available” warning appears to Nutanix clusters when amount of free disk space in clusters is lower than capacity of a single node. This means when you loose one node it is not possible to fully rebuild data resiliency. There are options to fix this – free up space or add more nodes to your cluster. You can free up space by removing obsolete VMs or reclaim dead space within running VMs. I covered here how to reclaim space within VMs.

To check how much storage capacity single node provides go to Hardware -> Table -> select one host -> check “Storage Capacity” value.

Storage space reclamation in VMware with Nutanix

I have been researching storage reclamation for a while. When I got my hands on a Nutanix running VMware I was interested how we could get maximum storage space efficiency out of that. Since Nutanix presents NFS share to ESXi hosts the datastore level reclaim will not be needed. This left me with in-guest reclamation.

After some testing I discovered that writing zeros to virtual machine disks had a interesting affect to VMDK files that resided on Nutanix. VM size had shrinked to a size which it actually consumes. No dead space was left inside VMDK files.

VM size before writing zeros to disk

VM size before writing zeros to disk

VM size after writing zeros to disk

VM size after writing zeros to disk

Storage container space did not change immediately – it was still using the same amount of space as before. Container space usage went down by next morning. Analysis page showed that container space usage went down gradually over night.

Writing zeros

Sdelete is the most widely known tool to write zeros to Windows disks. But caveat using Sdelete is that for some short time the disk is full which can cause problems for applications. I found a better solution – “Davidt Fast Space reclaimer” script written by David Tan. The script generates a 1GB file filled with zeros and copies it until less than 1GB of free space is left on the drive. You can download the script from here.

There is also another script written by Chris Duck called “PowerShell Alternative to SDelete” which can be found from here.

There are also commercial products available for Windows that will write zeros over dead space – Raxco PerfectStorage and Condusiv V-locity. They might be worth to checking out.

For Linux I wrote a script my self that will write zeros until there is less than 1GB space free from total mountpoint size minus 10%. I did minus 10% because to avoid out of space condition and unneeded triggers by monitoring software. My shell scripting skills are not that good so all ideas and suggestions are welcome how to make this script better.

— Script begins here —

#!/bin/bash
# Author: Kalle Pihelgas
# Version: 1.0
# Date: 10.12.2014

# list of mount points with ext3/ext4 file system
localmountpoints=`mount | grep “type ext” | awk ‘{ print $3 }’ `

# loop throug all mountpoints
for mountpoint in $localmountpoints; do

# Get free space
freespace=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f4 `
freespaceint=`echo $freespace`
# end of free space

# get 10% from total size
totalspace=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f2 `
totalspaceint=`echo $totalspace`
ten_totalspace=`echo $totalspaceint*0.1 | bc`
# end getting 10% from total space

# get free space amount that will be filled with zeros
freespace=`echo “($freespaceint-$ten_totalspace)/1024/1024” | bc`

# counter for zero files
a=1

# write zeros until 10% if free
while [ $freespace -gt 0 ]
do
echo Mount point: $mountpoint, zeros left to write: $freespace GB
dd if=/dev/zero of=$mountpoint/zerofile$a bs=1M count=1000
sleep 5
a=`expr $a + 1`

# Get free space again
freespacenew=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f4 `
freespaceint=`echo $freespacenew`
freespace=`echo “($freespaceint-$ten_totalspace)/1024/1024” | bc`
# end of free space recalculation
done
rm -rf $mountpoint/zerofile*
done

— Script ends here —

DISCLAIMER! I have not tested these scripts extensively. So I urge you to test these scripts thoroughly before running them on your server! I will not be responsible for any damage caused by these scripts!

IO performance issues with NOS 4.1.1 on NX-8K and NX-9K series nodes

Nutanix has reported that there is a performance issue with NX-8K and NX-9K series nodes after upgrading to NOX 4.1.1 (Nutanix Field Advisory #27). Nutanix engineering is working on the problem. Mean while the customers are advised not to upgrade to NOS 4.1.1 when running on NX-8K and NX-9K nodes. More information about the problem will be published on 9th of February.

Nutanix Controller VM console output under KVM

Recently I had a chance to be a part of setting up a Nutanix block (3 nodes). That block shipped with KVM hypervisor from the factory. Nutanix engineer was on site with us to do the first installation. Our first task was to replace the KVM with ESXi. We booted up all 3 nodes to do the re-imaging but for some reason only 2 nodes appeared on the management software. After some debugging we discovered that the Controller VM (CVM) on the 3-rd node was not working. There is a CVM console output file named NTNX* located in /tmp. From there we were able to see why the CVM was not booting up. For us the quickest way to fix the problem was to re-image the node manually with ESXi version.