VM disk consolidation failures

I had two cases where VM disk consolidation failed.

First case

Error message: An error occurred while consolidating disks: Could not open/create change tracking file.

Solution:

  1. Shutdown VM
  2. Edit VM configuration parameters and delete all lines that enable CBT (change block tracking). This might be not necessary, but I did it.
  3. Move all *ctk* files away from VM folder
  4. Consolidate the disks

I got help from: https://paulgrevink.wordpress.com/2013/12/02/vm-disk-consolidation-fails/

Second case

Error message: Unable to access the file since it is locked. An error occurred while consolidating disks: msg.fileio.lock

Solution: Reboot all hosts in cluster one by one and try to consolidate the VM disks again.

I hope this will help anyone who has same issues with consolidating VM disks.

Advertisements

Storage space reclamation in VMware with Nutanix

I have been researching storage reclamation for a while. When I got my hands on a Nutanix running VMware I was interested how we could get maximum storage space efficiency out of that. Since Nutanix presents NFS share to ESXi hosts the datastore level reclaim will not be needed. This left me with in-guest reclamation.

After some testing I discovered that writing zeros to virtual machine disks had a interesting affect to VMDK files that resided on Nutanix. VM size had shrinked to a size which it actually consumes. No dead space was left inside VMDK files.

VM size before writing zeros to disk

VM size before writing zeros to disk

VM size after writing zeros to disk

VM size after writing zeros to disk

Storage container space did not change immediately – it was still using the same amount of space as before. Container space usage went down by next morning. Analysis page showed that container space usage went down gradually over night.

Writing zeros

Sdelete is the most widely known tool to write zeros to Windows disks. But caveat using Sdelete is that for some short time the disk is full which can cause problems for applications. I found a better solution – “Davidt Fast Space reclaimer” script written by David Tan. The script generates a 1GB file filled with zeros and copies it until less than 1GB of free space is left on the drive. You can download the script from here.

There is also another script written by Chris Duck called “PowerShell Alternative to SDelete” which can be found from here.

There are also commercial products available for Windows that will write zeros over dead space – Raxco PerfectStorage and Condusiv V-locity. They might be worth to checking out.

For Linux I wrote a script my self that will write zeros until there is less than 1GB space free from total mountpoint size minus 10%. I did minus 10% because to avoid out of space condition and unneeded triggers by monitoring software. My shell scripting skills are not that good so all ideas and suggestions are welcome how to make this script better.

— Script begins here —

#!/bin/bash
# Author: Kalle Pihelgas
# Version: 1.0
# Date: 10.12.2014

# list of mount points with ext3/ext4 file system
localmountpoints=`mount | grep “type ext” | awk ‘{ print $3 }’ `

# loop throug all mountpoints
for mountpoint in $localmountpoints; do

# Get free space
freespace=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f4 `
freespaceint=`echo $freespace`
# end of free space

# get 10% from total size
totalspace=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f2 `
totalspaceint=`echo $totalspace`
ten_totalspace=`echo $totalspaceint*0.1 | bc`
# end getting 10% from total space

# get free space amount that will be filled with zeros
freespace=`echo “($freespaceint-$ten_totalspace)/1024/1024” | bc`

# counter for zero files
a=1

# write zeros until 10% if free
while [ $freespace -gt 0 ]
do
echo Mount point: $mountpoint, zeros left to write: $freespace GB
dd if=/dev/zero of=$mountpoint/zerofile$a bs=1M count=1000
sleep 5
a=`expr $a + 1`

# Get free space again
freespacenew=`df -k $mountpoint | tail -1 | tr -s ‘ ‘ | cut -d’ ‘ -f4 `
freespaceint=`echo $freespacenew`
freespace=`echo “($freespaceint-$ten_totalspace)/1024/1024” | bc`
# end of free space recalculation
done
rm -rf $mountpoint/zerofile*
done

— Script ends here —

DISCLAIMER! I have not tested these scripts extensively. So I urge you to test these scripts thoroughly before running them on your server! I will not be responsible for any damage caused by these scripts!

Getting more out of vSphere Metro Storage Cluster (vMSC)

I have been thinking about vMSC-s for some time now and I hate the idea of leaving 50% cluster memory resources free for fail over in case of data center failure. Why memory and not the CPU? In my 9 years of working with virtualization almost always the bottleneck has been memory. So I was thinking of different ways how to get more out of metro clusters. So I came up with following deployment scenario – mix test, development and production workloads in same cluster and abandon test and development workloads in case of site failure. NB! I have not tested this scenario – it is purely theoretical.

Deployment scenario

  • Two data centers – let’s call them Site A and Site B.
  • Six hosts with 512GB RAM – 3 in each data center. Let’s call them – host-a1, host-a2. host-a3, host-b1, host-b2 and host-b3.
  • Two datastores for production machines that are available in both data centers. Let’s call them DS-A1-PROD and DS-B1-PROD
  • Number of local datastores for test and development only available to local ESXi hosts within local data center. Let’s call them DS-A1-TEST, DS-A2-TEST, DS-B1-TEST and DS-B2-TEST.
  • Workloads will be divided into four groups – Site A production, Site A test and dev, Site B production and Site B test and dev.
    • “Site A production” will be allowed to consume 12% of cluster resources. Workload will be running in Site A.
    • “Site A test and dev” will be allowed to consume 22% of cluster resources. Workload will be running in Site A.
    • “Site B production” will be allowed to consume 12% of cluster resources. Workload will be running in Site B.
    • “Site B test and dev” will be allowed to consume 22% of cluster resources. Workload will be running in Site B.
  • 68% of cluster memory resources will be consumed.
Stretched cluster memory resource usage

Stretched cluster memory resource usage

Site failure scenario

  • Site A goes down.
  • Workload “Site A production” will be restarted by VMware HA in Site B.
  • Workload “Site A test and dev” will be be down until Site A is restored.
  • 92% of the remaining cluster memory resources will be consumed
Stretched cluster memory load in case of site failure. Resources marked with red X are unavailable.

Stretched cluster memory load in case of site failure. Resources marked with red X are unavailable.

Site failure scenario for Site B is the same. Production workload will be restated in Site A and test/dev workload will be unavailable until site B is restored.

Conclusion 

With this type of scenario it is possible to push the metro cluster memory usage from usual 50% to 68%.  Thinking back to this the site “local” workload doesn’t have to be always test or development. It can also be production workload that does not require site fail over. Multi-instance workloads like web front-end servers, Active Directory servers, terminal servers, etc. Although I only used memory as a resource now you should not forget the CPU resources.

As my experience with metro clusters is mostly theoretical I would appreciate if someone with real experience would comment this.