Storage space reclamation feature has been a part of VMware vSphere for some time now. But when I have been speaking with virtualization admins and storage admins here in Estonia not many are doing it and some them have not even heard of it. So I decided to do a series of blog posts to write about storage reclamation in different operating systems and arrays.
What is dead space?
Dead space is used space in storage array which is no longer used in file system. Dead space is a result of file operations – example Storage vMotion or a VM removal from a disk. In VMFS level this space is marked as free, but on array LUN level this space is still marked as used.
Below there is an example of disk space usage measurement from array (left) and measurement from datastore (right). As we see on the array LUN uses 2106,105 GB but on the datastore it has actually 984,50 GB free space. This means that this LUN has most likely more than 900GB of dead space.
How to get back the space?
To get back that 900GB of dead space we need to reclaim it somehow. In this post I will write about reclaiming storage in vSphere. In vSphere 5.0 VMware introduced a new feature called Space Reclamation, as part of VAAI Block Thin Provisioning. This feature allows us to inform storage array about unused blocks that can be unmapped and returned to free space pool.
Confirming if reclaim is supported on a LUN
Before unmap operations we need to determine if the array is supporting it. For this following commands cab be used from ESXi to determine array capabilities.
- List all the volumes: esxcli storage vmfs extent list
- Check VAAI options for the LUNs: esxcli storage core device vaai status get -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxx
vSphere 5.0 U1 or later and vSphere 5.1
In vSphere 5.0 U1 or later and vSphere 5.1 vmkfstools -y command has to used to initiate unmap operation.
To unmap 90% of the unused storage space on a datastore named Datastore1:
- Enable SSH on the host where Datastore1 is connected to.
- Login to host via SSH.
- run command: cd /vmfs/volumes/Datastore1
- run command: vmkfstools -y 90
Vmkfstools will then create a balloon file and unmaps the dead space from the LUN.
In vSphere 5.5 unmap command is integrated to esxcli command namespace. Old command vmkfstools -y is is deprecated in ESXi 5.5.
To reclaim unused storage blocks on a VMFS datastore, run the command:
esxcli storage vmfs unmap -l <datastorename>
esxcli storage vmfs unmap -l Datastore1
In vSphere 5.0 and 5.1 there is a noticeable performance impact to a LUN while the reclaim is running. It is recommended to run reclaim during maintenance hours.
To minimize the performance impact I created a script that is reclaiming space in increments and waiting some time between reclaim operations.
Example of the script:
for datastore in `ls /vmfs/volumes/T*`
echo “Datastore:” $datastore
for i in 10 20 30 40 50 60 70 80 90 95
vmkfstools -y $i
In vSphere 5.5 the reclaim has much less impact to performance. VMware now claims that it can run outside of the maintenance window.
Performance impact also is dependent from your storage array – some arrays have bigger impact some arrays smaller impact.
How to avoid it?
There is no way to completely avoid emerging of dead space. Try to avoid Storage vMotion operations, try to place VMs directly to proper location when creating them.
After you have done the reclaim compare used space before and after reclamation. Result – 1TB of disk space reclaimed!
How often should you run reclaim tasks?
It depends how much storage operations are you doing on you’re datastores. If you are creating and deleting VMs every day you probably need to run reclaim task more often to keep the space usage optimal. If you’re datastore content is more or less static you can probably do reclaim less frequently. Check LUN consumed capacity in storage array – high usage in array usually indicates that reclaim should be used for that datastore to get back unused disk space.
Using esxcli in vSphere 5.5 to reclaim VMFS deleted blocks on thin-provisioned LUNs (2057513)
vSphere 5.5 Storage Enhancements Part 4: UNMAP
vSphere 5.5 – Where Is My Space Reclaim Command??
Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849)
How to – Reclaim Dead Space from Thin Provisioned VMFS Datastores
Other space reclaim posts in this series:
Storage reclamation – part 2 – Windows
Storage reclamation – part 3 – Linux
Storage reclamation – part 4 – Zero fill and array level reclamation