Storage reclamation – part 1 – VMware vSphere

Storage space reclamation feature has been a part of VMware vSphere for some time now. But when I have been speaking with virtualization admins and storage admins here in Estonia not many are doing it and some them have not even heard of it. So I decided to do a series of blog posts to write about storage reclamation in different operating systems and arrays.

What is dead space?

Dead space is used space in storage array which is no longer used in file system. Dead space is a result of file operations – example Storage vMotion or a VM removal from a disk. In VMFS level this space is marked as free, but on array LUN level this space is still marked as used.

Below there is an example of disk space usage measurement from array (left) and measurement from datastore (right). As we see on the array LUN uses 2106,105 GB but on the datastore it has actually 984,50 GB free space. This means that this LUN has most likely more than 900GB of dead space.

datastore_used_space

How to get back the space? 

To get back that 900GB of dead space we need to reclaim it somehow. In this post I will write about reclaiming storage in vSphere. In vSphere 5.0 VMware introduced a new feature called Space Reclamation, as part of VAAI Block Thin Provisioning. This feature allows us to inform storage array about unused blocks that can be unmapped and returned to free space pool.

Confirming if reclaim is supported on a LUN

Before unmap operations we need to determine if the array is supporting it. For this following commands cab be used from ESXi to determine array capabilities.

  1. List all the volumes: esxcli storage vmfs extent list
  2. Check VAAI options for the LUNs: esxcli storage core device vaai status get -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxx

datastore_unmap_status

 

vSphere 5.0 U1 or later and vSphere 5.1

In vSphere 5.0 U1 or later and vSphere 5.1 vmkfstools -y command has to used to initiate unmap operation.

To unmap 90% of the unused storage space on a datastore named Datastore1:

  1. Enable SSH on the host where Datastore1 is connected to.
  2. Login to host via SSH.
  3. run command: cd /vmfs/volumes/Datastore1
  4. run command: vmkfstools -y 90

Vmkfstools will then create a balloon file and unmaps the dead space from the LUN.

storage_reclaim_vmkfstools

 

vSphere 5.5

In vSphere 5.5 unmap command is integrated to esxcli command namespace. Old command vmkfstools -y is is deprecated in ESXi 5.5.

To reclaim unused storage blocks on a VMFS datastore, run the command:
esxcli storage vmfs unmap -l <datastorename>

Example:
esxcli storage vmfs unmap -l Datastore1

Performance impact

In vSphere 5.0 and 5.1 there is a noticeable performance impact to a LUN while the reclaim is running. It is recommended to run reclaim during maintenance hours.

unmap_performance_impact

To minimize the performance impact I created a script that is reclaiming space in increments and waiting some time between reclaim operations.

Example of the script:

for datastore in `ls /vmfs/volumes/T*`
do
   echo “Datastore:” $datastore
   cd $datastore
   for i in 10 20 30 40 50 60 70 80 90 95
   do
       vmkfstools -y $i
       sleep 300
   done
done

In vSphere 5.5 the reclaim has much less impact to performance. VMware now claims that it can run outside of the maintenance window.

Performance impact also is dependent from your storage array – some arrays have bigger impact some arrays smaller impact.

How to avoid it?

There is no way to completely avoid emerging of dead space. Try to avoid Storage vMotion operations, try to place VMs directly to proper location when creating them.

Results

After you have done the reclaim compare used space before and after reclamation. Result – 1TB of disk space reclaimed!

before_after_reclaim

 How often should you run reclaim tasks?

It depends how much storage operations are you doing on you’re datastores. If you are creating and deleting VMs every day you probably need to run reclaim task more often to keep the space usage optimal. If you’re datastore content is more or less static you can probably do reclaim less frequently. Check LUN consumed capacity in storage array – high usage in array usually indicates that reclaim should be used for that datastore to get back unused disk space.

More information

Using esxcli in vSphere 5.5 to reclaim VMFS deleted blocks on thin-provisioned LUNs (2057513)

vSphere 5.5 Storage Enhancements Part 4: UNMAP

vSphere 5.5 – Where Is My Space Reclaim Command??

Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849)

How to – Reclaim Dead Space from Thin Provisioned VMFS Datastores

Other space reclaim posts in this series:

Storage reclamation – part 2 – Windows

Storage reclamation – part 3 – Linux

Storage reclamation – part 4 – Zero fill and array level reclamation

Advertisements

10 thoughts on “Storage reclamation – part 1 – VMware vSphere

  1. Pingback: Storage reclamation – part 2 – Windows | Kalle's playground
  2. Hi

    I just changed a little bit your script:

    for datastore in `ls /vmfs/volumes/ | grep P`
    do
    echo Datastore: $datastore
    cd /vmfs/volumes/$datastore
    for i in 20 60 90 95
    do
    echo $i
    pwd
    vmkfstools -y $i
    sleep 10
    done
    done

    All my Datastore begin with the letter P. This is for ESXi 5.1U2

  3. Pingback: How to reclaim storage on HP 3Par systems. | Mcloud.info
  4. Pingback: ReClaiming a VMware Datastore using vmkfstools Scheduled Task | Welcome to Pariswells.com
  5. Pingback: Experiments with dead space reclamation and the wonders of storage over provisioning | Arik Yavilevich's blog
  6. Pingback: Ahmed El Maraghy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s