Storage reclamation – part 4 – Zero fill and array level reclamation

One way to reclaim storage space is overwrite the dead space with zeros and then get rid of them.

Writing Zeros

One thing to keep in mind that when you write zeros to the disk/LUN it will grow to full size.

Windows

Sdelete – free tool from Microsoft that can be used to write zeros to disks.
Command: sdelete -z E:\

Raxco PerfectDisk – a commercial tool to intelligently overwrite dead space with zeros.

Condusiv V-locity – a commercial tool to overwrite dead space with zeros.

Linux

dd – is a command-line utility for Unix and Unix-like operating systems which can be used to copy disks and files.
Command: dd if=/dev/zero of=/home/zerofile.0 bs=1M

Getting rid of zeros in VMware level

Zeros can be removed using Storage vMotion. When performing Storage vMotion thin must be selected and the source and destination datastores have to have different block size. I have used following combinations:
VMFS5 -> select thin -> VMFS3 (8MB block) -> VMFS5
VMFS5 -> select thin -> NFS -> VMFS5

Reclaim zeros in array level

Different arrays support zero space reclamation in different ways. Check your vendor documents how exactly accomplish this.

EMC VMAX

Command can be executed to reclaim zero space from a LUN.
Solution Enabler command example: symconfigure -cmd “start free on tdev <TDEVID> start_cyl=0 end_cyl=last_cyl type=reclaim;” commit -sid <SID>

Hitachi HUSVM

Zero space from LUN can be reclaimed by running “Reclaim Zero Pages” from Hitachi Command Suite.

EMC VNX

Performing LUN migration will reclaim zero space from LUN. In addition compressing and then uncompressing the LUN will also discard zeros.

Other arrays

Any array that supports inline compression and/or deduplication will probably reclaim any zero space during write operation.

Other posts in this series:

Storage reclamation – part 1 – VMWare vSphere

Storage reclamation – part 2 – Windows

Storage reclamation – part 3  Linux

Storage reclamation – part 3 – Linux

This is the third post in the “Space reclamation” series focusing on Linux.

Like in Windows and in VMware ESXi also in Linux dead space is left behind when data is removed.

For space reclamation under Linux there is available a Linux version of EMC StorReclaim with same options as Windows version.

EMC StorReclaim

EMC StorReclaim

To get EMC StorReclaim contact your EMC representative.

With newer Linux operating systems there is a native trim operations available to get back unused space. I will write about trim in the future when I have had more time to test it.

If anyone knows more tools that can be used to reclaim space in Linux please let me know in the comments sections.

Other posts in this series:

Storage reclamation – part 1 – VMWare vSphere

Storage reclamation – part 2 – Windows

Storage reclamation – part 4 – Zero fill and array level reclamation

Storage reclamation – part 2 – Windows

In previous post I covered storage space reclamation in VMware vSphere. This article describes the methods that can be used in Microsoft Windows to reclaim dead space.

In Windows dead space is left behind when you delete a file or shrink a SQL database for example.

How to find dead space?

Compare Windows file system usage to LUN usage in array.

Windows Server 2012/2012 R2

Windows Server 2012/2012 R2  is capable of identifying the provisioning type and the UNMAP and TRIM capability of a disk. Space reclamation can be triggered by file deletion, a file system level trim, or a storage optimization operation.

You can check if your disks are identified as thin or not from “Defragment and Optimize Drives” tool. If your thinly provisioned disk does not identify as thin disk in Windows check with your array vendor. Array firmware update might be needed to enable support.

Windows Server 2012 thin disk

Windows Server 2012 thin disk

  • You can read more about Windows Server 2012 unmap from here.
  • More information about Windows Server 2012/2012 R2 Defrag can be found here.

Windows Server 2003/2008/2008 R2

Older Windows versions does not include automatic UNMAP support. Third party tools can be used to reclaim space.

EMC StorReclaim

EMC StorReclaim is a command line utility to UNMAP allocated but unused space from storage array. It can be used in physical servers and virtual servers for LUNs which are presented directly from storage array. To get a the tool contact your EMC representative.

Supported operating systems:

  • Windows Server 2003 x86/x64
  • Windows Server 2008 x86/x64
  • Windows Server 2008 R2
  • Windows Server 2008 R2 with SP1
StorReclaim options

StorReclaim options

storreclaim_unmap_status

StorReclaim status

 

VMWare Guest Reclaim

VMWare Labs has published a free tool called Guest Reclaim that works in Windows 2003/2008/2008R/XP/7.

Guest Reclaim reclaims dead space from NTFS volumes hosted on a thin provisioned SCSI disk. The tool can also reclaim space from full disks and partitions, thereby wiping off the file systems on it. As the tool deals with active data, please take all precautionary measures understanding the SCSI UNMAP framework and backing up important data. (Source: VMWare)

In my environment I had problems when VM HW version was 8. It worked when VM HW version 9 or higher.

VMWare Guest Reclaim status

VMWare Guest Reclaim status

More information and downloads can be found here.

Commercial tool – Raxco PerfectStorage

Raxco PerfectStorage is an intelligent storage management solution for thin-provisioned space reclamation that improves productivity, allows for better storage management and forecasting, and delays future storage purchases. (Source: Raxco)

More information can be found from vendor homepage.

PerfectStorage is a agent based system. Agent is installed to operating system (virtual or physical) and then based on schedule SCSI UNMAP or zero fill tasks are executed. Supported operating system today is Windows only. What makes PerfectStorage interesting is that it identifies the space it can reclaim and issue reclaim operations only to those blocks. Most free tools issue reclaim commands to all free space. This can significantly reduce performance impact of reclaim operation. Everything is managed through central console. Also reporting of reclaimed space can be done in management console.

Conclusion

So those are some of the options how to reclaim dead space from virtual or physical Windows servers. I did not write about zero fill options in this post because this mostly requires some more operations from storage side. I will talk about zero fill options later in the post series.

If anyone knows more tools that can be used to reclaim space in Windows please let me know in the comments sections.

Other space reclaim posts in this series:

Storage reclamation – part 1 – VMWare vSphere

Storage reclamation – part 3 – Linux

Storage reclamation – part 4 – Zero fill and array level reclamation

 

Storage reclamation – part 1 – VMware vSphere

Storage space reclamation feature has been a part of VMware vSphere for some time now. But when I have been speaking with virtualization admins and storage admins here in Estonia not many are doing it and some them have not even heard of it. So I decided to do a series of blog posts to write about storage reclamation in different operating systems and arrays.

What is dead space?

Dead space is used space in storage array which is no longer used in file system. Dead space is a result of file operations – example Storage vMotion or a VM removal from a disk. In VMFS level this space is marked as free, but on array LUN level this space is still marked as used.

Below there is an example of disk space usage measurement from array (left) and measurement from datastore (right). As we see on the array LUN uses 2106,105 GB but on the datastore it has actually 984,50 GB free space. This means that this LUN has most likely more than 900GB of dead space.

datastore_used_space

How to get back the space? 

To get back that 900GB of dead space we need to reclaim it somehow. In this post I will write about reclaiming storage in vSphere. In vSphere 5.0 VMware introduced a new feature called Space Reclamation, as part of VAAI Block Thin Provisioning. This feature allows us to inform storage array about unused blocks that can be unmapped and returned to free space pool.

Confirming if reclaim is supported on a LUN

Before unmap operations we need to determine if the array is supporting it. For this following commands cab be used from ESXi to determine array capabilities.

  1. List all the volumes: esxcli storage vmfs extent list
  2. Check VAAI options for the LUNs: esxcli storage core device vaai status get -d naa.xxxxxxxxxxxxxxxxxxxxxxxxxx

datastore_unmap_status

 

vSphere 5.0 U1 or later and vSphere 5.1

In vSphere 5.0 U1 or later and vSphere 5.1 vmkfstools -y command has to used to initiate unmap operation.

To unmap 90% of the unused storage space on a datastore named Datastore1:

  1. Enable SSH on the host where Datastore1 is connected to.
  2. Login to host via SSH.
  3. run command: cd /vmfs/volumes/Datastore1
  4. run command: vmkfstools -y 90

Vmkfstools will then create a balloon file and unmaps the dead space from the LUN.

storage_reclaim_vmkfstools

 

vSphere 5.5

In vSphere 5.5 unmap command is integrated to esxcli command namespace. Old command vmkfstools -y is is deprecated in ESXi 5.5.

To reclaim unused storage blocks on a VMFS datastore, run the command:
esxcli storage vmfs unmap -l <datastorename>

Example:
esxcli storage vmfs unmap -l Datastore1

Performance impact

In vSphere 5.0 and 5.1 there is a noticeable performance impact to a LUN while the reclaim is running. It is recommended to run reclaim during maintenance hours.

unmap_performance_impact

To minimize the performance impact I created a script that is reclaiming space in increments and waiting some time between reclaim operations.

Example of the script:

for datastore in `ls /vmfs/volumes/T*`
do
   echo “Datastore:” $datastore
   cd $datastore
   for i in 10 20 30 40 50 60 70 80 90 95
   do
       vmkfstools -y $i
       sleep 300
   done
done

In vSphere 5.5 the reclaim has much less impact to performance. VMware now claims that it can run outside of the maintenance window.

Performance impact also is dependent from your storage array – some arrays have bigger impact some arrays smaller impact.

How to avoid it?

There is no way to completely avoid emerging of dead space. Try to avoid Storage vMotion operations, try to place VMs directly to proper location when creating them.

Results

After you have done the reclaim compare used space before and after reclamation. Result – 1TB of disk space reclaimed!

before_after_reclaim

 How often should you run reclaim tasks?

It depends how much storage operations are you doing on you’re datastores. If you are creating and deleting VMs every day you probably need to run reclaim task more often to keep the space usage optimal. If you’re datastore content is more or less static you can probably do reclaim less frequently. Check LUN consumed capacity in storage array – high usage in array usually indicates that reclaim should be used for that datastore to get back unused disk space.

More information

Using esxcli in vSphere 5.5 to reclaim VMFS deleted blocks on thin-provisioned LUNs (2057513)

vSphere 5.5 Storage Enhancements Part 4: UNMAP

vSphere 5.5 – Where Is My Space Reclaim Command??

Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849)

How to – Reclaim Dead Space from Thin Provisioned VMFS Datastores

Other space reclaim posts in this series:

Storage reclamation – part 2 – Windows

Storage reclamation – part 3 – Linux

Storage reclamation – part 4 – Zero fill and array level reclamation