VMWare Workstation VMs does not work with Windows Server 2019 deduplicated volumes.

I have been running some of my home lab VMs on top of Windows Server 2012 R2 using VMware Workstation for years now. Over the holidays I decided to setup a test system with Windows Server 2019 and VMware Workstation 15. During the testing I discovered that some the VMs were randomly dying with a crash message in the eventlog.

Faulting application name: vmware-vmx.exe, version:, time stamp: 0x5bf5251a
Faulting module name: vmware-vmx.exe, version:, time stamp: 0x5bf5251a
Exception code: 0xc0000005
Fault offset: 0x000000000049db36
Faulting process id: 0x2a9c
Faulting application start time: 0x01d48a67966e2ea4
Faulting application path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Faulting module path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Report Id: 7dd5aa8c-2f51-471d-b9e4-e04d0caf099b
Faulting package full name:
Faulting package-relative application ID: 

All these VMs were located on disks with data deduplication enabled. One the disks was ReFS and other was NTFS file system. Deduplication settings on Windows Server 2019 and Windows Server 2012 R2 are the same. The issue does not occur on Windows Server 2012 R2.

I also tested other deduplication settings (General purpose file server and Virtualized Backup Server) and the result was the same. VMs on those deduplicated volumes crashed.


Deduplication and compression with ScaleIO

ScaleIO does not support deduplication and compression natively at the moment. Since ScaleIO can use almost any disk device I decided to test ScaleIO combined with QUADStor storage virtualization software which enables deduplication and compression.

For a test I built a small setup – three CentOS 7 servers with 200GB local disk running QUADStor software and ScaleIO SDS software and one ScaleIO MDM server and ScaleIO client based on Windows Server 2012 R2. On each CentOS server QUADStor was used to create 150GB disk with compression and deduplication enabled. The same 150GB disk was used by ScaleIO SDS as storage device.

To the client machine I presented one 200GB disk. To test the deduplication I copied some iso files to that disk. Below it is visible that my test data resulted almost 2x deduplication ratio. Deduplication ratio is affected by the way ScaleIO works – it distributes data to several nodes. Example: block “A” from “dataset1” will end up on servers “One” and “Two”. Block “A” from “dataset2” will end up on servers “One” and “Three”. On server “One” block “A” will deduplicated since it already had the block but on server “Three” the block “A” will not be deduplicated since it’s unique for this server.

QuadStor stats

I did not perform any performance test since my test systems were running on single host and on singe SSD drive.


For conclusion I can say that using 3rd party software it is possible to add features to ScaleIO – deduplication, tiering, etc. Mixing and matching different software can add complexity but sometimes the added value makes sense.

Related posts 

Enabling data deduplication in Linux with QUADStor

Speeding up writes for ScaleIO with DRAM

Automatic storage tiering with ScaleIO


QUADStor homepage

How to estimate deduplication and compession ratios?

In my opinion the honest answer is that you can’t estimate. There is no good and accurate way to estimate deduplication and compression ratios because there are many variables that will affect the ratio. There are estimation tools available from different vendors but you will get most accurate numbers by testing different solutions with your actual data. I have tested several solutions and deduplication and/or compression ratio has varied between 2.5x to 8x.


  • Test different solutions – to understand how well different solutions work. Example solution #1 has dedupe ratio of 4 which looks good but solution #2 has dedupe ration 8 with same data.
  • Try to use the same test data during different tests so the results would be comparable.


  • Microsoft Windows Server 2012 R2 – built in post-process deduplication engine. Check this page for more information.
  • QuadStor software – inline deduplication and compression. Check this page for more information.
  • Nutanix Community Edition – has both deduplication and compression options.
  • All Flash Arrays – most AFA-s include deduplication and/or compression for data reduction. If you are interested of AFA-s most vendors can hook you up with POC equipment which you can use to test the solution. AFA vendors to check EMC, Pure Storage, Kaminario, SolidFire, etc.


Results will vary between different solutions. Deduplication works well for similar data (VDI, Server OS disks), compression works better for databases (Oracle, MSSQL). Deduplication ratio is also affected from deduplication chunk size – 512bytes, 4K, 8K, 16K, etc. Usually smaller chunk size results better ratio.

More info

More info about this topic can be found on following links.

Deduplication ratios – What should be included in the reported ratio?

Understanding Data Deduplication (SNIA)

Enabling data deduplication in Linux with QUADStor

QUADStor is storage virtualization software which features inline deduplication and/or compression. It can be used to present disks to local server or remote servers over iSCSI, FC and Infiniband. Local disks can be shared with NFS or SMB. QUADStor supports VAAI for VMware (ISCSI and FC) and ODX for Microsoft Windows. Documentation and downloads are available at QUADStor homepage – http://www.quadstor.com.


Virtual Machine in VMware Workstation
6 vCPUs
25GB virtual disk on Samsung 840 EVO
CentOS 7 64 bit
QUADStor 3.1.81.


Install needed packages
yum install httpd gcc perl kernel-devel sg3_utils

Install QUADStor
rpm -i quadstor-virt-3.1.81-rhel.x86_64.rpm

Make Apache and QUADStor start during boot
chkconfig httpd on
chkconfig quadstor on


Creating a VDisk

  1. Open web browser and go to http://<yourserver>/
  2. Go to “Physical Storage” and add your disk which you want to enable deduplication to “Default” storage pool. When adding the disk enable Log and Compression.
  3. Wait until disk has been initialized.
  4. Go to “Virtual Disks” and add VDisk. Specify name and size and click Submit. Modify the VDisk and enable compression. Since VDisk is thin provisioned it’s size can be bigger than physical disk.

Accessing the VDisk locally

  1. Identify the VDisk using fdisk -l command. In my system the VDisk appeared as /dev/sdc
  2. Use fdisk or some other tools to create partition to the disk.
  3. Create file system to newly created partition -> mkfs.ext4 /dev/sdc1 
  4. Mount the file system.


Before testing large block writes and small block writes the disk was cleaned and all data was unmapped.  Test file /mnt/a.iso was cached into memory before testing. System cache was flushed before each read test with following command – “sync; echo 3 > /proc/sys/vm/drop_caches”.

Looking at read speed results I can assume that QUADStor has it’s own cache where it caches read blocks. Every read test after first test was above 1GB/s.

One concerning thing I noticed during write testing was that Linux load spiked significantly – up to 30 (1 min).

Large block write

First write

dd if=/mnt/a.iso of=/mnt/dedupe/a.iso bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 52.8046 s, 78.6 MB/s

Second write

dd if=/mnt/a.iso of=/mnt/dedupe/a2.iso bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 33.1746 s, 125 MB/s

Third write

dd if=/mnt/a.iso of=/mnt/dedupe/a3.iso bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 10.3697 s, 400 MB/s

Small block writes

First write

dd if=/mnt/a.iso of=/mnt/dedupe/a1.iso bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 12.5778 s, 330 MB/s

Second write

dd if=/mnt/a.iso of=/mnt/dedupe/a2.iso bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 11.1917 s, 371 MB/s

Third write

dd if=/mnt/a.iso of=/mnt/dedupe/a3.iso bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 11.9529 s, 347 MB/s

Large block reads

First read

dd if=/mnt/dedupe/a1.iso of=/dev/null bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 8.77728 s, 473 MB/s

Second read

dd if=/mnt/dedupe/a2.iso of=/dev/null bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 2.97584 s, 1.4 GB/s

Third read

dd if=/mnt/dedupe/a3.iso of=/dev/null bs=128K
31648+0 records in
31648+0 records out
4148166656 bytes (4.1 GB) copied, 3.08643 s, 1.3 GB/s

Small block reads

First read

dd if=/mnt/dedupe/a1.iso of=/dev/null bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 3.60003 s, 1.2 GB/s

Second read

dd if=/mnt/dedupe/a2.iso of=/dev/null bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 3.24465 s, 1.3 GB/s

Third read

dd if=/mnt/dedupe/a3.iso of=/dev/null bs=8K
506368+0 records in
506368+0 records out
4148166656 bytes (4.1 GB) copied, 3.34678 s, 1.2 GB/s

Use cases

Backup destination – deduplication is very effective on reducing the disk space needed to store backups.
Image library – store for ISO files and VM templates. Shared with SMB or NFS to virtualization hosts.
Document library – store for all kind of documents.
Virtual Machine storage – storage for virtual machines.

Deduplication ratios

I also did another test to measure deduplication ratios. I stored about 800GB of Windows virtual machines on QUADStor disk and I achieved more than 4x deduplication ratio.


QUADStor is interesting software and I will definitely will look into using it more. Performance seems acceptable and deduplication ratios seem compelling.

Finding backup jobs that dedupe badly in EMC Data Domain

Some time ago we suddenly saw a significant drop in free space on our EMC Data Domain appliances.  So we started to investigate what was causing it.

Our setup is Symantec Netbackup, Linux based Media server + DD Boost and Data Domain appliances.

We checked the usual things:

  1. Was some new backups added recently? … No
  2. Any unusual changes in size of the backups? … No

All the usual things did not pan out. Next thing we struggled was a questions how to see how much space each backup is taking in data domain and what data has the worst dedupe ratio. We could not find much from manuals and nothing much about it was available online.

Finally we found location in Data Domain System Manager where you can download a file “Compression Details” which contains each backup job, time, size of the backup, post dedupe and compression size and dedupe ratio.

How to find it? Go to Data Domain System Manager -> Data Management -> Storage Units > select Storage Unit -> Download Compression Details. After some number crunching you are getting a TSV file.



After saving the TSV file it was easy to open it in Excel and sort data into different columns. Columns that we were interested were post_lc and bytes/post_lc. Post_lc contains a byte value of data saved to Data Domain and bytes/post_lc is dedupe-compression ratio. Using values in post_lc we were able to identify backup which was causing us to run out of free space in Data Domain appliance.

Using the Compression Details file it is possible to see what backups dedupe nicely and which are not.

Just for clarification I’m not sure if this solution is valid for any other solutions other than Symantec Netbackup, Linux Media server + DD Boost and EMC Data Domain.


Deduplicate VMWare virtual machines with Windows Server 2012 R2 – part 2

In my previous post I wrote how I used Windows Server 2012 R2 server to provide deduplicated storage for VMWare environment. I achieved some great deduplication results but I was not satisfied with performance. So I built another solution.


  • Physical Windows Server 2012 R2
  • 2×2 Gbit teamed NICs (HP Flex Adapter)
  • 2x 2,9Ghz Intel CPU
  • 192GB RAM
  • Storage Spaces disk pool
    • 100GB SSD
    • 4TB HDD

In to the pool I created one fixed 3TB disk with 92GB (usable SSD space) SSD Write-back cache. I used following powershell command to do it:

New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName vDisk1 -Size 3TB -ResiliencySettingName Simple -ProvisioningType Fixed -WriteCacheSize 92GB

With this setup all small random writes are landing on SSD disk. This ensures much better overall performance than HDD only solution.


During Storage vMotion IO is going to both disks. SSD takes most of the write IOs. Throughput is up to 400MB/sec with multiple concurrent Storage vMotions. Throughput slows down when cache gets full and data has to be destaged to HDD. After destage speed increases again.

NFS datastore performance


With this setup both read latency and write latency are acceptable. Datastore contains 87 running VMs. Dedupe ratio on the volume is 94%. About 30GB of RAM is used for file system cache.

My main goal was to to test if a Windows Server 2012 R2 is a viable option for some of the workloads running on VMWare. From performance perspective I think that this type of setup is viable to run VMs. From availability perspective there is still some things to be done e.g clustering.

Deduplicate VMWare virtual machines with Windows Server 2012 R2

I previously played with VMWare ESXi using NFS from Windows Server 2012. When Windows Server 2012 R2 became available I decided to add deduplication into the mix. For those who does not know Microsoft included a new feature in Windows Server 2012 called Data Deduplication. Data Deduplication enables you to dedupe files stored on NTFS volumes. Previously I was not able to run deduplication on Windows Server 2012 because it does not support deduplication for open files. Starting from Windows Server 2012 R2 Microsoft supports deduplication for open files.

For NFS share I decided to reuse already existing Windows Server 2012 R2 running on Hyper-V. This server is also used to provide deduplicated storage to Hyper-V hosts via SMB3. For NFS I created 500GB LUN and enabled deduplication (VDI mode). Since I was not sure about the performance of that deduplicated NFS datastore I decided to select VMs that did not have high I/O demand. I ended up moving 19 Linux based development servers to this NFS datastore.

Deduplication result is quite good. I achieved about 8x data reduction rate. It helped that all those 19 Linux servers are quite similar.

NFS statsPerformance numbers on the datastore are following:
IOPS: between 20-300 iops
Read Latency: < 5ms
Write Latency: <15ms

Write latency is not that good. But since those servers are only running some script jobs during the night no one has complained.

One thing to remember when using Windows Server based NFS with VMware is that when you move the VM to Windows Server based NFS datastore all the disks get inflated. So after storage vMotion VM will consume all the space it’s allocated. In my case I can not move any VMs which have been allocated more then 500 GBs to this datastore .

In future I’m planning to expand the usage of deduplicated storage for VMWare because it can deliver significant disk space and cost savings.

More information:

Introduction to Data Deduplication in Windows Server 2012

What’s New in Data Deduplication in Windows Server 2012 R2

WS2012 Storage – NFS Server – Configure NFS for VMware vSphere 5.1 Home Lab

Running data deduplication on Hyper-V host

With Windows Server 2012 R2 Microsoft announced a extended feature of data deduplication – deduplication for Virtual Desktop Infrastructure (VDI) deployments. It means that data deduplication is now able to optimize open files.  But only supported configuration is when storage for Hyper-V host is connected remotely through SMB. So running deduplication directly on Hyper-V host is unsupported. Also currently deduplication is only supported for VDI workloads not for server workloads. But unsupported does not mean that it does not work.  So I decided to try dedupe some Windows Server virtual machines running on Hyper-V which are stored locally on SAN disk.

My setup:

  • 1 Hyper-V 2012 R2 host
  • 1TB disk from EMC VNX array – NTFS, 64K
  • Four Windows Server 2008 R2 virtual machines with low  IO requirement

Deduplication settings:

  • Data deduplication type: Virtual Desktop Infrastructure (VDI) server
  • Deduplicate files older than (in days): 0
  • Custom file extentions to exclude: empty
  • Excluded folders: empty
  • Schedule: Throughput optimization from 2:00 AM. Duration 4 hours.
  • Background optimization disabled.

By limiting optimization window from 2:00 AM to 6:00 AM I avoid high load during daytime.


So far I have had no problems. I collected and analyzed some disk performance logs from Hyper-V host and guests and I did not see any problems. There were some latency spikes during optimization process but nothing really bad that would affect the guests.

With four virtual machines on the volume deduplication achieved the ratio of 65% which comes to 113GB of space savings in this case.


Over the course of the day new data which has been written and not yet optimized will consume additional space on the volume. After optimization process has moved unique data  to ChunkStore virtual machines are reduced to consume small amount of space again. ChunkStore is located on the same disk under  System Volume Information\Dedup folder.  As more unique data has been processed by optimization ChunkStore folder will grow. To clean out the data no longer in use the deduplication runs garbage collection job every week. Below you see screenshots from TreeSizeFree. First screenshot is disk usage before optimization. Second screenshot is after the optimization and third is after garbage collection process.


Even though I have not had any problems I do not recommend anyone to run any important workload in such configuration. There has do be a reason why Microsoft is not supporting it. Hopefully in future it will be supported. Meanwhile anyone running deduplication directly on Hyper-V host has to take the risk that something may go wrong.  If anyone has experience with similar setup please let me know in the comments section.