VMWare Workstation VMs does not work with Windows Server 2019 deduplicated volumes.

I have been running some of my home lab VMs on top of Windows Server 2012 R2 using VMware Workstation for years now. Over the holidays I decided to setup a test system with Windows Server 2019 and VMware Workstation 15. During the testing I discovered that some the VMs were randomly dying with a crash message in the eventlog.

Faulting application name: vmware-vmx.exe, version:, time stamp: 0x5bf5251a
Faulting module name: vmware-vmx.exe, version:, time stamp: 0x5bf5251a
Exception code: 0xc0000005
Fault offset: 0x000000000049db36
Faulting process id: 0x2a9c
Faulting application start time: 0x01d48a67966e2ea4
Faulting application path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Faulting module path: C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe
Report Id: 7dd5aa8c-2f51-471d-b9e4-e04d0caf099b
Faulting package full name:
Faulting package-relative application ID: 

All these VMs were located on disks with data deduplication enabled. One the disks was ReFS and other was NTFS file system. Deduplication settings on Windows Server 2019 and Windows Server 2012 R2 are the same. The issue does not occur on Windows Server 2012 R2.

I also tested other deduplication settings (General purpose file server and Virtualized Backup Server) and the result was the same. VMs on those deduplicated volumes crashed.


Windows Defender Credential Guard on VMWare

I got a question about using Windows Defender Credential Guard in VMware virtual machines. I did some digging and found following things:

Info from https://docs.microsoft.com/en-us/windows/security/identity-protection/credential-guard/credential-guard-requirements

Windows Defender Credential Guard requires:

  • Support for Virtualization-based security (required)
  • Secure boot (required)
  • TPM 2.0 either discrete or firmware (preferred – provides binding to hardware)
  • UEFI lock (preferred – prevents attacker from disabling with a simple registry key change)

The Virtualization-based security requires:

  • 64-bit CPU
  • CPU virtualization extensions plus extended page tables
  • Windows hypervisor

As of today only VMware Workstation 14 has option to enable Virtualization-based security. I created a case to VMWare where I inquired about support for Virtualization-based security in vSphere and answer I got that it will be available in the future version.

Update: VMWare has now released vSphere 6.7 which supports Virtualization-based security options.

Windows Server 2016 Tech Preview 2 Storage Spaces disk FriendlyName values on VMware Workstation are all the same

I was playing with Storage Spaces under Windows Server 2016 Tech Preview 2. As usual in virtual environment Storage Spaces was unable to detect MediaType correctly. When I went to fix it with a usual command “Set-PhysicalDisk -FriendlyName PhysicalDisk1 -MediaType SSD” it did not work because disk FriendlyName was not PhysicalDisk1. Instead it was “VMware, VMware Virtual S”. Actually all the disks had the same FriendlyName – “VMware, VMware Virtual S“.

Disk FriendlyNames before changing them

To fix this I assigned each disk unique FriendlyName and also fixed MediaType.

To change disk FriendlyName and MediaType I used command:
Set-PhysicalDisk -UniqueId “{8433bca0-314d-11e5-94f3-000c29b96d5e}” -NewFriendlyName “SSD1” -MediaType SSD

To get disk UniqueId values I used command:
Get-StoragePool Pool2 | Get-PhysicalDisk | Select FriendlyName,MediaType,UniqueId,Size

After changing the FriendlyNames and MediaTypes it was much more clearer which disk is SSD and which disk is HDD.

Disk FriendlyNames changed

PrimoCache – Disk caching software for Windows

PrimoCache is caching software for Windows servers and workstations that can use DRAM and/or SSD to speed up both read and write opeartions. PrimoCache has two levels of cache – level-1 is DRAM and level-2 is SSD. A 90-day trial version can be downloaded from Romex Software website.

Main Features (source Romex Software website)

  • Supports physical memory, solid-state drives and flash drives as cache storage.
  • Implements a two-level caching architecture.
  • Supports persistent level-2 caching.
  • Supports Write-Through and Write-Deferred caching modes.
  • Supports TRIM command.
  • Supports OS Invisible Memory.
  • Implements an intelligent and self-tuning cache replacement algorithm.
  • Supports caching strategies: Read/Write Caching, Read-Only Caching and Write-Only Caching.
  • Supports performance statistics and monitor.
  • Supports caching for multiple volumes
  • Supports caching for volumes with proprietary file system.
  • Supports caching for volumes on basic and dynamic disks.
  • Supports plug and play.
  • Supports command-line interface.

More info: http://www.romexsoftware.com/en-us/primo-cache/index.html

Testing and results

For testing I selected a virtual machine which is running some management software with integrated mysql db server. For some reason mysql was causing heavy read IO – 2000 to 3000 iops. I created a 4GB read-only cache. After 30 minutes cache was warmed and mysql was no longer stressing the storage array. Cache hit rate is almost 100% with 1,76GB of cache still free.

PrimoCache application

PrimoCache application

From VMware performance charts it is visible that read IO dropped almost to zero.

Read IO before and after enabling cache

Read IO before and after enabling cache

Read IOPS also disappeared.

Read IOPS before and after enabling cache

Read IOPS before and after enabling cache

Other use case

Write-deferred cache will turn small random IOs into large sequential IOs. This caching option could potentially speed up non-ssd disks which are not that great for small random writes.


PrimoCache is licensed per PC/server. Non-commercial license starts at $29.95, business license starts at $119. License includes lifetime free updates and technical support.

Other similar solutions

SuperSpeed SuperCache Express – http://www.superspeed.com/servers/supercache.php
HGST ServerCache – http://www.hgst.com/software/HGST-server-cache


Typically in virtualized environment one would look for a solution in hypervisor layer. But this typically means you need to license several hosts and probably equip them with SSD disks. All of this adds up to cost. This cost might not make sense if you only have one or two machines that would benefit from the solution. Here’s where PrimoCache or other similar solutions might help.

Storage reclamation – part 2 – Windows

In previous post I covered storage space reclamation in VMware vSphere. This article describes the methods that can be used in Microsoft Windows to reclaim dead space.

In Windows dead space is left behind when you delete a file or shrink a SQL database for example.

How to find dead space?

Compare Windows file system usage to LUN usage in array.

Windows Server 2012/2012 R2

Windows Server 2012/2012 R2  is capable of identifying the provisioning type and the UNMAP and TRIM capability of a disk. Space reclamation can be triggered by file deletion, a file system level trim, or a storage optimization operation.

You can check if your disks are identified as thin or not from “Defragment and Optimize Drives” tool. If your thinly provisioned disk does not identify as thin disk in Windows check with your array vendor. Array firmware update might be needed to enable support.

Windows Server 2012 thin disk

Windows Server 2012 thin disk

  • You can read more about Windows Server 2012 unmap from here.
  • More information about Windows Server 2012/2012 R2 Defrag can be found here.

Windows Server 2003/2008/2008 R2

Older Windows versions does not include automatic UNMAP support. Third party tools can be used to reclaim space.

EMC StorReclaim

EMC StorReclaim is a command line utility to UNMAP allocated but unused space from storage array. It can be used in physical servers and virtual servers for LUNs which are presented directly from storage array. To get a the tool contact your EMC representative.

Supported operating systems:

  • Windows Server 2003 x86/x64
  • Windows Server 2008 x86/x64
  • Windows Server 2008 R2
  • Windows Server 2008 R2 with SP1
StorReclaim options

StorReclaim options


StorReclaim status


VMWare Guest Reclaim

VMWare Labs has published a free tool called Guest Reclaim that works in Windows 2003/2008/2008R/XP/7.

Guest Reclaim reclaims dead space from NTFS volumes hosted on a thin provisioned SCSI disk. The tool can also reclaim space from full disks and partitions, thereby wiping off the file systems on it. As the tool deals with active data, please take all precautionary measures understanding the SCSI UNMAP framework and backing up important data. (Source: VMWare)

In my environment I had problems when VM HW version was 8. It worked when VM HW version 9 or higher.

VMWare Guest Reclaim status

VMWare Guest Reclaim status

More information and downloads can be found here.

Commercial tool – Raxco PerfectStorage

Raxco PerfectStorage is an intelligent storage management solution for thin-provisioned space reclamation that improves productivity, allows for better storage management and forecasting, and delays future storage purchases. (Source: Raxco)

More information can be found from vendor homepage.

PerfectStorage is a agent based system. Agent is installed to operating system (virtual or physical) and then based on schedule SCSI UNMAP or zero fill tasks are executed. Supported operating system today is Windows only. What makes PerfectStorage interesting is that it identifies the space it can reclaim and issue reclaim operations only to those blocks. Most free tools issue reclaim commands to all free space. This can significantly reduce performance impact of reclaim operation. Everything is managed through central console. Also reporting of reclaimed space can be done in management console.


So those are some of the options how to reclaim dead space from virtual or physical Windows servers. I did not write about zero fill options in this post because this mostly requires some more operations from storage side. I will talk about zero fill options later in the post series.

If anyone knows more tools that can be used to reclaim space in Windows please let me know in the comments sections.

Other space reclaim posts in this series:

Storage reclamation – part 1 – VMWare vSphere

Storage reclamation – part 3 – Linux

Storage reclamation – part 4 – Zero fill and array level reclamation


Deduplicate VMWare virtual machines with Windows Server 2012 R2 – part 2

In my previous post I wrote how I used Windows Server 2012 R2 server to provide deduplicated storage for VMWare environment. I achieved some great deduplication results but I was not satisfied with performance. So I built another solution.


  • Physical Windows Server 2012 R2
  • 2×2 Gbit teamed NICs (HP Flex Adapter)
  • 2x 2,9Ghz Intel CPU
  • 192GB RAM
  • Storage Spaces disk pool
    • 100GB SSD
    • 4TB HDD

In to the pool I created one fixed 3TB disk with 92GB (usable SSD space) SSD Write-back cache. I used following powershell command to do it:

New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName vDisk1 -Size 3TB -ResiliencySettingName Simple -ProvisioningType Fixed -WriteCacheSize 92GB

With this setup all small random writes are landing on SSD disk. This ensures much better overall performance than HDD only solution.


During Storage vMotion IO is going to both disks. SSD takes most of the write IOs. Throughput is up to 400MB/sec with multiple concurrent Storage vMotions. Throughput slows down when cache gets full and data has to be destaged to HDD. After destage speed increases again.

NFS datastore performance


With this setup both read latency and write latency are acceptable. Datastore contains 87 running VMs. Dedupe ratio on the volume is 94%. About 30GB of RAM is used for file system cache.

My main goal was to to test if a Windows Server 2012 R2 is a viable option for some of the workloads running on VMWare. From performance perspective I think that this type of setup is viable to run VMs. From availability perspective there is still some things to be done e.g clustering.

Deduplicate VMWare virtual machines with Windows Server 2012 R2

I previously played with VMWare ESXi using NFS from Windows Server 2012. When Windows Server 2012 R2 became available I decided to add deduplication into the mix. For those who does not know Microsoft included a new feature in Windows Server 2012 called Data Deduplication. Data Deduplication enables you to dedupe files stored on NTFS volumes. Previously I was not able to run deduplication on Windows Server 2012 because it does not support deduplication for open files. Starting from Windows Server 2012 R2 Microsoft supports deduplication for open files.

For NFS share I decided to reuse already existing Windows Server 2012 R2 running on Hyper-V. This server is also used to provide deduplicated storage to Hyper-V hosts via SMB3. For NFS I created 500GB LUN and enabled deduplication (VDI mode). Since I was not sure about the performance of that deduplicated NFS datastore I decided to select VMs that did not have high I/O demand. I ended up moving 19 Linux based development servers to this NFS datastore.

Deduplication result is quite good. I achieved about 8x data reduction rate. It helped that all those 19 Linux servers are quite similar.

NFS statsPerformance numbers on the datastore are following:
IOPS: between 20-300 iops
Read Latency: < 5ms
Write Latency: <15ms

Write latency is not that good. But since those servers are only running some script jobs during the night no one has complained.

One thing to remember when using Windows Server based NFS with VMware is that when you move the VM to Windows Server based NFS datastore all the disks get inflated. So after storage vMotion VM will consume all the space it’s allocated. In my case I can not move any VMs which have been allocated more then 500 GBs to this datastore .

In future I’m planning to expand the usage of deduplicated storage for VMWare because it can deliver significant disk space and cost savings.

More information:

Introduction to Data Deduplication in Windows Server 2012

What’s New in Data Deduplication in Windows Server 2012 R2

WS2012 Storage – NFS Server – Configure NFS for VMware vSphere 5.1 Home Lab