Read all about VAAI (Overview, Setup, Troubleshooting, Monitoring) on NetApp Storage

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !

VMware introduced vSphere APIs for Array Integration (VAAI) in vSphere 4.1 where VMkernel Data Mover uses certain API primitives to offload storage tasks from the ESXi host to the Storage Array. This results not only in reduced utilization of the ESXi host CPU, Memory, Network but also the load on the Fabric & Ethernet switches. VAAI also results in completing the tasks faster and more efficiently. Note that the VAAI primitives for Block & NAS storage are different. In vSphere 5.0 the NAS primitives and the new primitives for Block storage were introduced. The UNMAP primitive has been improved in vSphere 5.5.

In my previous blog posts I have shared information related to setting up, monitoring & troubleshooting VAAI. In this blog post I have summarized VAAI in general and also shared links to my other blog posts regarding VAAI.

Let’s discuss the VAAI primitives for block storage.

• Full Copy – This primitive uses Extended Copy (XCOPY) to offload data copy initiated during creating vCenter clone, Storage vMotion & deploying from VM templates. Using this primitive the ESXi host can offload copy to the storage array by providing the source and destination logical block address (LBA). This results in reduced utilization of CPU & FC switch.

• Block Zeroing – Block Zero primitive can be used by the ESXi host to zero large number of blocks. This is most commonly used while creating eager zeroed thick disk which is a requirement for VMs used for Fault Tolerance (FT) and MSCS (Microsoft Cluster Services). This results in creating virtual disks faster thus resulting in faster provisioning of VMs.

• Hardware Assisted Locking – This primitive is commonly known as Atomic Test & Sets (ATS). ATS is used for improved locking of the VMFS metadata when compared to traditional SCSI-2 reservations. ATS provides improved datastore scalability and efficiency.

• Thin Provisioning Stun (Introduced in vSphere 5.0) – This primitive prevents VMs on a thin provisioned datastore which has reached out-of-space from crashing. When a thin provisioned datastore has reached out-of-space Thin provisioning Stun pauses VMs with outstanding write I/Os to the disk while the remaining VMs which don’t have any write I/Os continue to run.

• Thin Provisioning Reclamation (Introduced in vSphere 5.0) – This primitive (commonly known as UNMAP) is used to reclaim dead space (storage blocks no longer used) on a thin provisioned LUN. Dead space is created on the LUN when VMs are deleted or Storage vMotion from the datastore. Using UNMAP the storage array release the dead space resulting in more free space on the LUN. Note that this is not automated and you have to manually run vmkfstools –y OR esxcli storage vmfs unmap command to invoke UNMAP.

The following table summarizes the block VAAI primitives and their use case.

VAAI Primitive	SCSI Command	Tasks Used	vSphere Advanced Setting (GUI)	vSphere Advanced Setting (CLI)
Full Copy	Extended Copy (XCOPY)	vCenter Clone, Storage vMotion	DataMover.HardwareAcceleratedMove	/DataMover/HardwareAcceleratedMove
Block Zero	Write Same	Creating eager zeroed & lazy zeroed virtual disks	DataMover.HardwareAcceleratedInit	/DataMover/HardwareAcceleratedInit
ATS	Compare & Write	Datastore locking to make VMFS metadata changes	VMFS3.HardwareAcceleratedLocking	/VMFS3/HardwareAcceleratedLocking
UNMAP		Space reclamation	VMFS3.EnableBlockDelete	/VMFS3/EnableBlockDelete

Let’s discuss the NFS VAAI primitives

• Full File Clone – This is similar to Full Copy block primitive where cloning operations are offloaded to the NAS Array. Note that this primitive does not work with Storage vMotion unlike Full Copy. Only when powered off VMs are migrated from one NFS datastore to another, this primitive is used.

• Fast File Clone/Native Snapshot Support – This primitive offloads VM snapshot creation to the NAS Array. VMware View Composer Array Integration (VCAI) uses this primitive to create linked clone pool desktops where hypervisor snapshots are replaced with Native Snapshots created by the NAS Array.

• Reserve Space – This primitive allows you to create eager zero OR lazy zero thick virtual disks. Without the support of this primitive you cannot create thick virtual disks on NFS datastores.

• Extended Statistics – This primitive provides additional visibility about space usage of the NFS datastores.

I have written the following blogs which provide more information about using VAAI with NetApp Storage.

• vStorage APIs for Array Integration (VAAI) & NetApp – How to set it right? – This blog post provides details about the different pre-checks to be performed before using Block & NAS VAAI with NetApp Storage.

• esxtop Statistics for Block VAAI – This blog post shares details about different esxtop statistics that can be used for block VAAI primitives. Read this blog to monitor the contribution using VAAI. It will help you do a performance assessment and also troubleshoot VAAI related issues.

• NFS VAAI Statistics for NetApp Storage – In this blog post I have shared the NFS VAAI counters that you can refer while troubleshooting VAAI offloaded tasks (Clone, Snapshots) on your storage.

• Troubleshooting NFS VAAI Plugin on NetApp Storage – In one of my blogs posts I have discussed that NFS VAAI plugin is not enabled by default on the ESXi host. This plugin is provided by your storage vendor which needs to be installed on the ESXi host. In this blog post I have discussed some commands and log files that you can use while troubleshooting NFS VAAI Plugin installation and NFS VAAI primitives.
• Create a custom ESXi ISO for NetApp NAS Plugin using PowerCLI and SAVE TIME !! – This blog post provides the PowerCLI cmdlets which enable you to create custom ISO with NetApp NFS VAAI Plugin. This reduces the initial setup times for the ESXi host because now you don’t need another reboot after installing the NFS VAAI Plugin.

• Using VAAI UNMAP on vSphere 5.5 & NetApp Storage – This blog post provides the details about changes in the VAAI UNMAP plugin. I have also demonstrated the use of UNMAP primitive using ESXi 5.5 and NetApp Storage.
   

This blog provides and overview about VAAI for detailed information refer to the following documents. I have also used them as a reference while writing these blog posts.

VMware vSphere ® Storage APIs – Array Integration (VAAI)  by Cormac Hogan from VMware.
Understanding and Using vStorage APIs for Array Integration and NetApp Storage by Peter Learmonth from NetApp.

Please leave a comment if you want me to cover a specific topic about VAAI on NetApp Storage. You can also follow me @arunpande to get updates about my blogs posts. 

esxtop Statistics for Block VAAI

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !!

 
We all know how important esxtop is while troubleshooting various vSphere related issues. In this blog I will share the esxtop metrics that you can use while troubleshooting various VAAI primitives. This will help you not only to diagnose VAAI related issues but will also help you measure the performance benefits that VAAI provides.  

To demonstrate this I have replicated some scenarios where VAAI is used so that I can capture the esxtop stats.

To access the esxtop metrics, login to the ESXi host using SSH

# esxtop

# press u for disk view

# press f to change fields

# press o for VAAI stats

# press p for VAAI latency stats

# press Enter 

Block Zero & Hardware Assisted Locking (ATS)

In this section we will cover the Block Zero VAAI primitive.

Scenario 1: Test BLOCK ZEROING primitive by creating a new Windows 2008 R2 VM with Lazy Zeroed Thick disk.

 

On monitoring the ZERO statistics I observed that it incremented from 4 to 7007 during the OS reinstallation.

Scenario 2: Test BLOCK ZEROING primitive by adding a new Eager Zeroed Thick virtual disk.

In this scenario I have added a 150 GB Eager Zeroed thick disk and on monitoring esxtop I observed that the ZERO statistics incremented from 7013 to 148020.

UNMAP

Scenario 3: You can either delete a VM or Storage vMotion the VM to a different datastore to demonstrate this.

We will now use the UNMAP primitive from the ESXi shell using the command

# esxcli storage vmfs unmap -l iscsi_2

On monitoring the esxtop I have observed that the DELETE statistics has increased to 52527.

Full Copy

In this section we will cover the Full Copy VAAI primitive.

Scenario 4: Test VAAI FULL COPY primitive, create multiple clones of the same VM.

In this scenario we will initiate a clone of a Windows 2008 R2 VM from vCenter. While monitoring the esxtop I have observed that the CLONE_RD & CLONE_WR statistics incremented. Note that MBC_RD/s & MBC_WR/s is the throughput for Full Copy Read & Write.

Scenario 5: Test VAAI FULL COPY primitive by relocating VM using Storage vMotion.

In this scenario we have migrated the windows VM to another iSCSI LUN that is being managed by the same controller in the same vServer. While monitoring esxtop I have observed that the CLONE_RD (source datasource), CLONE_WR (destination datastore), ATS, ZERO (destination datastore), AAVG (destination datastore) metrics were incremented.

To all VMware & NetApp Administrators go prepared when you walk into the War Room to discuss VAAI related (break/fix & performance) issues, all the best .