Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !
VMware introduced vSphere APIs for Array Integration (VAAI) in vSphere 4.1 where VMkernel Data Mover uses certain API primitives to offload storage tasks from the ESXi host to the Storage Array. This results not only in reduced utilization of the ESXi host CPU, Memory, Network but also the load on the Fabric & Ethernet switches. VAAI also results in completing the tasks faster and more efficiently. Note that the VAAI primitives for Block & NAS storage are different. In vSphere 5.0 the NAS primitives and the new primitives for Block storage were introduced. The UNMAP primitive has been improved in vSphere 5.5.
In my previous blog posts I have shared information related to setting up, monitoring & troubleshooting VAAI. In this blog post I have summarized VAAI in general and also shared links to my other blog posts regarding VAAI.
Let’s discuss the VAAI primitives for block storage.
- Full Copy – This primitive uses Extended Copy (XCOPY) to offload data copy initiated during creating vCenter clone, Storage vMotion & deploying from VM templates. Using this primitive the ESXi host can offload copy to the storage array by providing the source and destination logical block address (LBA). This results in reduced utilization of CPU & FC switch.
- Block Zeroing – Block Zero primitive can be used by the ESXi host to zero large number of blocks. This is most commonly used while creating eager zeroed thick disk which is a requirement for VMs used for Fault Tolerance (FT) and MSCS (Microsoft Cluster Services). This results in creating virtual disks faster thus resulting in faster provisioning of VMs.
- Hardware Assisted Locking – This primitive is commonly known as Atomic Test & Sets (ATS). ATS is used for improved locking of the VMFS metadata when compared to traditional SCSI-2 reservations. ATS provides improved datastore scalability and efficiency.
- Thin Provisioning Stun (Introduced in vSphere 5.0) – This primitive prevents VMs on a thin provisioned datastore which has reached out-of-space from crashing. When a thin provisioned datastore has reached out-of-space Thin provisioning Stun pauses VMs with outstanding write I/Os to the disk while the remaining VMs which don’t have any write I/Os continue to run.
- Thin Provisioning Reclamation (Introduced in vSphere 5.0) – This primitive (commonly known as UNMAP) is used to reclaim dead space (storage blocks no longer used) on a thin provisioned LUN. Dead space is created on the LUN when VMs are deleted or Storage vMotion from the datastore. Using UNMAP the storage array release the dead space resulting in more free space on the LUN. Note that this is not automated and you have to manually run vmkfstools –y OR esxcli storage vmfs unmap command to invoke UNMAP.
The following table summarizes the block VAAI primitives and their use case.
VAAI Primitive
|
SCSI Command
|
Tasks Used
|
vSphere Advanced Setting (GUI)
|
vSphere Advanced Setting (CLI)
|
Full Copy
|
Extended Copy (XCOPY)
|
vCenter Clone, Storage vMotion
|
DataMover.HardwareAcceleratedMove
|
/DataMover/HardwareAcceleratedMove
|
Block Zero
|
Write Same
|
Creating eager zeroed & lazy zeroed virtual disks
|
DataMover.HardwareAcceleratedInit
|
/DataMover/HardwareAcceleratedInit
|
ATS
|
Compare & Write
|
Datastore locking to make VMFS metadata changes
|
VMFS3.HardwareAcceleratedLocking
|
/VMFS3/HardwareAcceleratedLocking
|
UNMAP
|
Space reclamation
|
VMFS3.EnableBlockDelete
|
/VMFS3/EnableBlockDelete
|
Let’s discuss the NFS VAAI primitives
- Full File Clone – This is similar to Full Copy block primitive where cloning operations are offloaded to the NAS Array. Note that this primitive does not work with Storage vMotion unlike Full Copy. Only when powered off VMs are migrated from one NFS datastore to another, this primitive is used.
- Fast File Clone/Native Snapshot Support – This primitive offloads VM snapshot creation to the NAS Array. VMware View Composer Array Integration (VCAI) uses this primitive to create linked clone pool desktops where hypervisor snapshots are replaced with Native Snapshots created by the NAS Array.
- Reserve Space – This primitive allows you to create eager zero OR lazy zero thick virtual disks. Without the support of this primitive you cannot create thick virtual disks on NFS datastores.
- Extended Statistics – This primitive provides additional visibility about space usage of the NFS datastores.
I have written the following blogs which provide more information about using VAAI with NetApp Storage.
- vStorage APIs for Array Integration (VAAI) & NetApp – How to set it right? – This blog post provides details about the different pre-checks to be performed before using Block & NAS VAAI with NetApp Storage.
- esxtop Statistics for Block VAAI – This blog post shares details about different esxtop statistics that can be used for block VAAI primitives. Read this blog to monitor the contribution using VAAI. It will help you do a performance assessment and also troubleshoot VAAI related issues.
- NFS VAAI Statistics for NetApp Storage – In this blog post I have shared the NFS VAAI counters that you can refer while troubleshooting VAAI offloaded tasks (Clone, Snapshots) on your storage.
- Troubleshooting NFS VAAI Plugin on NetApp Storage – In one of my blogs posts I have discussed that NFS VAAI plugin is not enabled by default on the ESXi host. This plugin is provided by your storage vendor which needs to be installed on the ESXi host. In this blog post I have discussed some commands and log files that you can use while troubleshooting NFS VAAI Plugin installation and NFS VAAI primitives.
- Create a custom ESXi ISO for NetApp NAS Plugin using PowerCLI and SAVE TIME !! – This blog post provides the PowerCLI cmdlets which enable you to create custom ISO with NetApp NFS VAAI Plugin. This reduces the initial setup times for the ESXi host because now you don’t need another reboot after installing the NFS VAAI Plugin.
- Using VAAI UNMAP on vSphere 5.5 & NetApp Storage – This blog post provides the details about changes in the VAAI UNMAP plugin. I have also demonstrated the use of UNMAP primitive using ESXi 5.5 and NetApp Storage.
VMware vSphere ® Storage APIs – Array Integration (VAAI) by Cormac Hogan from VMware.
Understanding and Using vStorage APIs for Array Integration and NetApp Storage by Peter Learmonth from NetApp.
Please leave a comment if you want me to cover a specific topic about VAAI on NetApp Storage. You can also follow me @arunpande to get updates about my blogs posts.
