Questions while designing Storage for VMware - Part 1

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !

When you evaluate your IT expenses you would find that Storage covers a sizable amount hence it’s very important that you design the storage effectively. I wrote this blog post to make a note of some important points and questions that should be considered while designing your storage infrastructure. The contents in this blog can also be used for evaluating your current storage technology, identifying and bridging the gaps during the storage refresh. 

Capacity: While sizing the storage consider the usable

storage and not RAW storage because an appreciable size of RAW storage would be used by Raid overhead, hot spares, snapshot best practices etc. Also keep the data growth for the next 3-5 years when you size the storage. The storage solution should allow you to choose different types of disks (SSD, SAS, SATA) with different speeds. Ask the following questions:

How much usable capacity do I need in my infrastructure?

Does the storage capacity address the IOPS requirement?

What is the average rate of growth of data for the next 3-5 years?

What is the estimated size required for hot data and how will this be addressed for e.g. SSD/Flash?
Additional space required for VM swap & VM snapshots?

Storage Architecture, Scalability & Flexibility: For the past few years we have seen some noticeable changes in the storage domain for e.g. most vendors now provide Unified Storage solution which aims at covering multi-protocol support (FC, FCoE, iSCSI & NFS). This gives the flexibility to use the any protocol without making major changes in your storage setup. Along with Unified Storage there is also a lot of noise about Converged Infrastructure which aims at providing industry standard, certified and tested end-to-end solution for your IT needs. This covers Compute, Networking, Storage and OS & Application best practices. Ask the following questions when you are evaluating these solutions:

What is the maximum capacity that is supported by the Storage Array?

What are the different disk drives (speed & protocol) supported by this solution?

What is the impact to the existing environment while upgrading Storage or adding more Storage Controllers?

Understand the architecture of Unified Storage solution, does it provide equal capabilities/features for both block & file storage?

Does the storage array provide same OS, management entities for File & Block storage?

High Availability: HA is considered as one of the key factors while designing your storage infrastructure. While designing HA one should first understand the impact of downtime for the business. You should have a clear understanding about the Mean time between failure (MTBF) and Mean time to repair (MTTR). It’s MTTR that will be a measure of downtime and it should not exceed the accepted Recovery Time Objective (RTO) as per the Service Level Agreement (SLA). In simple words Availability is the percentage of time when the Storage was available. Ask the following questions:

How many “Nines” do you need in your Infrastructure? Generally 99.999% or 99.9999% is desired for business which can withstand a downtime of few seconds to few minutes. The solution should not have any Single Point of Failure (SPOF) and this should cover all the components in your storage i.e. Controller or SP, Cache Cards, data in NVRAM, Disk Shelves, Disk Drives, Fans, Power Supplies etc.

How will the data in NVRAM or Cache be handled during power failure?

Have you considered the specific tuning/settings to be done on the OS or Application to handle failover/failback during controller or path failure?

Is any downtime required to perform any storage maintenance tasks like firmware upgrading, adding more storage, replacing disk drives etc.

Automate vSphere Networking (Create VMkernel & VM Portgroups) using PowerCLI

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !

This blog post provides a simple and easy to use PowerCLI script which allows vSphere administrators & consultants to perform the following tasks AUTOMATICALLY !

• Add multiple ESXi hosts in a cluster
• Create vSphere Standard Switch
• Assign Physical Network cards
• Create Virtual Machine portgroups and assign VLAN
• Create VMkernel Port groups
• Assign IP address, Netmask & Gateway to the VMkernel ports

I have provided comments before each script to help you understand the script better. Please leave a comment if you have any questions.

#Create an Empty array which will save all the values in $esxi variable, 
#you can input values for #ESXi host IP/FQDN using multiple. 
#In this example I have used Read-Host option.

$esxi = @()

#Enter the IP/FQDN of the ESXi servers and this script will 
#continue to prompt you to add hosts untill you stop entering them.

do {
  $input =  Read-Host "Enter the name or IP of the ESXi server"
  if ($input -ne ""){
      $esxi += $input
  }
 }
until ($input -eq "")

#The ESXi hosts that you add in the above step are saved in an array. 
#The below commandlet lets you addd these ESXi host to the cluster "DR-Stie". 
#The credentials have been pre-entered.

foreach ($name in $esxi)
{
add-vmhost -Name $name -location DR-Site -user root -password XXXXXX -force: $true
}

#The below cmdlet allows you to define the last octet for VMkernel interfaces 
#that you want to configure. 
#For example I have used 101 as the last octet for VMkernel interfaces used for 
#vMotion & NFS. Though they would belong to different Subnets & VLANs.

    $vMotionip = 101
    $nfsip = 101

#I am using a for-each loop to perform the below tasks on all the ESXi hosts 
#reported using the get-vmhost cmdlet. 
#Here we are querying all the ESXi host managed by vCenter. 
#You can modify this to query only specific ESXi hosts in cluster.

    foreach ($vmhost in (get-vmhost))
    {

#Here I am defining a variable $vmnic with value of vmnic1. 
#You can use multiple vmnics by separating them using ","

    $vmnic = vmnic1

#Here is am defining a variable $vs for vSiwtch2 while creating the vSwitch.
    
    $vs =  New-VirtualSwitch -VMHost $vmHost -Name vSwitch2

#Here I am creating a new virtual machine portgroup named "VLAN-600-Std-Switch" 
#and assigning it VLAN 600. 
#The output of this cmdlet is saved in $vlan

    $vlan600 =  New-VirtualPortGroup -VirtualSwitch $vs -Name VLAN-600-Std-Switch -VLanId 600

# I have created a new virtual machine portgroup named "Infra".

    New-VirtualPortGroup -VirtualSwitch $vs -Name Infra -Vlanid 0

# I have created a new virtual machine portgroup named "Infra".

    $vmotion = New-VirtualPortGroup -VirtualSwitch $vs -Name vMotion -VLanId 500

# I have created a new virtual machine portgroup named "NFS" and assigned it VLAN 400. 
#The output of this cmdlet is saved in $nfs

    $nfs = New-VirtualPortGroup -VirtualSwitch $vs -Name NFS -VLanId 400

# We are now creating a VMkernel network interface that will be used for vMotion. 
#I am using the vMotionip+=1 which will increment the last octet of the IP by one.

    New-VMHostNetworkAdapter -VMHost $vmhost -PortGroup $vmotion 
    -VirtualSwitch $vs -IP 192.168.50.$vMotionip -SubnetMask 255.255.255.0 
    -VMotionEnabled: $true

    $vMotionip+=1

# We are now creating a VMkernel network interface that will be used for NFS. 
#I am using the nfsip+=1 which will increment the last octet of the IP by one.

    New-VMHostNetworkAdapter -VMHost $vmhost -PortGroup $nfs 
    -VirtualSwitch $vs -IP 192.168.40.$nfsip -SubnetMask 255.255.255.0 
    -VMotionEnabled: $false

    $nfsip+=1

    }

Read all about VAAI (Overview, Setup, Troubleshooting, Monitoring) on NetApp Storage

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !

VMware introduced vSphere APIs for Array Integration (VAAI) in vSphere 4.1 where VMkernel Data Mover uses certain API primitives to offload storage tasks from the ESXi host to the Storage Array. This results not only in reduced utilization of the ESXi host CPU, Memory, Network but also the load on the Fabric & Ethernet switches. VAAI also results in completing the tasks faster and more efficiently. Note that the VAAI primitives for Block & NAS storage are different. In vSphere 5.0 the NAS primitives and the new primitives for Block storage were introduced. The UNMAP primitive has been improved in vSphere 5.5.

In my previous blog posts I have shared information related to setting up, monitoring & troubleshooting VAAI. In this blog post I have summarized VAAI in general and also shared links to my other blog posts regarding VAAI.

Let’s discuss the VAAI primitives for block storage.

• Full Copy – This primitive uses Extended Copy (XCOPY) to offload data copy initiated during creating vCenter clone, Storage vMotion & deploying from VM templates. Using this primitive the ESXi host can offload copy to the storage array by providing the source and destination logical block address (LBA). This results in reduced utilization of CPU & FC switch.

• Block Zeroing – Block Zero primitive can be used by the ESXi host to zero large number of blocks. This is most commonly used while creating eager zeroed thick disk which is a requirement for VMs used for Fault Tolerance (FT) and MSCS (Microsoft Cluster Services). This results in creating virtual disks faster thus resulting in faster provisioning of VMs.

• Hardware Assisted Locking – This primitive is commonly known as Atomic Test & Sets (ATS). ATS is used for improved locking of the VMFS metadata when compared to traditional SCSI-2 reservations. ATS provides improved datastore scalability and efficiency.

• Thin Provisioning Stun (Introduced in vSphere 5.0) – This primitive prevents VMs on a thin provisioned datastore which has reached out-of-space from crashing. When a thin provisioned datastore has reached out-of-space Thin provisioning Stun pauses VMs with outstanding write I/Os to the disk while the remaining VMs which don’t have any write I/Os continue to run.

• Thin Provisioning Reclamation (Introduced in vSphere 5.0) – This primitive (commonly known as UNMAP) is used to reclaim dead space (storage blocks no longer used) on a thin provisioned LUN. Dead space is created on the LUN when VMs are deleted or Storage vMotion from the datastore. Using UNMAP the storage array release the dead space resulting in more free space on the LUN. Note that this is not automated and you have to manually run vmkfstools –y OR esxcli storage vmfs unmap command to invoke UNMAP.

The following table summarizes the block VAAI primitives and their use case.

VAAI Primitive	SCSI Command	Tasks Used	vSphere Advanced Setting (GUI)	vSphere Advanced Setting (CLI)
Full Copy	Extended Copy (XCOPY)	vCenter Clone, Storage vMotion	DataMover.HardwareAcceleratedMove	/DataMover/HardwareAcceleratedMove
Block Zero	Write Same	Creating eager zeroed & lazy zeroed virtual disks	DataMover.HardwareAcceleratedInit	/DataMover/HardwareAcceleratedInit
ATS	Compare & Write	Datastore locking to make VMFS metadata changes	VMFS3.HardwareAcceleratedLocking	/VMFS3/HardwareAcceleratedLocking
UNMAP		Space reclamation	VMFS3.EnableBlockDelete	/VMFS3/EnableBlockDelete

Let’s discuss the NFS VAAI primitives

• Full File Clone – This is similar to Full Copy block primitive where cloning operations are offloaded to the NAS Array. Note that this primitive does not work with Storage vMotion unlike Full Copy. Only when powered off VMs are migrated from one NFS datastore to another, this primitive is used.

• Fast File Clone/Native Snapshot Support – This primitive offloads VM snapshot creation to the NAS Array. VMware View Composer Array Integration (VCAI) uses this primitive to create linked clone pool desktops where hypervisor snapshots are replaced with Native Snapshots created by the NAS Array.

• Reserve Space – This primitive allows you to create eager zero OR lazy zero thick virtual disks. Without the support of this primitive you cannot create thick virtual disks on NFS datastores.

• Extended Statistics – This primitive provides additional visibility about space usage of the NFS datastores.

I have written the following blogs which provide more information about using VAAI with NetApp Storage.

• vStorage APIs for Array Integration (VAAI) & NetApp – How to set it right? – This blog post provides details about the different pre-checks to be performed before using Block & NAS VAAI with NetApp Storage.

• esxtop Statistics for Block VAAI – This blog post shares details about different esxtop statistics that can be used for block VAAI primitives. Read this blog to monitor the contribution using VAAI. It will help you do a performance assessment and also troubleshoot VAAI related issues.

• NFS VAAI Statistics for NetApp Storage – In this blog post I have shared the NFS VAAI counters that you can refer while troubleshooting VAAI offloaded tasks (Clone, Snapshots) on your storage.

• Troubleshooting NFS VAAI Plugin on NetApp Storage – In one of my blogs posts I have discussed that NFS VAAI plugin is not enabled by default on the ESXi host. This plugin is provided by your storage vendor which needs to be installed on the ESXi host. In this blog post I have discussed some commands and log files that you can use while troubleshooting NFS VAAI Plugin installation and NFS VAAI primitives.
• Create a custom ESXi ISO for NetApp NAS Plugin using PowerCLI and SAVE TIME !! – This blog post provides the PowerCLI cmdlets which enable you to create custom ISO with NetApp NFS VAAI Plugin. This reduces the initial setup times for the ESXi host because now you don’t need another reboot after installing the NFS VAAI Plugin.

• Using VAAI UNMAP on vSphere 5.5 & NetApp Storage – This blog post provides the details about changes in the VAAI UNMAP plugin. I have also demonstrated the use of UNMAP primitive using ESXi 5.5 and NetApp Storage.
   

This blog provides and overview about VAAI for detailed information refer to the following documents. I have also used them as a reference while writing these blog posts.

VMware vSphere ® Storage APIs – Array Integration (VAAI)  by Cormac Hogan from VMware.
Understanding and Using vStorage APIs for Array Integration and NetApp Storage by Peter Learmonth from NetApp.

Please leave a comment if you want me to cover a specific topic about VAAI on NetApp Storage. You can also follow me @arunpande to get updates about my blogs posts. 

Troubleshooting NetApp NFS VAAI Plugin on ESXi host

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !!

 
In this blog I will discuss some commands and logs that you can use while troubleshooting NetApp NFS VAAI plugin on ESXi host.

• Ensure that the specific ONTAP version supports the NFS VAAI primitives for the corresponding ESXi hosts. Note that NFS VAAI plugins were introduced in vSphere 5.0. To check the compatibility use How to determine if VAAI features are being used in a vSphere and NetApp environment? I have also provided more information in my previous blog vStorage APIs for Array Integration (VAAI) & NetApp – How to set it right?

• Check if NFS VAAI Plugin is installed, you can use the following command

~ # esxcli software vib list | grep -i netapp

NetAppNasPlugin                1.0-020                             NetApp  VMwareAccepted    2013-06-05

OR

~ # esxcli software vib get -n NetAppNasPlugin

NetApp_bootbank_NetAppNasPlugin_1.0-020

Name: NetAppNasPlugin

Version: 1.0-020

Type: bootbank

Vendor: NetApp

Acceptance Level: VMwareAccepted

Summary: NAS VAAI NetApp Plugin

Description: NetApp NAS VAAI Module for ESX Server

ReferenceURLs:

Creation Date: 2012-10-29

Depends:

Conflicts:

Replaces:

Provides:

Maintenance Mode Required: False

Hardware Platforms Required:

Live Install Allowed: False

Live Remove Allowed: False

Stateless Ready: False

Overlay: False

Tags:

Payloads: NetAppNasPlugin

• If you are using ESXi CLI or NetApp Virtual Storage Console to install NFS VAAI plugin but the installation does not complete successfully, use the /var/log/esxupdate.log to find more information.

• To troubleshoot issues related to various NFS VAAI primitives like copy offload use the /var/log/hostd.log.