NFS VAAI Statistics for NetApp Storage

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !!

 
In this blog I will discuss the NFS VAAI statistics that can be used on the NetApp storage to measure the performance and troubleshoot VAAI related issues. These statistics will help you determine if Copy Offload is being used by the Storage Array. In this blog I will cover 7 Mode and Clustered Data ONTAP.

On the NetApp Storage use the following command to monitor the NFS VAAI statistics. I have highlighted the important stats in RED throughout the blog. Note that I have deliberately removed some metrics from the output to make it more readable.

In general irrespective of the version of ONTAP, you can use sysstat –x 1 to monitor/measure the CPU, Memory, Disk, Network and other parameters. When VAAI primitives are used the Network utilization would be comparatively low compared to the disk usage because the clone & snapshots are now offloaded to the Storage Array. Hence resulting in reduced network usage between the ESXi hosts and the NetApp Storage Array. This command could give you some indication about Copy Offload and other primitives. However this may not be conclusive because there may be other workloads resulting in high network usage even when VAAI is being used. To precisely monitor the Copy Success and Errors use the following commands.

1. Data ONTAP 7 Mode – In 7 Mode there are two commands available that can be used to view the NFS VAAI statistics.

fas2040> nfs vstorage stats

NFS COL counters are :

                    Copy Reqs: 0

                   Abort Reqs: 0

                  Status Reqs: 0

                  Notify Reqs: 0

                  Revoke Reqs: 0

                Invalid Parms: 0

       Authorization Failures: 0

      Authentication Failures: 0

              Copy Fail ISDIR: 0

            Copy Fail OFFLINE: 0

              Copy Fail STALE: 0

                 Copy Fail IO: 0

            Copy Fail NOSPACE: 0

          Copy Fail DISKQUOTA: 0

           Copy Fail READONLY: 0

               Copy Fail PERM: 0

            Copy Fail EXPIRED: 0

           Copy Fail RESOURCE: 0

           Copy Fail TOOSMALL: 0

        Copy Fail BAD STATEID: 0

              Copy Fail OTHER: 0

               Intravol Moves: 0

               Intervol Moves: 0

               Fail Space RES: 0

fas2040> nfs stat

Server rpc:

TCP:

calls       badcalls    nullrecv    badlen      xdrcall

2           0           0           0           0

UDP:

calls       badcalls    nullrecv    badlen      xdrcall

0           0           0           0           0

IPv4:

calls       badcalls    nullrecv    badlen      xdrcall

2           0           0           0           0

IPv6:

calls       badcalls    nullrecv    badlen      xdrcall

0           0           0           0           0

Server nfs:

calls       badcalls

2           0

Server nfs V3: (2 calls)

null       getattr    setattr    lookup     access     readlink   read

2 100%     0 0%       0 0%       0 0%       0 0%       0 0%       0 0%

write      create     mkdir      symlink    mknod      remove     rmdir

0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%

rename     link       readdir    readdir+   fsstat     fsinfo     pathconf

0 0%       0 0%       0 0%       0 0%       0 0%       0 0%       0 0%

commit

0 0%

Read request stats (version 3)

0-511      512-1023   1K-2047    2K-4095    4K-8191    8K-16383   16K-32767  32K-65535  64K-131071 > 131071

0          0          0          0          0          0          0          0          0          0

Write request stats (version 3)

0-511      512-1023   1K-2047    2K-4095    4K-8191    8K-16383   16K-32767  32K-65535  64K-131071 > 131071

0          0          0          0          0          0          0          0          0          0

2. Clustered Data ONTAP 8.x

NOTE: For Clustered Data ONTAP 8.2 you have to execute this command from diagnostic mode and use statistics-v1 command to get the copy_manager statistics.

To enter diagnostic mode use the following:

cluster1::> set diag

Warning: These diagnostic commands are for use by NetApp personnel only.

Do you want to continue? {y|n}: y

cluster1::*>

cluster1::*> statistics-v1 show -node cluster1-01 -object copy_manager

For previous versions of Clustered Data ONTAP use the following:

cluster1::> statistics show -node cluster1-01 -object copy_manager

Node: cluster1-01

   Object.Instance.Counter                                 Value         Delta

   ----------------------------------------------- ------------- -------------

   copy_manager.copy_stats.instance_name             copy_stats

            -

   copy_manager.copy_stats.node_name                           -             -

   copy_manager.copy_stats.instance_uuid                       -             -

   copy_manager.copy_stats.copy_success                        1             -

   copy_manager.copy_stats.copy_failure                        0             -

   copy_manager.copy_stats.copyStatus_success                  0             -

   copy_manager.copy_stats.copyStatus_failure                  0             -

   copy_manager.copy_stats.copyAbort_success                   0             -

   copy_manager.copy_stats.copyAbort_failure                   0             -

   copy_manager.copy_stats.copyCallback_success                0             -

   copy_manager.copy_stats.copyCallback_failure                0             -

   copy_manager.copy_stats.copyNotify_success                  1             -

   copy_manager.copy_stats.copyNotify_failure                  0             -

   copy_manager.copy_stats.copyRevoke_success                  1             -

   copy_manager.copy_stats.copyRevoke_failure                  0             -

   copy_manager.copy_stats.copyAuthCheck_success               0             -

   copy_manager.copy_stats.copyAuthCheck_failure               0             -

   copy_manager.copy_stats.bytes_copied                        0             -

Node: cluster1-01

   Object.Instance.Counter                                 Value         Delta

   ----------------------------------------------- ------------- -------------

   copy_manager.copy_stats.intra_vol_copy_cnt                  1             -

   copy_manager.copy_stats.inter_vol_copy_cnt                  0             -

   copy_manager.copy_stats.inter_node_copy_cnt                 0             -

   copy_manager.copy_stats.inter_clust_copy_cnt                0             -

   copy_manager.copy_stats.fail_mem_alloc                      0             -

   copy_manager.copy_stats.fail_isdir                          0             -

   copy_manager.copy_stats.fail_offline                        0             -

   copy_manager.copy_stats.fail_stale                          0             -

   copy_manager.copy_stats.fail_io                             0             -

   copy_manager.copy_stats.fail_nospace                        0             -

   copy_manager.copy_stats.fail_readonly                       0             -

   copy_manager.copy_stats.fail_authcheck                      0             -

   copy_manager.copy_stats.fail_no_resource                    0             -

   copy_manager.copy_stats.fail_other                          0             -

   copy_manager.copy_stats.intra_volume_copy_success           1             -

   copy_manager.copy_stats.intra_volume_copy_failure           0             -

   copy_manager.copy_stats.intra_volume_copyStatus_success     0             -

   copy_manager.copy_stats.intra_volume_copyStatus_failure     0             -

   copy_manager.copy_stats.intra_volume_copyAbort_success      0             -

Node: cluster1-01

   Object.Instance.Counter                                 Value         Delta

   ----------------------------------------------- ------------- -------------

   copy_manager.copy_stats.intra_volume_copyAbort_failure      0             -

   copy_manager.copy_stats.inter_volume_copy_success           0             -

   copy_manager.copy_stats.inter_volume_copy_failure           0             -

   copy_manager.copy_stats.inter_volume_copyStatus_success     0             -

   copy_manager.copy_stats.inter_volume_copyStatus_failure     0             -

   copy_manager.copy_stats.inter_volume_copyAbort_success      0             -

   copy_manager.copy_stats.inter_volume_copyAbort_failure      0             -

   copy_manager.copy_stats.inter_volume_copyCallback_success   0             -

   copy_manager.copy_stats.inter_volume_copyCallback_failure   0             -

In addition to the above command you can also check the nps1 status to troubleshoot NFS VAAI related issues.

cluster1::> system node run -node cluster1-01 -command stats show nps1

nps1:nps1:instance_name:nps1

nps1:nps1:node_name:

nps1:nps1:instance_uuid:

nps1:nps1:null_success:0

nps1:nps1:null_error:0

nps1:nps1:compound_success:0

nps1:nps1:compound_error:0

nps1:nps1:access_success:0

nps1:nps1:access_error:0

nps1:nps1:verify_success:0

nps1:nps1:verify_error:0

nps1:nps1:write_success:0

nps1:nps1:write_error:0

nps1:nps1:set_ssv_error:0

nps1:nps1:test_stateid_success:0

nps1:nps1:test_stateid_error:0

nps1:nps1:want_delegation_success:0

nps1:nps1:want_delegation_error:0

nps1:nps1:destroy_clientid_success:0

nps1:nps1:destroy_clientid_error:0

nps1:nps1:reclaim_complete_success:0

nps1:nps1:reclaim_complete_error:0

nps1:nps1:copy_notify_success:1

nps1:nps1:copy_notify_error:0

nps1:nps1:copy_revoke_success:1

nps1:nps1:copy_revoke_error:0

nps1:nps1:copy_success:1

nps1:nps1:copy_error:0

nps1:nps1:copy_abort_success:0

nps1:nps1:copy_abort_error:0

nps1:nps1:copy_status_success:0

nps1:nps1:copy_status_error:0

esxtop Statistics for Block VAAI

Welcome: To stay updated with all my Blog posts follow me on Twitter @arunpande !!

 
We all know how important esxtop is while troubleshooting various vSphere related issues. In this blog I will share the esxtop metrics that you can use while troubleshooting various VAAI primitives. This will help you not only to diagnose VAAI related issues but will also help you measure the performance benefits that VAAI provides.  

To demonstrate this I have replicated some scenarios where VAAI is used so that I can capture the esxtop stats.

To access the esxtop metrics, login to the ESXi host using SSH

# esxtop

# press u for disk view

# press f to change fields

# press o for VAAI stats

# press p for VAAI latency stats

# press Enter 

Block Zero & Hardware Assisted Locking (ATS)

In this section we will cover the Block Zero VAAI primitive.

Scenario 1: Test BLOCK ZEROING primitive by creating a new Windows 2008 R2 VM with Lazy Zeroed Thick disk.

 

On monitoring the ZERO statistics I observed that it incremented from 4 to 7007 during the OS reinstallation.

Scenario 2: Test BLOCK ZEROING primitive by adding a new Eager Zeroed Thick virtual disk.

In this scenario I have added a 150 GB Eager Zeroed thick disk and on monitoring esxtop I observed that the ZERO statistics incremented from 7013 to 148020.

UNMAP

Scenario 3: You can either delete a VM or Storage vMotion the VM to a different datastore to demonstrate this.

We will now use the UNMAP primitive from the ESXi shell using the command

# esxcli storage vmfs unmap -l iscsi_2

On monitoring the esxtop I have observed that the DELETE statistics has increased to 52527.

Full Copy

In this section we will cover the Full Copy VAAI primitive.

Scenario 4: Test VAAI FULL COPY primitive, create multiple clones of the same VM.

In this scenario we will initiate a clone of a Windows 2008 R2 VM from vCenter. While monitoring the esxtop I have observed that the CLONE_RD & CLONE_WR statistics incremented. Note that MBC_RD/s & MBC_WR/s is the throughput for Full Copy Read & Write.

Scenario 5: Test VAAI FULL COPY primitive by relocating VM using Storage vMotion.

In this scenario we have migrated the windows VM to another iSCSI LUN that is being managed by the same controller in the same vServer. While monitoring esxtop I have observed that the CLONE_RD (source datasource), CLONE_WR (destination datastore), ATS, ZERO (destination datastore), AAVG (destination datastore) metrics were incremented.

To all VMware & NetApp Administrators go prepared when you walk into the War Room to discuss VAAI related (break/fix & performance) issues, all the best .