Getting started with vscsiStats
I have had a few occasions recently to start using vscsiStats. For those of you who may be unfamiliar, this is a great tool for virtual machine disk I/O workload characterization. Have you ever wondered about the most common I/O size generated by the Guest OS? What about the latency of those I/Os? What about checking to see the I/O generated by a Guest OS when it is in a so-called ‘idle’ state? vscsiStats can help with all of these queries, as well as providing some excellent troubleshooting options. The tool has been around since the ESX 3.5 days. This blog will take you through some of the steps in getting started with vscsiStats.
To get started, simply type vscsiStats at the ESXi command line. This will display the help/usage information. The useful options to begin with will be -s (start), -x (stop) & -l (list virtual machines and their disks). Let’s get the ball rolling by starting the data collections and displaying my VMs and VMDKs.
~ # vscsiStats -l
Virtual Machine worldGroupID: 76634, Virtual Machine Display Name: win-1, Virtual Machine Config File: /vmfs/volumes/vsan:5295b3cf23da4f38-33b970fc769c2c69/14a8cc51-f8f5-59fc-3584-1cc1de253de4/win-1.vmx, {
Virtual SCSI Disk handleID: 8192 (scsi0:0)
}
Virtual Machine worldGroupID: 81265, Virtual Machine Display Name: win-2, Virtual Machine Config File: /vmfs/volumes/vsan:5295b3cf23da4f38-33b970fc769c2c69/5fadcc51-e023-18a6-b53d-1cc1de252264/win-2.vmx, {
Virtual SCSI Disk handleID: 8193 (scsi0:0)
}
Let’s keep things simple to begin with. It appears that I have two virtual machines running on my ESXi host, each with a single VMDK. The information you will need here are the worldGroupID & Virtual SCSI Disk handleID. You need this information to start the data collection.
~ # vscsiStats -s -w 76634 -i 8192
vscsiStats: Starting Vscsi stats collection for worldGroup 76634, handleID 8192 (scsi0:0)
Success.
With the statistics now being collected, you can now decide on the type of histogram that you would like to display. The type of histogram can be one of: ‘all, ioLength, seekDistance, outstandingIOs, latency, interarrival’.
Let’s begin by displaying the histogram ‘ioLength’. The command is as follows (-c is for comma delimited output):
~ # vscsiStats -p ioLength -c -w 76634 -i 8192
Histogram: IO lengths of commands,virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,512
max,16384
mean,5341
count,37
Frequency,Histogram Bucket Limit
5,512
0,1024
0,2048
3,4095
23,4096
0,8191
1,8192
0,16383
5,16384
0,32768
0,49152
0,65535
0,65536
0,81920
0,131072
0,262144
0,524288
0,524288
Histogram: IO lengths of Read commands,virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,0
max,0
mean,0
count,0
Frequency,Histogram Bucket Limit
0,512
0,1024
0,2048
0,4095
0,4096
0,8191
0,8192
0,16383
0,16384
0,32768
0,49152
0,65535
0,65536
0,81920
0,131072
0,262144
0,524288
0,524288
Histogram: IO lengths of Write commands,virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,512
max,16384
mean,5341
count,37
Frequency,Histogram Bucket Limit
5,512
0,1024
0,2048
3,4095
23,4096
0,8191
1,8192
0,16383
5,16384
0,32768
0,49152
0,65535
0,65536
0,81920
0,131072
0,262144
0,524288
0,524288
~ #
This has been run against an ‘idle’ Windows VM, but we can still see a number of writes taking place, mostly 4KB writes. Note that there appears to be two entries for 512KB I/Os. The first is actual 512KB I/Os, the second is a bucket to capture I/Os > 512KB. A quick port of ioLength into a spreadsheet will produce a nice graph of the I/O sizes.
Lets display another histogram. This time lets look at the I/O latency on the same VM and the same disk.
~ # vscsiStats -p latency -c -w 76634 -i 8192
Histogram: latency of IOs in Microseconds (us),virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,945
max,6716
mean,1983
count,128
Frequency,Histogram Bucket Limit
0,1
0,10
0,100
0,500
6,1000
114,5000
8,15000
0,30000
0,50000
0,100000
0,100000
Histogram: latency of Read IOs in Microseconds (us),virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,0
max,0
mean,0
count,0
Frequency,Histogram Bucket Limit
0,1
0,10
0,100
0,500
0,1000
0,5000
0,15000
0,30000
0,50000
0,100000
0,100000
Histogram: latency of Write IOs in Microseconds (us),virtual machine worldGroupID,76634,virtual disk handleID,8192 (scsi0:0)
min,945
max,6716
mean,1983
count,128
Frequency,Histogram Bucket Limit
0,1
0,10
0,100
0,500
6,1000
114,5000
8,15000
0,30000
0,50000
0,100000
0,100000
~ #
Another quick port of this information into a spreadsheet, and we once again have a very nice graph of latency values for this virtual machine disk:
Again, we can see that the majority of the latency values is in the 5000 microsecond (or 5 millisecond) mark. When finished, remember to stop the stats collection with -x.
~ # vscsiStats -x -w 76634 -i 8192
vscsiStats: Stopping all Vscsi stats collection for worldGroup 76634, handleID 8192 (scsi0:0)
Success.
~ #
I think this is a very nice tool, and one that not many people may know about. I expect I’ll becoming more familiar with this tool over the coming months, and certainly want to play with the SCSI command traces feature. I’ll follow-up with another post once I’ve done that.
In the meantime, here are some additional references to vscsiStats to familiarize yourselves.
Community forums link (with links to other useful info):
http://communities.vmware.com/docs/DOC-10095
William Lam’s post on using APIs:
http://www.virtuallyghetto.com/2013/01/retrieving-vscsistats-using-vsphere-51.html
Andre & Paul Dunn have some pretty EXCEL macros for vscsiStats here:
http://myvirtualcloud.net/?p=1480
http://dunnsept.wordpress.com/2010/03/11/new-vscsistats-excel-macro/
A much underrated troubleshooting tool. 🙂
You can also create graphs using PowerShell and Microsoft Chart Controls. I haven’t tested it recently but it was very handy around the time that I wrote it.
http://www.vspecialist.co.uk/display-vscsistats-data-with-microsoft-chart-controls/
Thanks for covering this tool Cormac.
Please link to the paper from IISWC 2007 explaining vscsiStats. It’s a quick and easy read–if I do say so myself 🙂 and provides an essential primer for anyone who wants to do storage analysis for virtual machines.
http://www.vmware.com/files/pdf/iiswc_2007_distribute.pdf
Thanks Irfan. I didn’t link it directly as it is in the community doc links, but yes, a great presentation to get started on vscsiStats.
Cormac nice work bringing vscsiStats to peoples attention, I’ve used it a lot in the past, you can create some interesting pivot tables with the data.
The only problem is that the stats collected are limited to a maximum size of 512K. Windows 7 and 8 have a maximum block size of 1MB, is there any way you can ask what vscsiStats does with the 1MB block size?
Thanks
Jim
If you look at the vscsiStats histogram output, you’ll see that it has an entry for greater than 512K. The >512K entry counts all the IO larger than 512K at any size. So, for Windows Vista, 7, 8, etc., the tool still counts IOs in the largest bucket. Let me know if you have more questions.
Cormac: for the size histogram excel plot, consider adding the last histogram bucket for >512K. In your example case, that won’t do anything but as a reference for others, all the buckets are important to add. For example, see Jim Moyle’s comment.
Thanks Irfan – I’ll update that now.