{"id":37819,"date":"2019-11-05T09:00:42","date_gmt":"2019-11-05T16:00:42","guid":{"rendered":"http:\/\/devblogs.microsoft.com\/premier-developer\/?p=37819"},"modified":"2019-11-01T09:05:03","modified_gmt":"2019-11-01T16:05:03","slug":"azure-vm-and-disk-throttling","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/azure-vm-and-disk-throttling\/","title":{"rendered":"Azure VM and Disk Throttling"},"content":{"rendered":"<p>App Dev Manager <a href=\"https:\/\/www.linkedin.com\/in\/omer-amin\/\">Omer Amin<\/a> describes an improved approach for monitoring disk throttling in Azure virtual machines.<\/p>\n<hr \/>\n<p>In my previous blog article (<a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/azure-ultra-disk-storage-is-here\/\">Azure Ultra Disk Storage is here<\/a>), I described a solution for monitoring disk throttling. After getting feedback from customers, I found that the performance was quite slow if you have many virtual machines. It took a lot of time to iterate through the performance data to find instances where the disk or virtual machines was throttled.<\/p>\n<p>To reduce run times, I found that I could drastically reduce end-to-end time by relying on Log Analytics to perform the work. This did require an additional step to upload the Disk and VM Throttling limits to a custom log in Log Analytics. However, the payload was so small that it did not really impact the total time. Log Analytics does require some additional <a href=\"https:\/\/docs.microsoft.com\/azure\/azure-monitor\/platform\/data-ingestion-time\">setup time<\/a> when data is first uploaded to a log. This means that running this script the first time may take up to 20-30 minutes to complete. This is a one-time delay and subsequent run times do not have this delay. This limitation was mitigated by adding a loop to wait and check every minute for the data to show up. Once the data is ready, the script can proceed and compute the results.<\/p>\n<p>See the updated script at <a href=\"https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS\">https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS<\/a><\/p>\n<p>The new version uses the following logic:<\/p>\n<ol>\n<li>Get list of all VMs and Disks in subscription.<\/li>\n<li>Get Azure Compute capabilities for each SKU using Azure REST API.<\/li>\n<li>Match Disks to each corresponding VM, add in the VM IOPS, Disk Bytes Limit.<\/li>\n<li>Post Disk\\VM data to Log Analytics.\n<ul>\n<li>Doing this for performance reasons. It is faster to run the throttling queries in Log Analytics compared to doing it locally.<\/li>\n<\/ul>\n<\/li>\n<li>Run a log analytics query that will output Disks and VMs that were throttled.<\/li>\n<li>Output results.<\/li>\n<\/ol>\n<p>I am including the full setup instructions here as well so that you can easily get started with this solution.<\/p>\n<h3>Connect VMs to Log Analytics Workspace<\/h3>\n<p>Create a new Log Analytics workspace, or you can use an existing one. Once you have a Log Analytics workspace, you will need to configure your VMs to point to this workspace.<\/p>\n<p><em>Note: data can take up to 30 minutes to arrive when you first connect the workspace.<\/em><\/p>\n<p><img decoding=\"async\" width=\"2456\" height=\"1645\" class=\"wp-image-37866\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-1.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-1.png 2456w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-1-300x201.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-1-768x514.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-1-1024x686.png 1024w\" sizes=\"(max-width: 2456px) 100vw, 2456px\" \/><\/p>\n<p>&nbsp;<\/p>\n<h3>Save the Workspace ID and Primary Key to a text file<\/h3>\n<p>You will need these values later in the PowerShell script.<\/p>\n<p><img decoding=\"async\" width=\"2534\" height=\"1639\" class=\"wp-image-37877\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-12.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-12.png 2534w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-12-300x194.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-12-768x497.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-12-1024x662.png 1024w\" sizes=\"(max-width: 2534px) 100vw, 2534px\" \/><\/p>\n<h3>Configure the Workspace<\/h3>\n<p>Once the Log Analytics workspace is setup, then you need to configure the Workspace to collect the \u201cPhysicalDisk\\Disk Bytes\/sec\u201d and \u201cPhysicalDisk\\Disk Transfers\/sec\u201d counters.<\/p>\n<p><em>Note: there is no space in PhysicalDisk. Make sure to have the correct spelling, otherwise data will not be collected properly.<\/em><\/p>\n<p><img decoding=\"async\" width=\"2442\" height=\"1546\" class=\"wp-image-37878\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-13.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-13.png 2442w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-13-300x190.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-13-768x486.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-13-1024x648.png 1024w\" sizes=\"(max-width: 2442px) 100vw, 2442px\" \/><\/p>\n<h3>Create a new Azure AD Application Registration<\/h3>\n<p>This is required so that you can make Azure REST API calls from PowerShell.<\/p>\n<p><img decoding=\"async\" width=\"2329\" height=\"1506\" class=\"wp-image-37879\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-14.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-14.png 2329w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-14-300x194.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-14-768x497.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-14-1024x662.png 1024w\" sizes=\"(max-width: 2329px) 100vw, 2329px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p><img decoding=\"async\" width=\"2127\" height=\"1624\" class=\"wp-image-37880\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-15.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-15.png 2127w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-15-300x229.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-15-768x586.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-15-1024x782.png 1024w\" sizes=\"(max-width: 2127px) 100vw, 2127px\" \/><\/p>\n<h3>Save the Application (Client) ID and Directory (Tenant) ID<\/h3>\n<p>You can see this on the App Registration page for your app.<\/p>\n<p><img decoding=\"async\" width=\"875\" height=\"819\" class=\"wp-image-37881\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-16.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-16.png 875w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-16-300x281.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-16-768x719.png 768w\" sizes=\"(max-width: 875px) 100vw, 875px\" \/><\/p>\n<h3>Create a Client Secret<\/h3>\n<p>Once the Azure AD App Registration is created, create a Client Secret that will be used for authentication. Save the Client Secret to a text file. You will need this later for the PowerShell script.<\/p>\n<p><img decoding=\"async\" width=\"2536\" height=\"1579\" class=\"wp-image-37882\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-17.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-17.png 2536w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-17-300x187.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-17-768x478.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-17-1024x638.png 1024w\" sizes=\"(max-width: 2536px) 100vw, 2536px\" \/><\/p>\n<h3>Assign Azure AD App Registration permissions<\/h3>\n<p>Give your new Azure AD App Registration permissions to connect to the Log Analytics workspace as a Contributor.<\/p>\n<p><img decoding=\"async\" width=\"2549\" height=\"1680\" class=\"wp-image-37883\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-18.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-18.png 2549w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-18-300x198.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-18-768x506.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-18-1024x675.png 1024w\" sizes=\"(max-width: 2549px) 100vw, 2549px\" \/><\/p>\n<p>Use your Azure AD App Registration name here.<\/p>\n<h3>Download the PowerShell script<\/h3>\n<p>Download the PowerShell script from <a href=\"https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS\/blob\/master\/DiskThrottle.ps1\">https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS\/blob\/master\/DiskThrottle.ps1<\/a><\/p>\n<p><img decoding=\"async\" width=\"905\" height=\"544\" class=\"wp-image-37884\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-19.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-19.png 905w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-19-300x180.png 300w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-19-768x462.png 768w\" sizes=\"(max-width: 905px) 100vw, 905px\" \/><\/p>\n<p>Update the values in the script from the values you saved earlier during setup.<\/p>\n<p>You can Subscription ID from the Subscriptions pane in the portal, or run <a href=\"https:\/\/docs.microsoft.com\/en-us\/powershell\/module\/az.accounts\/get-azsubscription?view=azps-2.7.0\">Get-AzSubscription<\/a> in PowerShell<\/p>\n<h3>Run the PowerShell script<\/h3>\n<p>You should see output similar to the following.<\/p>\n<p>At the end of the script, you can view the Disk and VM Throttling events by dumping the objects, or you can also export the results to CSV for further analysis.<\/p>\n<p>Display in PowerShell:<\/p>\n<ul>\n<li>$DiskThrottleResults<\/li>\n<li>$VMThrottleResults<\/li>\n<\/ul>\n<p>Export to CSV:<\/p>\n<ul>\n<li>Export-Csv -Path C:\\temp\\DiskThrottle.csv $DiskThrottleResults<\/li>\n<li>Export-Csv -Path C:\\temp\\VMThrottle.csv $VMThrottleResults<\/li>\n<\/ul>\n<p>Example:<\/p>\n<p><img decoding=\"async\" width=\"1433\" height=\"1760\" class=\"wp-image-37885\" src=\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-20.png\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-20.png 1433w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-20-244x300.png 244w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-20-768x943.png 768w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2019\/10\/word-image-20-834x1024.png 834w\" sizes=\"(max-width: 1433px) 100vw, 1433px\" \/><\/p>\n<h3>Troubleshooting &amp; Improvements<\/h3>\n<p>If you run into any issues, please open an <a href=\"https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS\/issues\">Issue<\/a>, or submit a <a href=\"https:\/\/github.com\/omeramin\/Azure-Disk-Throttling-PS\/pulls\">Pull Request<\/a> for any improvements.<\/p>\n<p>&nbsp;<\/p>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li style=\"list-style-type: none;\"><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"<p>Using Log Analytics to efficiently monitor Azure virtual machine disk throttling.<\/p>\n","protected":false},"author":582,"featured_media":37702,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25,128],"tags":[8040,3],"class_list":["post-37819","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-performance","tag-azure-virtual-machines","tag-team"],"acf":[],"blog_post_summary":"<p>Using Log Analytics to efficiently monitor Azure virtual machine disk throttling.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/37819","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=37819"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/37819\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/37702"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=37819"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=37819"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=37819"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}