{"id":23184,"date":"2014-12-22T03:02:00","date_gmt":"2014-12-22T03:02:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/maoni\/2014\/12\/22\/gc-etw-events-1\/"},"modified":"2021-09-30T16:11:01","modified_gmt":"2021-09-30T23:11:01","slug":"gc-etw-events-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/gc-etw-events-1\/","title":{"rendered":"GC ETW events &#8211; 1"},"content":{"rendered":"<p><strong><span>GC ETW series &#8211;<\/span><\/strong><\/p>\n<p><a href='https:\/\/devblogs.microsoft.com\/dotnet\/gc-etw-events-1\/'><span>GC ETW Events &#8211; Part 1<\/span><\/a><span> (this post)<\/span><\/p>\n<p><a href='https:\/\/devblogs.microsoft.com\/dotnet\/gc-etw-events-2\/'><span>GC ETW Events &#8211; Part 2<\/span><\/a><\/p>\n<p><a href='https:\/\/devblogs.microsoft.com\/dotnet\/gc-etw-events-3\/'><span>GC ETW Events &#8211; Part 3<\/span><\/a><\/p>\n<p><a href='https:\/\/devblogs.microsoft.com\/dotnet\/gc-etw-events-4\/'><span>GC ETW Events &#8211; Part 4<\/span><\/a><\/p>\n<p><a href='https:\/\/devblogs.microsoft.com\/dotnet\/556-2\/'><span>Processing GC ETW Events Programmatically with the GLAD Library<\/span><\/a><\/p>\n<p><span>A lot of people have mentioned to me that I have not posted  anything for a long time. I do realize it and do appreciate being asked  to write more. Well, it&#39;s end of year and I am starting vacation so I thought I&#39;d write something light that perhaps makes good reading around Christmas time \ud83d\ude42<\/span><\/p>\n<p><strong><span>Perf counters vs ETW events<\/span><\/strong><span> <\/span><\/p>\n<p><span>Perf counters and ETW events, like anything else, each have their pros and cons. In my <\/span><a href='https:\/\/devblogs.microsoft.com\/dotnet\/the-defrag-tools-channel9-series-on-managed-memory-diagnostics\/'><span>Defrag Tools series of videos<\/span><\/a><span> I mentioned that I really would like folks to start using ETW (via the <\/span><a href='https:\/\/github.com\/microsoft\/perfview\/blob\/master\/documentation\/Downloading.md'><span>PerfView<\/span><\/a><span> tool) if they haven&#39;t. I would state that again here because ETW just  gives you richer info that you simply cannot get with perf counters. And if you only get the basic GC events, the overhead is very, very low  that you could afford to have it on in your production environment. So what&#39;s the con? Well, ETW is a newer thing (internally at MS many groups have started using ETW events, perhaps for a long time now; but clearly there are still groups that are using perf counters and not ETW events) which means there&#39;s a learning curve. And the learning curve of using  ETW is steeper than what you had with perf counters. I&#39;d like to think  that the PerfView made that fairly easy to get you started though. Aside from the less rich info, perf counters, as I explained <\/span><a href='https:\/\/devblogs.microsoft.com\/dotnet\/gc-performance-counters\/'><span>here<\/span><\/a><span>, also suffer from the precision problem. Just last week I had someone  asking me why he&#39;s seeing % time in GC being 99% for a few minutes  straight. It was because the % time in GC counter is only updated at the end of a GC and in his process since it was idling for a long time  after the last gen2 GC (ie, no GCs were happening), the 99% value lasted till next time a GC happened so to him this was misleading. <\/span><\/p>\n<p><strong><span>Collecting GC events with PerfView<\/span><\/strong><span> <\/span><\/p>\n<p><span>After you download perfview from Microsoft.com, you just unzip it and  run perfview.exe \u2013 yes it&#39;s that simple \u2013 there&#39;s no install involved.  There are many different options in perfview to collect ETW events with  but for our purpose we want to collect just some GC events to start  with. There are 2 ways you can do that: <\/span><\/p>\n<p><span>1) run perfview.exe, click on  Collect, then Collect again (or just do Alt+C). You will see a dialog  box popping up, click on Advanced options, uncheck \u201cKernel Base\u201d and  .NET and check \u201cGC Collect Only\u201d. This tells it to only collect events  fired <\/span><em><span>during<\/span><\/em><span> GCs. If you don&#39;t do this and simply click on \u201cStart  Collection\u201d it will collect these GC events as well, but a lot of other  events which makes it less sustainable if you aim to run this for many  hours on your production server as the problem will likely take that  long to repro. <\/span><\/p>\n<p><span>2) use the commandline \u2013 I am a commandline person so  this is what I use. It also makes it much easier to include in your  automation scripts. Run this commandline (if you run this from a non  admin cmd window it will pop up some UI asking for permission to run as  admin as collecting ETW requires admin privilege; or you just run it  from an admin cmd window): <\/span><\/p>\n<p><span>perfview \/GCCollectOnly \/AcceptEULA \/nogui  collect <\/span><\/p>\n<p><span>\/GCCollectOnly, as name suggests, is the one that&#39;s telling it  to collect only GC events. <\/span><\/p>\n<p><span>Then you run your scenario for long enough for it to show the symptoms. And you can stop it by either clicking on \u201cStop collection\u201d or press \u2018s&#39; in the cmd window to stop the  collection. If you know approximately how long it takes for the symptoms to show up, say 10 mins, you can also do <\/span><\/p>\n<p><span>perfview \/GCCollectOnly  \/AcceptEULA \/nogui \/MaxCollectSec:600 collect <\/span><\/p>\n<p><span>which will stop after it&#39;s collected for 600 seconds, ie, 10 mins. <\/span><\/p>\n<p><span>BTW, you can see all the  available commandline args in Help\\Command Line Help. This gives you 2  resulting files: PerfViewGCCollectOnly.etl.zip and  PerfViewGCCollectOnly.log.txt. The latter is logging what perfview did.  The former is what we will be diving into. <\/span><\/p>\n<p><strong><span>The GCStats view in PerfView<\/span><\/strong><span> <\/span><\/p>\n<p><span>Open the PerfViewGCCollectOnly.etl.zip file in perfview, meaning either by running Perfview and browsing to that directory and double clicking  on that file; or running the \u201cperfview PerfViewGCCollectOnly.etl.zip\u201d  commandline. You will see that there are multiple nodes under that  filename. What we are interested in is the GCStats view under the  \u201cMemory Group\u201d node. Double click on it to open it. At the top we have  something like this:<\/span><\/p>\n<p><strong><span>GCStats<\/span><\/strong><\/p>\n<ul>\n<li><span>Process 3768: WDExpress<\/span><\/li>\n<li><span>Process 6484: perfview \/GCCollectOnly \/AcceptEULA \/nogui collect<\/span><\/li>\n<li><span>Process 1108: sqlservr -sMSSQLSERVER <\/span><\/li>\n<\/ul>\n<p><span>I ran Visual Studio Express which is a  managed app \u2013 that&#39;s the WDExpress process at the top. For each of these processes it will have the following format: <\/span><\/p>\n<p><em><span>Summary<\/span><\/em><span> &#8211; this includes  things like the commandline, CLR startup flags, % time pause for GC and etc. <\/span><\/p>\n<p><em><span>GC stats rollup by generation<\/span><\/em><span> &#8211; for gen0\/1\/2 it has things like how  many GCs of that gen were done, their mean\/average pauses and etc. <\/span><\/p>\n<p><em><span>GC  stats for GCs whose pause time was &gt;200ms<\/span><\/em><span> <\/span><\/p>\n<p><em><span>LOH Allocation Pause (due  to background GC) &gt; 200 Msec for this process<\/span><\/em><span> <\/span><\/p>\n<p><em><span>Gen2 GC stats<\/span><\/em><span> <\/span><\/p>\n<p><em><span>All GC  stats<\/span><\/em><span> <\/span><\/p>\n<p><em><span>Condemned reasons for GCs<\/span><\/em><span> <\/span><\/p>\n<p><strong><span>Summary explanation<\/span><\/strong><\/p>\n<p><span>I will explain the summary section in this blog entry and the others in the next one.<\/span><\/p>\n<p><span>Commandline and runtime version are self-explanatory. <\/span><\/p>\n<p><span>CLR Startup Flags: the main thing for GC  investigation purpose is to look for CONCURRENT_GC and SERVER_GC. If you don&#39;t have either one it means you are using Workstation GC with  concurrent GC disabled. <\/span><\/p>\n<p><span>Total CPU Time and Total GC CPU Time: these would always be 0 unless you are actually collecting CPU samples. <\/span><\/p>\n<p><span>Total Allocs: total allocations you&#39;ve done during this trace for this  process. <\/span><\/p>\n<p><span>MSec\/MB Alloc: this is 0 unless you collect CPU samples (it  would be Total GC CPU time \/ Total allocated bytes). <\/span><\/p>\n<p><span>Total GC pause:  total wallclock time paused by GCs. Note that this includes suspension  time, ie, the time it takes to suspend managed threads before GC can  start. <\/span><\/p>\n<p><span>% Time paused for Garbage Collection: this is the total GC pause \/ total process running time. <\/span><\/p>\n<p><span>% CPU Time spent Garbage Collecting: this  is NaN% unless you collect CPU samples. <\/span><\/p>\n<p><span>Max GC Heap Size: maximum  managed heap size during this trace for this process. <\/span><\/p>\n<p><span>Something worth  mentioning is the difference between GC pauses and GC CPU time. The  former is elapsed time (ie, the time elapsed between suspension starts  and resumption ends for this GC) and the latter is, as the name  suggests, how many CPU samples we actually collected for this process during GC.  <\/span><\/p>\n<p><em><span>Edited on 03\/01\/2020 to add the indices to GC ETW Event blog entries<\/span><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>GC ETW series &#8211; GC ETW Events &#8211; Part 1 (this post) GC ETW Events &#8211; Part 2 GC ETW Events &#8211; Part 3 GC ETW Events &#8211; Part 4 Processing GC ETW Events Programmatically with the GLAD Library A lot of people have mentioned to me that I have not posted anything for a [&hellip;]<\/p>\n","protected":false},"author":3542,"featured_media":58792,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685],"tags":[3011,108],"class_list":["post-23184","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","tag-maoniposts","tag-performance"],"acf":[],"blog_post_summary":"<p>GC ETW series &#8211; GC ETW Events &#8211; Part 1 (this post) GC ETW Events &#8211; Part 2 GC ETW Events &#8211; Part 3 GC ETW Events &#8211; Part 4 Processing GC ETW Events Programmatically with the GLAD Library A lot of people have mentioned to me that I have not posted anything for a [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/23184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/3542"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=23184"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/23184\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/58792"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=23184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=23184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=23184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}