{"id":25141,"date":"2019-11-02T00:54:28","date_gmt":"2019-11-02T07:54:28","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=25141"},"modified":"2019-11-02T00:54:28","modified_gmt":"2019-11-02T07:54:28","slug":"the-history-of-the-gc-configs","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/the-history-of-the-gc-configs\/","title":{"rendered":"The history of the GC configs"},"content":{"rendered":"<p>Recently, Nick from Stack Overflow <a href=\"https:\/\/twitter.com\/Nick_Craver\/status\/1189685719897460739\" rel=\"noopener noreferrer\" target=\"_blank\">tweeted<\/a> about his experience of using the .NET Core GC configs \u2013 he seemed quite happy with them (minus the fact they are not documented well which is something I\u2019m talking to our doc folks about). I thought it\u2019d be fun to tell you about the history of the GC configs \u2018cause it\u2019s almost the weekend and I want to contribute to your fun weekend reading.<\/p>\n<p>I started working on the GC toward the end of .NET 2.0. At that time we had really, really few public GC configs. \u201cPublic\u201d in this context means \u201cofficially supported\u201d. I\u2019m talking just <a href=\"https:\/\/docs.microsoft.com\/en-us\/dotnet\/framework\/configure-apps\/file-schema\/runtime\/gcserver-element\" rel=\"noopener noreferrer\" target=\"_blank\">gcServer<\/a> and <a href=\"https:\/\/docs.microsoft.com\/en-us\/dotnet\/framework\/configure-apps\/file-schema\/runtime\/gcconcurrent-element\" rel=\"noopener noreferrer\" target=\"_blank\">gcConcurrent<\/a>, which you could specify as application configs and I think the retain VM one which was exposed as a startup flag <a href=\"https:\/\/docs.microsoft.com\/en-us\/dotnet\/framework\/unmanaged-api\/hosting\/startup-flags-enumeration\" rel=\"noopener noreferrer\" target=\"_blank\">STARTUP_HOARD_GC_VM<\/a>, not an app config. Those of you who only worked with .NET Core may not have come across \u201capplication configs\u201d \u2013 that\u2019s a concept that only exists on .NET, not .NET Core. If you have an .exe called a.exe, and you have a file called a.exe.config in the same directory, then .NET will look at things you specify in this file under the runtime element for things like gcServer or some other non GC configs.<\/p>\n<p>At that time the official ways to configure the CLR were:<\/p>\n<ul>\n<li>App configs, under the runtime element<\/li>\n<\/ul>\n<ul>\n<li>Startup flags when you load the CLR via Hosting API, strictly speaking, you could use some of the hosting APIs to customize other aspects of GC like providing memory for the GC instead of having GC acquire memory via the OS VirtualAlloc\/VirtualFree APIs but I will not get into those \u2013 the only customer of those was SQLCLR AFAIK.<\/li>\n<\/ul>\n<p>(There were actually also other ways like the machine config but I will also not get into those as they have the same idea as the app configs)<\/p>\n<p>Of course there have always been the (now (fairly) famous) COMPlus environment variables. We used them only for internal testing &#8211; it\u2019s easy to specify env vars and our testing framework read them, set them and unset them as needed. There were actually not many of those either \u2013 one example was GCSegmentSize that was heavily used in testing (to test the expand heap scenario) but not officially supported so we never documented them as app configs.<\/p>\n<p>I was told that env vars were not a good customer facing way to config things because people tend to set them and forget to unset them and then later they wonder why they were seeing some unexpected behavior. And I did see that happen with some internal teams so this seemed like a reasonable reason.<\/p>\n<p>Startup flags are just a hosting API thing and hosting API was something few people heard of and way fewer used. You could say things like you want to start the runtime with Server GC and domain neutral. It\u2019s a native API and most of our customers refused to use it when they were recommended to try. Today I\u2019m aware of only one team who\u2019s actively using it \u2013 not surprisingly many people on that team used to work on SQLCLR \ud83d\ude1b<\/p>\n<p>For things you could specify as app configs you could also specify them with env vars or even registry values because on .NET our internal API to read these configs always check all 3 places. While we had a different kind of attitude toward configs you could specific via app config, which were considered officially supported, implementation wise this was great because devs didn\u2019t need to worry about which place the config would be read from \u2013 they knew if they added a new config in clrconfigvalues.h it could be specified via any of the 3 ways automatically.<\/p>\n<p>During the .NET 4.x timeframe We needed to add public configs for things like CPU group (we started seeing machines with > 64 procs) or creating objects of >2gb due to customer requests. Very few customers used these configs. So they could be thought of as special case configs, in other words, the majority of the scenarios were run with no configs aside from the gcServer\/gcConcurrent ones.<\/p>\n<p>I was pretty wary of adding new public configs. Adding internal ones was one thing but actually telling folks about them means we\u2019d basically be saying we are supporting them forever \u2013 in the older versions of .NET the compatibility bar was ultra high. And tooling was of course not as advanced then so perf analysis was harder to do (most of the GC configs were for perf).<\/p>\n<p>For a long time folks used the 2 major flavors of the GC, Server and Workstation, mostly according to the way they were designed. But you know how the rest of this story goes \u2013 folks didn\u2019t exactly use them \u201cas designed originally\u201d anymore. And as the GC diagnostic space also advanced customers were able to debug and understand GC perf better and also used .NET on larger, more stressful and more diverse scenarios. So there was more and more desire from them to do more configuration on their own.<\/p>\n<p>Good thing was Microsoft internally had plenty of customers who had very stressful workloads that called for configuration so I was able to test on these stressful real world scenarios. Around the time of .NET 4.6 I started adding configs more aggressively. One of our 1st party customers was running a scenario with many managed processes. They had configed some to use Server GC and others to use Workstation. But there was nothing inbetween. This was when configs like GCHeapCount\/GCNoAffinitize\/GCHeapAffinitizeMask were added.<\/p>\n<p>Around that time we also open sourced coreclr. The distinction of \u201cofficially supported configs\u201d vs internal only configs was still there \u2013 in theory that line had become a little blurry because our customers could see what internal configs we had \ud83d\ude42 but it also took time for Core adoption so I wasn\u2019t aware of really anyone who was using internal only configs. Also we changed the way config values were read \u2013 we no longer had the \u201cone API reads them all\u201d so today on Core where the \u201cofficial configs\u201d are specified via the runtimeconfig.json, you\u2019d need to use a different API and specify the name in the json and the name for the env var if you want to read from both.<\/p>\n<p>My development was still on CLR mostly, just because we had very few workloads on CoreCLR at that time and being able to try things on large\/stressful workloads was a very valuable thing. Around this time I added a few more configs for various scenarios \u2013 notable ones are GCLOHTheshold and GCHighMemPercent. A team had their product running in a process coexisting with a much large process on the same machine which had a lot of memory. So the default memory load that GC considered as \u201chigh memory load situation\u201d, which was 90%, worked well for the much larger process but not for them. When there\u2019s 10% physical memory left that was still a huge amount for their process so I added this for them to specify a higher value (they specified 97 or 98) which meant their process didn\u2019t need to do full compacting GCs nearly as often.<\/p>\n<p>Core 3.0 was when I unified the source between .NET and .NET Core so all the configs (\u201cinternal\u201d or not) from .NET were made available on Core as well. The json way is obviously the official way to specify a config but it appeared specifying configs via env vars was becoming more common, especially with folks who work on scenarios with high perf requirements. I know quite a few internal and external customers use them (and have yet to hear any incidents that involved setting an env var in an undesirable fashion). A few more GC configs were added during Core 3.0 \u2013 GCHeapHardLimit, GCLargePages, GCHeapAffinitizeRanges and etc.<\/p>\n<p>One thing that took folks (who used env vars) by surprise was the number you specific for a config in an env var format is interpreted as a hex number, not decimal. As far as why it was this way, it completely predates my time on the runtime team\u2026 since everyone remembered this for sure after they used it wrong the first time \ud83d\ude1b and it was an internal only thing, no one bothered to change it.<\/p>\n<p>I am still of the opinion that the officially supported configs should not require you to have internal GC knowledge. Of course internal is up for interpretation \u2013 some people might view anything beyond gcServer as internal knowledge. I\u2019m interpreting \u201cnot having internal GC knowledge\u201d in this context as \u201conly having general perf knowledge to influence the GC\u201d. For example, GCHeapHardLimit tells the GC how much memory it\u2019s allowed to use; GCHeapCount tells the GC how many cores it\u2019s allowed to use. Memory\/CPU usage are general perf knowledge that one already needs to have if they work on perf. GCLOHThreshold is actually violating this policy somewhat so it\u2019s something we\u2019d like to dynamically tune in GC instead of having users specify a number. But that\u2019s work we haven\u2019t done yet.<\/p>\n<p>I don\u2019t want to have configs that would need users to config things like \u201cif this generation\u2019s free list ratio or survival rate is > some threshold I would choose this particular GC to handle collections on that generation; but use this other GC to collect other generations\u201d. That to me is definitely \u201crequiring GC internal knowledge\u201d.<\/p>\n<p>So there you have it \u2013 the history of the GC configs in .NET\/.NET Core.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Recently, Nick from Stack Overflow tweeted about his experience of using the .NET Core GC configs \u2013 he seemed quite happy with them (minus the fact they are not documented well which is something I\u2019m talking to our doc folks about). I thought it\u2019d be fun to tell you about the history of the GC [&hellip;]<\/p>\n","protected":false},"author":3542,"featured_media":58792,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685],"tags":[],"class_list":["post-25141","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet"],"acf":[],"blog_post_summary":"<p>Recently, Nick from Stack Overflow tweeted about his experience of using the .NET Core GC configs \u2013 he seemed quite happy with them (minus the fact they are not documented well which is something I\u2019m talking to our doc folks about). I thought it\u2019d be fun to tell you about the history of the GC [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/25141","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/3542"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=25141"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/25141\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/58792"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=25141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=25141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=25141"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}