{"id":27854,"date":"2021-04-07T23:31:51","date_gmt":"2021-04-07T23:31:51","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cppblog\/?p=27854"},"modified":"2021-04-07T23:31:51","modified_gmt":"2021-04-07T23:31:51","slug":"finding-bugs-with-addresssanitizer-msvc-compiler","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/finding-bugs-with-addresssanitizer-msvc-compiler\/","title":{"rendered":"Finding Bugs with AddressSanitizer: MSVC Compiler"},"content":{"rendered":"<p><em>Special thanks to Aaron Gorenstein for authoring this blog post. <\/em><\/p>\n<p>The AddressSanitizer (ASan) is <a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/address-sanitizer-for-msvc-now-generally-available\/\">generally available<\/a> for MSVC since the recently-released Visual Studio 2019 version 16.9. We\u2019ve already <a href=\"https:\/\/www.youtube.com\/watch?v=4YzFdiYPeVw&amp;t=4065s\">shown<\/a> how easy it can be to find bugs in even production-ready code like EASTL. Here I\u2019ll share an example of how it found a real bug in the MSVC compiler itself.<\/p>\n<p>The idea was straightforward: ASan finds bugs, and we\u2019re always interested in finding bugs in the compiler. Just like you can turn ASan on in your projects and run your tests, we\u2019ve been turning on ASan on our project (the compiler) and running it on our tests. Sure enough, this found bugs.<\/p>\n<p>&nbsp;<\/p>\n<h3>Building our Binary with ASan<\/h3>\n<p>It was easy to turn on ASan in our build system. We\u2019ve documented ways to turn ASan on in <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/sanitizers\/asan?view=msvc-160#ide-msbuild\">common<\/a> <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/sanitizers\/asan?view=msvc-160#ide-cmake\">build<\/a> scenarios. In our case, I added <code>\/fsanitize=address<\/code> to the build\u2019s cl.exe command line, and our old, ever-evolving build system needed the extra manual step of specifying where our <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/sanitizers\/asan-debugger-integration?view=msvc-160\">extension<\/a> library lived.<\/p>\n<p>That was all it took! I now was able to build my binary, c2.dll, \u201cjust like normal\u201d, but now it had lots of excellent ASan instrumentation imbued to help find bugs. I was ready to run our inner-ring test suite and see if anything cropped up.<\/p>\n<p>&nbsp;<\/p>\n<h3>Finding the Bug<\/h3>\n<p>Our inner-test loop is about 4,000 separate C++ files, containing a mix of real-world-code, synthetic tests, benchmarks, and regression tests. We have a home-made test-runner that is only accessible from the command-line. Running it, we almost passed, but hit exactly 1 failure. I looked in our log file and see the characteristic trace:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-27861\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1.png\" alt=\"Image ASan MSVC error - stack buffer underflow\" width=\"1875\" height=\"253\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1.png 1875w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1-300x40.png 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1-1024x138.png 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1-768x104.png 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-1-1536x207.png 1536w\" sizes=\"(max-width: 1875px) 100vw, 1875px\" \/><\/a><\/p>\n<p>A couple things I would like to highlight:<\/p>\n<ol>\n<li>The error reported is \u201cstack-buffer-underflow\u201d: this is a <em>stack <\/em> ASan is able to find both stack and heap issues.<\/li>\n<li>Note the line \u201cstack of thread T3\u201d. As that suggests, there is also a T1 and T2 (and more): c2.dll executes many threads in parallel. ASan can handle multiple threads like that no problem!<\/li>\n<\/ol>\n<p>Most importantly: <strong>ASan never has false positives<\/strong>. This trace I found is definitely a bug, so I already know I found something to fix.<\/p>\n<p>Fortunately, the triggering input is a single file. I can easily repeat the command that repro\u2019s the bug manually. To be clear, at this point this was all I needed to do to hit the issue:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-27863\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2.png\" alt=\"Image ASan MSVC error - stack buffer underflow\" width=\"2373\" height=\"358\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2.png 2373w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2-300x45.png 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2-1024x154.png 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2-768x116.png 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2-1536x232.png 1536w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-MSVC-error-2-2048x309.png 2048w\" sizes=\"(max-width: 2373px) 100vw, 2373px\" \/><\/a><\/p>\n<p>I\u2019ve truncated the output, but the terminal contained the full ASan command-line diagnostics. I could use that information (starting with the stack-trace you can see above) to investigate the issue. However, I like examining these in the full IDE and debugging experience. With this command line, I can reproduce the ASan issue but attach it to the debugger:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-27864\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging.png\" alt=\"Image ASan debugging\" width=\"2365\" height=\"118\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging.png 2365w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging-300x15.png 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging-1024x51.png 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging-768x38.png 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging-1536x77.png 1536w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/ASan-debugging-2048x102.png 2048w\" sizes=\"(max-width: 2365px) 100vw, 2365px\" \/><\/a><\/p>\n<p>Starting the debugger-attached version of my binary, I see:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS.png\"><img decoding=\"async\" class=\"alignnone size-full wp-image-27866\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS.png\" alt=\"Image Debugging ASan in VS\" width=\"1874\" height=\"1178\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS.png 1874w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS-300x189.png 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS-1024x644.png 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS-768x483.png 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/04\/Debugging-ASan-in-VS-1536x966.png 1536w\" sizes=\"(max-width: 1874px) 100vw, 1874px\" \/><\/a><\/p>\n<p>The IDE is able to provide a wealth\u2014<em>interactively<\/em>\u2014of information about what\u2019s going on <em>the moment<\/em> a memory violation is detected. You can see the ASan issue is reported as an exception, bringing me right to the correct line number, along with my familiar debugger call-stack and everything else. The output window is still available for those accustomed to it.<\/p>\n<p>Any guesses of where the bug may be lurking?<\/p>\n<p>Hint: \u201c<code>sz<\/code>\u201d likely stands for \u201csize\u201d. Observe recall how ASan reports \u201cstack buffer <em>underflow<\/em>\u201d.<\/p>\n<p>&nbsp;<\/p>\n<h3>Fixing the Bug<\/h3>\n<p>Examining the value of <code>sz<\/code> made it clear enough: <code>MscIsFloatOrVectorConstant<\/code> returns the size of the constant <em>if it is found<\/em>, and 0 otherwise. In this buggy case, it returns 0, and we underflow the array field in the function-local struct <code>vval<\/code>. The fix is equally straightforward: following idioms in the rest of the file, we simply add a check for that before line 16828. This fix has been integrated and will be included in version 16.10.<\/p>\n<p>This particular bug is very unlikely to strike \u201cin the wild\u201d: the stack would need to have garbage values in just-the-right way (to pass the condition on line 16831). However, in theory this bug\u2014and more generally, bugs just like it\u2014could lead to an improper optimization in your code. That is among the worst sort of bug a compiler can have: silent-bad-codegen. I\u2019m very pleased to have squashed this one. I\u2019m also very glad to have been able to share just how easy ASan can make bugfixing with you.<\/p>\n<p>&nbsp;<\/p>\n<h3>Conclusion<\/h3>\n<p>We don\u2019t typically write blog posts about fixing a bug in the compiler, but of course the real story is just how easily and effectively ASan helps find and fix bugs:<\/p>\n<ul>\n<li>Our bespoke, command-line driven build system needed just a few lines of changes to integrate the build-with-ASan option.<\/li>\n<li>Once built, testing our binary was seamless: I ran my typical inner-dev-loop test suite.<\/li>\n<li>Once it found an issue, it was equally seamless to repeat the steps in our IDE\u2019s debugger, pointing me right to the source line to examine.\n<ul>\n<li>In more sophisticated circumstances, <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/sanitizers\/asan-offline-crash-dumps?view=msvc-160\">consider the ability to save a dump file<\/a>!<\/li>\n<\/ul>\n<\/li>\n<li>The exact source line, coupled with ASan\u2019s ability to characterize the issue (a stack <em>underflow<\/em>), made the investigation quick and easy. No long hours or brilliant insight needed: of course I still had to confirm and actually fix the issue, but compared to a traditional bugfix so much of the investigation was short-circuited.<\/li>\n<\/ul>\n<p>It is exactly the speed, effectiveness, and simplicity of ASan that I hope this story helps illustrate. Moreover, and most compellingly to me: ASan found a memory violation <em>that hadn\u2019t yet manifested as bad behavior<\/em> in our program. It <em>could<\/em> manifest, but here we were able to pin it down and squash it without an expensive, more indirect investigation\u2014and hopefully without it ever having affected our customers!<\/p>\n<p>&nbsp;<\/p>\n<h3>Try out AddressSanitizer for Windows<\/h3>\n<p>To get started with this experience, check out our <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/sanitizers\/asan\">AddressSanitizer documentation for MSVC and Visual Studio<\/a> as well as our announcement blog post: <a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/address-sanitizer-for-msvc-now-generally-available\/\">Address Sanitizer for MSVC Now Generally Available<\/a>.<\/p>\n<p>ASan for MSVC was done due to feedback from developers like you. If you have suggestions on the future of undefined-behavior-, memory-, thread-, or other-sanitizers, please share them as a <a href=\"https:\/\/developercommunity2.visualstudio.com\/search?entry=suggestion&amp;space=62&amp;ftype=idea\">suggestion on Developer Community<\/a>! If you suspect you\u2019ve hit an issue or bug, please also don\u2019t hesitate to <a href=\"https:\/\/developercommunity2.visualstudio.com\/search?entry=suggestion&amp;space=62&amp;ftype=problem&amp;q=asan\">open a ticket on Developer Community<\/a>!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Special thanks to Aaron Gorenstein for authoring this blog post. The AddressSanitizer (ASan) is generally available for MSVC since the recently-released Visual Studio 2019 version 16.9. We\u2019ve already shown how easy it can be to find bugs in even production-ready code like EASTL. Here I\u2019ll share an example of how it found a real bug [&hellip;]<\/p>\n","protected":false},"author":1063,"featured_media":27866,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-27854","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cplusplus"],"acf":[],"blog_post_summary":"<p>Special thanks to Aaron Gorenstein for authoring this blog post. The AddressSanitizer (ASan) is generally available for MSVC since the recently-released Visual Studio 2019 version 16.9. We\u2019ve already shown how easy it can be to find bugs in even production-ready code like EASTL. Here I\u2019ll share an example of how it found a real bug [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27854","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/1063"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=27854"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27854\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/27866"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=27854"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=27854"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=27854"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}