{"id":27431,"date":"2021-01-26T16:49:13","date_gmt":"2021-01-26T16:49:13","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cppblog\/?p=27431"},"modified":"2023-02-02T19:12:10","modified_gmt":"2023-02-02T19:12:10","slug":"blizzard-diablo-iv-debugs-linux-core-dumps-from-visual-studio","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/blizzard-diablo-iv-debugs-linux-core-dumps-from-visual-studio\/","title":{"rendered":"Blizzard Diablo IV debugs Linux core dumps from Visual Studio"},"content":{"rendered":"<p style=\"text-align: center;\"><iframe src=\"\/\/www.youtube.com\/embed\/IuTcKwVZziQ\" width=\"560\" height=\"314\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>Blizzard is using Visual Studio 2019 to <a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/debug-linux-core-dumps-in-visual-studio\/\">debug Linux core dumps on WSL<\/a>. <strong>The following blog post is written by Bill Randolph, a Senior Software Engineer at Blizzard working on the development of Diablo IV.<\/strong> Thanks for your partnership, Bill!<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith.jpg\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-27432\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith.jpg\" alt=\"A character from Diablo IV with the title &quot;Diablo IV&quot;.\" width=\"1920\" height=\"1080\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith.jpg 1920w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith-300x169.jpg 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith-1024x576.jpg 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith-768x432.jpg 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/16_9_background_lilith-1536x864.jpg 1536w\" sizes=\"(max-width: 1920px) 100vw, 1920px\" \/><\/a><\/p>\n<h4>Introduction<\/h4>\n<p>On Diablo IV we develop all our code on Windows and compile for multiple platforms.\u00a0 This includes our servers, which run on Linux.\u00a0 (The code includes conditional compilation and custom platform-specific code where necessary).\u00a0 There are multiple reasons for this workflow.\u00a0 For one, our team\u2019s core competency is on Windows.\u00a0 Even our server programmers are most familiar with Windows development, and we appreciate the ability for all the programmers on our team to use a common toolset and knowledge base.<\/p>\n<p>The other, and most important reason that we develop on Windows is the functionality and robust toolset provided by Visual Studio.\u00a0 There is nothing quite comparable in the Linux world, even if we were to develop natively in Linux.<\/p>\n<p>However, this presents us with some challenges when a deployed server crashes and we want to debug the resulting core dump.\u00a0 There is the option to remote login to the VM (or more specifically the container) that crashed and run gdb to diagnose the crash there.\u00a0 But there are numerous disadvantages to this.\u00a0 For one, we don\u2019t deploy source with our binaries, so source is not available in a gdb session on a VM or container.<\/p>\n<p>Another hurdle is gdb itself: unless you use gdb on a very regular basis, you don\u2019t retain a level of proficiency with it that makes it convenient for our use.\u00a0 Putting it simply, our developers would much rather use familiar tools to debug.\u00a0 Since only 2 or 3 of our developers have much proficiency with gdb, they become the de-facto resource for diagnosing production crashes, and that\u2019s not optimal.<\/p>\n<p>We have always wanted a more intuitive approach for debugging our Linux cores.\u00a0 That\u2019s why we are so excited to be able to utilize the new Visual Studio feature that lets us do just that in the familiar environment of Visual Studio!\u00a0 It really is not an exaggeration to say that this is a dream come true.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls.jpg\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-27433\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls.jpg\" alt=\"An image of combat gameplay in Diablo IV.\" width=\"1773\" height=\"1080\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls.jpg 1773w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls-300x183.jpg 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls-1024x624.jpg 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls-768x468.jpg 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/Combat-Crypts-Sorceress-Ghouls-1536x936.jpg 1536w\" sizes=\"(max-width: 1773px) 100vw, 1773px\" \/><\/a><\/p>\n<h4>Our debugging workflow<\/h4>\n<p>The Visual Studio Linux core debug workflow is enabled only if you install WSL or <a href=\"https:\/\/docs.microsoft.com\/en-us\/cpp\/linux\/connect-to-your-remote-linux-computer?view=msvc-160&amp;viewFallbackFrom=vs-2019\">add a Linux connection to the Connection Manager<\/a>. All our server developers install WSL, using the distribution we deploy on.\u00a0 We run a script I wrote that also installs all the development tools and support libraries needed to build our server within WSL.<\/p>\n<p>(As a brief side topic, I want to emphasize that we have found WSL to be the best available Linux environment for developers to test their changes in a Linux build.\u00a0 It\u2019s incredibly convenient to hop into WSL, cd into the shared code directory, and build right from there.\u00a0 This is a much better solution than running a VM, or even a container. \u00a0If you are building with CMake, then you can also leverage Visual Studio\u2019s <a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/c-with-visual-studio-2019-and-windows-subsystem-for-linux-wsl\/\">native support for WSL<\/a>.)<\/p>\n<p>Let me provide a little background about our build.\u00a0 We develop our code on Windows and have a Windows version of our servers that can run under Windows.\u00a0 This is useful for normal feature development.\u00a0 However, we deploy our servers on Linux, which requires a build generated on Linux itself.\u00a0 The Linux build is generated on a build farm that uses a build system on a Linux box to build the server, and its container that gets deployed.\u00a0 The Linux executable is only deployed in a container and the developers normally don\u2019t have access to it.<\/p>\n<p>When a server crashes in our infrastructure an automated process notifies us and the core file is archived to a network share.\u00a0 To debug a core in either Linux, or using Visual Studio, you must have the executable that was running; it also helps to debug with the exact shared libraries used on the deployed container.\u00a0 We use another script to obtain these files.\u00a0 First, we copy the core to our local machine then run the script and point it to the core. The script downloads the Docker container that was built with that version, extracts the server binary from it, along with certain shared runtime libraries for gdb\u2019s use.\u00a0 (This avoids gdb compatibility problems you may encounter if your WSL version does not exactly match the deployed Linux version.)\u00a0 The script writes to ~\/.gdbinit to set up the shared libraries as system libraries for the debug session.<\/p>\n<p>Then we switch over to Visual Studio, where the fun begins. We load the solution to build our Windows version of our servers. Then we open the new debug dialog under <strong>Debug -&gt; Other Debug Targets -&gt; Debug Linux Core Dump with Native Only<\/strong>. We enable the checkbox that says \u201cDebug on WSL\u201d and fill in the (WSL-specific!) path to both the core file and the server binary. After that, we hit Debug &amp; watch the show!<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/coreDumpUI.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-27434\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/coreDumpUI.png\" alt=\"A dialog in Visual Studio with the title &quot;Debug Linux core dump (native only). The checkbox &quot;Debug on WSL&quot; is selected.\" width=\"819\" height=\"382\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/coreDumpUI.png 819w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/coreDumpUI-300x140.png 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/coreDumpUI-768x358.png 768w\" sizes=\"(max-width: 819px) 100vw, 819px\" \/><\/a><\/p>\n<p>Visual Studio invokes gdb in our WSL behind the scenes. After some disk activity, up pops a call stack for the crash with the instruction pointer on the relevant line of code. It\u2019s a brave new world!<\/p>\n<p>So next comes the task of identifying the crash. We have a crash handler that intercepts the crash to perform some housekeeping, so the actual crash will be down the call stack in a single-threaded server. However, some of our servers are multi-threaded, and the crash could have originated from any of those threads. Our crash handler logs the source of the crash\u2019s file and line number, so examining those variables gives us our first lead; we will look for the call stack that was executing that code.<\/p>\n<p>In the old days of a few weeks ago, we would use gdb to get a backtrace of all threads and peruse the resulting list to see which thread had the most-likely call stack that would have crashed. For example, if a thread is just sleeping, it is most likely not the crashed thread. We would look for a stack that had some more content than a few frames capping with a \u201csleep\u201d and examine the code to see if a problem is evident, or go into gdb itself to examine the process state.<\/p>\n<p>However, Visual Studio gives us considerably more powerful options than that. For a multi-threaded core you can open the Threads window in your debug session and poke around in each thread to see what the stack looks like. This is pretty similar to the gdb approach, and if there are 50 threads it can be very tedious. Fortunately, there is a feature that makes this much easier: <a href=\"https:\/\/docs.microsoft.com\/en-us\/visualstudio\/debugger\/using-the-parallel-stacks-window?view=vs-2019\"><strong>Parallel Stacks<\/strong><\/a>.<\/p>\n<p>I confess most of us did not know about Parallel Stacks until Erika Sweet and her team told us about it. Invoking Debug -&gt; Windows -&gt; Parallel Stacks (only available during your debug session) opens a new window that shows the call stack of every thread in your process. It\u2019s a fascinating 30,000-foot view of your entire process space. You can double-click any stack frame in any thread, and Visual Studio will jump to that frame in both source and the call stack window. This is a huge time-saver for us.<\/p>\n<p>Once we can see the code near the crash, we can inspect variables using mouse-hover, QuickWatch or any of the other plethora of tools in Visual Studio. It\u2019s true that in a Release build, many variables are optimized out, but at the same time, many are not! We can hone in on a problem much faster using Visual Studio\u2019s interface than we ever could have using just gdb.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo.jpg\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-27435\" src=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo.jpg\" alt=\"Heroes from Diablo IV with the &quot;Diablo IV&quot; logo.\" width=\"1926\" height=\"1080\" srcset=\"https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo.jpg 1926w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo-300x168.jpg 300w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo-1024x574.jpg 1024w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo-768x431.jpg 768w, https:\/\/devblogs.microsoft.com\/cppblog\/wp-content\/uploads\/sites\/9\/2021\/01\/blizzcon_heroes_w_logo-1536x861.jpg 1536w\" sizes=\"(max-width: 1926px) 100vw, 1926px\" \/><\/a><\/p>\n<h4>Summary<\/h4>\n<p>Our team is very excited about the ability to debug Linux cores from our production environment in Visual Studio! It is a game changer for us, as it allows many more developers to actively diagnose problems \u201cin the wild\u201d, and it makes the powerful toolset of Visual Studio debugging available to all of us. Once our initial setup is complete, it only takes a minute or so to be in a debugging session in Visual Studio. This feature will make finding problems in our code much faster and more efficient! Thanks to Erika and her team for working with us on this!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Blizzard is using Visual Studio 2019 to debug Linux core dumps on WSL. The following blog post is written by Bill Randolph, a Senior Software Engineer at Blizzard working on the development of Diablo IV. Thanks for your partnership, Bill! Introduction On Diablo IV we develop all our code on Windows and compile for multiple [&hellip;]<\/p>\n","protected":false},"author":2953,"featured_media":27435,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3927,279],"tags":[],"class_list":["post-27431","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-game-development","category-linux"],"acf":[],"blog_post_summary":"<p>Blizzard is using Visual Studio 2019 to debug Linux core dumps on WSL. The following blog post is written by Bill Randolph, a Senior Software Engineer at Blizzard working on the development of Diablo IV. Thanks for your partnership, Bill! Introduction On Diablo IV we develop all our code on Windows and compile for multiple [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27431","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/2953"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=27431"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27431\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/27435"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=27431"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=27431"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=27431"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}