Electronic Arts and Visual Studio’s Linux Support

Erika Sweet

Electronic Arts (EA) is using Visual Studio’s cross-platform support to cross-compile on Windows and debug on Linux. The following post is written by Ben May, a Senior Software Engineer of Engineering Workflows at EA. Thanks Ben and EA for your partnership, and for helping us make Visual Studio the best IDE for C++ cross-platform development.

At EA our Frostbite Engine has a Linux component used for our dedicated servers that service many of our most popular games.  When we saw that Microsoft was adding support for Linux in a workload in Visual Studio, this caught my interest!  At EA our game developers are used to a Windows environment for development so we thought that forcing them to develop in a Linux environment directly would be a difficult ask, so we decided to use clang and cross-compile from Windows and target Linux.  Initially we had wired this up ourselves using Visual Studio Makefile Projects which called make to build our source, and then used a variety of tools to copy binaries over ssh to Linux machines, then wrote tooling to startup gdbserver on the remote Linux machine to be able to debug from PC.  After the release of the Visual Studio Linux Workload, we found that Microsoft had basically wrapped up all of the tools/processes up nicely into a Visual Studio Workload we could ask our Developers to install and be able to debug directly in Visual Studio!  So far the integration with WSL and remote debugging the workload provides has been a success and has drastically cleaned up our tools/processes surrounding Linux debugging/development.  Our developers have been really happy with the improved experience.

I will now explain in more detail what we actually do.

Internal build setup

Our internal build setup uses our own proprietary tool to take our own cross platform build format and generate many types of outputs (vcxproj/csproj/make etc.) When we decided to add Linux to our list of supported platforms, we decided that we would set up our primary workflow for our developers to be initiated from a Windows based PC with Visual Studio, since this is the environment that we use for almost all of our other platforms.  Another requirement was for our CI (Continuous Integration/Build Farm) to be able to validate that our code compiled on Linux without needing to setup Linux host based CI VMs or needing a remote Linux system to compile the code, since that would be much more expensive and complicated to manage and support.  These requirements basically led us to deciding to cross-compile our codebase on Windows directly using clang on PC.

For our cross-compiler we use something called a “Canadian cross” compiler setup.  See toolchain types for more details on the types of cross-compile you can do, and a Wikipedia link for why its called “Canadian cross”.  The primary reason for it being a “Canadian cross” is that we have built the LLVM and GCC toolchains on a Linux machine and moved their pieces to be used on a Windows machine combined with Windows clang.  Based on that our cross-compiler setup on Windows has the following in it:

1. We use LLVM
2. We combine the Windows version of LLVM with the Linux one on the Windows machine.  This is to get all of the libs/headers required for targeting Linux.
3. We also use the GCC toolchain with LLVM.  In order to build the gcc tools for Windows we use crosstool-NG on a Linux host to build it.
4. Then when building you need to pass -target x86_64-pc-linux-gnu and -sysroot=<path to gcc cross tools>
5. You may need to initially use -Wno-nonportable-include-path warning suppression since Windows is not case-sensitive, and fixing all of the include path errors might be a bit of a lengthy task (although I recommend doing it!)

After we have assembled our toolchain, we then use our proprietary generator to generate makefiles that build our code for us but referencing the above cross-compiler setup, and then a set of vcxproj files which are of type “Linux Makefile” and .sln file.  It is at this point where we move into Visual Studio for integration of our workflows into the IDE using the Visual Studio Linux Workload.

Visual Studio integration

Developers need to ensure they have the ‘Linux development with C++’ Workload installed:

After ensuring the correct components are installed, we use the built in features of the Linux Makefile projects for working.  To build code we simply select Build from within Visual Studio, this executes our cross-compiler and outputs binaries.  Built into the Visual Studio Linux Projects is the ability to deploy and debug on a Linux host. 

Debugging on WSL

We can configure our generator to use 2 different deployment/debugging setups:

1. WSL (Windows Subsystem For Linux)
2. Remote Linux Host

The most convenient setup is WSL assuming you do not have to render anything to screen.  In other words, if it’s only headless unit tests or console applications you need to develop this is the easiest and fastest way to iterate.

If the developer is using WSL, then the binaries do not actually need to be deployed since WSL can access the binaries directly from the current Windows machine, this saves time since they no longer need to be copied/deployed to a remote machine (some of our binaries can get quite large so can sometimes add several seconds to an incremental build + debug session)

Here is an example of me building EASTL, an open-source library of EAs, using Visual Studio Linux Makefile Projects and our cross-compiler:

You can see I’ve placed a breakpoint there, and I configured my environment to use WSL when running, so when I debug it will launch the test binary in WSL and connect the debugger using Visual Studio’s gdb debugger without needing to first copy the binary.  This achieved by setting the Remote Build Root, Project and Deploy directories with the path being the WSL path to the same folder on my Windows machine.

Here is a quick example of me debugging and hitting a breakpoint, then continuing to run and finish the unit test:

Debugging on a remote Linux system

For a Remote Linux machine setup where we do not use WSL, the only additional thing we need to worry about is the deployment of the built executable and its dependent dynamic libraries or content files.  The way we do this is we setup a source file to remote mapping, and have Visual Studio’s post build event do the copying of the file.  Inside the properties of the exe’s Project under “Build Events” -> “Post-Build Event” -> “Additional Files To Copy”, we specify the list of files needed to be copied to the remote machine after the build completes.  This is needed to happen so that when we click “Debug” the binaries are already there on the Remote Machine.

You can see that the syntax is a mapping of local path to remote path, which is quite handy for mapping files between the 2 file systems.

Asks for the future

One downside to this is that the deployment is done during the “Build” phase, what we would ideally like is to have 3 distinct phases when working with Linux:

1. Build
2. Deploy
3. Debug

This would be so that you could build the code without needing a connection to a remote machine, this is useful in CI environments, or in environments where someone just wants to locally build and fix compile issues for Linux and submit them and let automated testing validate their fixes.  Having Deploy and Debug distinct phases is also nice so that you could deploy from visual studio, but then potentially invoke/debug etc. directly from the Linux Machine.

It is also worth noting at this point that we are still using make “under the hood” to execute our builds for Linux, but the Visual Studio Linux Workload also supports a full MSBuild-based Linux Project.  We have not spent much time trying that out at this point, but it would be nice if we could use that, in an effort to be using MSBuild to build Linux just like we do for most of our other platforms.

We have been working closely with the Visual Studio team on the Linux Component, and have been following the Visual Studio 2019 Preview builds very closely to test and iterate on these workflows with them, our hope is that in future releases we will be able to:

1. Fully separate Build from Deployment and Debug for local cross-compilation scenarios.
2. Setup “incremental” build + deployment detection in the Linux Makefile Projects so that we don’t need to respawn make for all projects in our Solutions (some of our large solutions have > 500 projects).  This is mainly for a faster incremental iteration times.
3. We have asked for direct WSL debugging be added to the Linux Makefile Projects, currently in our setup since the Linux Makefile Projects don’t support WSL directly we still need to debug over an ssh connection to wsl which means we have to have WSL running with sshd on it. This support is already integrated with MSBuild-based Linux Applications and CMake projects, but not yet for Makefile projects.
4. Try the MSBuild-based Linux Project files and work with Microsoft to get those to potentially operate with a local toolchain (our cross-compiler) but still yield the same features for Deployment and Debug.  This would also help us solve the Makefile incremental problem mentioned above.

All in all this workflow is very slick for us!  It allows our developers to use an IDE and Operating System they are comfortable working in, but are still able to build and debug Linux applications!

-Ben May, Senior Software Engineer, Engineering Workflows at EA

Thanks again for your partnership, Ben! Our team looks forward to continuing to improve the product based on feedback we receive from the community. If you’re interested in building the same project on both Windows and Linux, check out our native support for CMake. You can check out a similar story written by the MySQL Server Team at Oracle.

Erika Sweet Senior Product Manager, C++ Team

Follow