{"id":42658,"date":"2022-10-07T08:15:00","date_gmt":"2022-10-07T15:15:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=42658"},"modified":"2024-12-13T14:20:40","modified_gmt":"2024-12-13T22:20:40","slug":"bing-ads-campaign-platform-journey-to-dotnet-6","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/bing-ads-campaign-platform-journey-to-dotnet-6\/","title":{"rendered":"Bing Ads Campaign Platform \u2013 Journey to .NET 6"},"content":{"rendered":"

The campaign platform component of Microsoft\u2019s search advertising\nplatform<\/a>\nis central to delivering a great experience to our advertising platform\nusers. It supports millions of advertisers, providing the engine under\nthe hood allowing them to create ad campaigns that reach customers with\nmaximum impact.<\/p>\n

In a given second this platform will process thousands of web requests,\nwith a typical latency of under 100 milliseconds for a given request.<\/p>\n

\"Requests<\/p>\n

Behind the scenes, dozens of distributed services work in concert to\nsupport all of the rich functionality provided by the platform.<\/p>\n

Today, all of this is built on top of NET\n6<\/a>, running on\nLinux containers in Azure Kubernetes\nService<\/a> (AKS) clusters.<\/p>\n

Getting here, however, was quite the journey!<\/p>\n

In the rest of this blog post we will look at the multi-year journey we\nundertook to move this codebase to .NET 6, along with the challenges\nwe faced and how we ultimately solved them.<\/p>\n

Note: the re-branding from “.NET Core” to simply “.NET” sometimes leads to some confusion over whether we are discussing .NET Framework or what was previously known as .NET Core. In this post, “.NET” generally refers to the latter and if we are referring to .NET Framework that will be spelled out explicitly.<\/p>\n

Overview of the codebase<\/h2>\n

We have a fairly large amount of code. To give a bit of context, the git\nrepo that houses the middle tier code for our campaign platform contains\nover 600 C# projects.<\/p>\n

Breaking down the code:<\/p>\n

\"Table<\/p>\n

So, more than 7 million lines of C# code and more than 8 million lines\ntotal. Counting lines of code does not tell you everything about a\ncodebase, but it does give a rough idea of the relative size we are\ndealing with.<\/p>\n

Over those 600+ csproj instances, we reference more than 500 distinct\nNuGet<\/a> packages. That many dependencies had an\nimpact on the process of migrating to .NET, as we will see.<\/p>\n

Our starting point<\/h2>\n

Our services have evolved a lot over the years and the underlying\ntechnology used to host them has changed dramatically. Originally hosted\nentirely on-premise with our own physical hardware sitting in racks in\ndata-centers that we had to maintain, our team followed much of the rest\nof the industry by eventually embracing the cloud.<\/p>\n

The migration to Azure is a story in itself, but the legacy of the\noriginal hosting environment has an impact on where we were when we\nstarted our .NET migration: traditionally we had run our code on Windows\nservers, hosting our web services in IIS<\/a>.<\/p>\n

In addition, a significant portion of our web services were built as\nSOAP<\/a> services using\nWCF<\/a>.\nThis complicated things, as we shall see.<\/p>\n

While we had previously migrated to Azure, we had simply done a \u201clift\nand shift\u201d from running on our own hardware to running on Windows VMs in\nthe cloud.<\/p>\n

All of our code was at the time targeting the original Windows-only .NET\nFramework and when we started contemplating migrating to .NET we\nwere running on .NET Framework version 4.6.<\/p>\n

Why the new .NET?<\/h2>\n

Why did we choose to expend the engineering effort to migrate away from\n.NET Framework and onto the open source .NET? After all, this was\nnot a small effort \u2013 it took more than two years to accomplish the bulk\nof the work.<\/p>\n

There were multiple reasons that our team specifically called out after\nan initial investigation into .NET:<\/p>\n

Cross-Platform<\/h3>\n

Moving to a cross-platform solution had an obvious benefit: freeing us\nfrom being inextricably tied to the Windows operating system would give\nus the potential to explore if we would be better served running on\nalternatives such as Linux.<\/p>\n

Unlike some other teams at Microsoft, we did not always intend to move\nto something like AKS running on Linux containers. We were, however,\nvery interested in exploring that as a possibility.<\/p>\n

Future of .NET Development<\/h3>\n

It was clear when we started our migration journey that .NET was\nthe future. All of the really exciting innovations in the runtime to\nsupport high performance computing were happening there, not on .NET\nFramework, and high performance is one of our main requirements.<\/p>\n

We also considered it very important that we give the developers on our\nteam the best possible tools for the job. At the time, some features of\nC# 8 were already only available on .NET and that situation would\nbecome more common for future versions of the language.<\/p>\n

It was also known that .NET Framework 4.8 was planned to be the last\nversion. Sticking with it was a technological dead-end.<\/p>\n

Less friction for innovation<\/h3>\n

.NET being developed as an independent open source project means\nthat it can iterate at a much faster cadence than the traditional .NET\nFramework, improvements to which are burdened by the costs associated\nwith being part of Windows. That means we get improvements and new\nfeatures much more quickly.<\/p>\n

Open source<\/h3>\n

Microsoft\u2019s embrace of open source has been inspiring and having your\nunderlying tech stack being developed out in the open has many benefits.\nGitHub issues provide a simple and convenient mechanism to report\nproblems, request features or get feedback from the development team.<\/p>\n

The ability to contribute back fixes and improvements is also highly\nappealing.<\/p>\n

Much better tooling<\/h3>\n

The .NET team clearly took a hard look at some of the major pain points\nfor .NET developers. Not only did they throw out a variety of \u201cseemed\nlike a good idea at the time\u201d features that ultimately caused more\ntrouble than they solved (multiple app domains and .NET remoting come to\nmind), they dramatically improved the tooling used for a developers\n\u2018inner loop\u2019.<\/p>\n

The dotnet CLI<\/a>\ntooling that allows you to easily accomplish many tasks without even\nneeding an IDE has improved productivity a lot, especially for\ndevelopers who enjoy working from the terminal.<\/p>\n

The much-simplified project format where sensible defaults are used\neverywhere and source code files are implicitly included was a huge\nquality-of-life improvement. Refactoring source code files among\nprojects went from a huge chore to being nearly painless.<\/p>\n

Additionally, .NET mercifully brought a solution to the nightmare\nof diamond\ndependency<\/a>\nissues between projects. These often caused the need to add binding\nredirects<\/a>\nthat were hard to get right across all of our many projects and were\noften only discovered at runtime. .NET solved the problem by simply\ngetting rid of binding redirects entirely.<\/p>\n

Our conversion process<\/h2>\n

Our eventual migration process can be summarized as:<\/p>\n

For class libraries, we followed the following overall path:<\/p>\n

.NET Framework 4.6 -> .NET Framework 4.7 -> .NET Standard 2.0<\/p>\n

For our actual services and applications, the process looked like this:<\/p>\n

.NET Framework 4.6 -> .NET Framework 4.7 -> .NET Core 3.1 -> .NET 5\n-> .NET 6<\/p>\n

.NET Standard<\/h3>\n

There was absolutely no way we were going to be able to convert all of\nour code to .NET in one shot. Our codebase churns constantly so the\nidea of forking the repo and doing all of the conversion work there was\nconsidered and quickly discarded.<\/p>\n

We needed to be able to do the work iteratively, over time, and that\u2019s\nwhere .NET\nStandard<\/a>\ncame in.<\/p>\n

\"Diagram<\/p>\n

As the diagram shows, a library targeting the subset of APIs supported\nby .NET Standard can be consumed by both .NET Framework projects and\n.NET projects. The vast majority of our code consists of class\nlibraries \u2013 the hundreds of DLLs (assemblies) produced by most of our C#\nprojects. If those could easily be converted to .NET Standard we could\ncontinue to consume them from our existing .NET Framework services, then\niteratively convert those to .NET for testing and eventual\ndeployment.<\/p>\n

(Narrator: they were not easily converted to .NET Standard.)<\/p>\n

Converting all of our projects first from .NET Framework 4.6 to 4.7 was\nactually forced on us by the fact that .NET Framework 4.6 did not fully\nsupport .NET Standard. We ended up trying for weeks to get .NET Standard\nlibraries to successfully be consumed by our existing 4.6 projects, but\nran into issue after issue. The truth is that .NET Standard, while\nadvertised to work with 4.6, really did not work correctly until 4.7.<\/p>\n

Ultimately we were able to convert the majority of our code to .NET\nStandard 2.0 but there were\u2026challenges.<\/p>\n

Challenges<\/h2>\n

We hit many, many issues attempting to convert hundreds of projects to\n.NET Standard. Remember those 500 NuGet package references? What happens\nwhen you convert a project to .NET Standard that depends on a NuGet\npackage that in turn targets .NET 4.6? That won\u2019t build \u2013 a .NET\nStandard library can only depend on other .NET Standard libraries. So\nyou have to find a new version of the NuGet package that supports .NET\nStandard 2.0.<\/p>\n

Many times, those packages were much newer than the old ones our code\nhad been relying on for years.<\/p>\n

They would have breaking changes. To use one example, we relied heavily\non an old Dependency Injection framework called\nUnity<\/a> (not to be confused\nwith the Unity game engine.) The available version that supported .NET\nStandard had had its API completely rewritten and we had to update tens\nof thousands of lines of code to be compatible with the changes.<\/p>\n

Often, no such package even existed.<\/p>\n

Binding redirect hell<\/h3>\n

One particularly insidious problem that we encountered constantly and\nwhich lead to countless hours of work attempting to wrestle into\nsubmission was the problem of binding redirects. For a variety of\nreasons, things like the AutoGenerateBindingRedirects property did not\nalways solve this problem in our codebase and we had to continually add\nnew binding redirects to the configuration files for our services.<\/p>\n

Ultimately, we had to find ways to help solve the constant diamond\ndependency conflicts so we could make progress.<\/p>\n

WCF<\/h3>\n

Another problem that was particularly painful for the campaign platform\nconversion was our heavy reliance on WCF. We have over 45 services built\non top of WCF, constituting a significant portion of the code that would\nbenefit from the higher performance of using .NET.<\/p>\n

For better or worse, in the .NET community SOAP-based services and,\nsomewhat by extension, WCF, had become persona non grata. The prevailing\nsentiment is that REST-based services were the future and at first the\npath forward for WCF in .NET was not clear \u2013 Microsoft was\nconsidering leaving it as a deprecated, .NET Framework-only technology.<\/p>\n

This was not going to work for our team, as we had countless existing\ncustomers who had built heavily on top of these WCF services and used\nour SDKs to call them directly. Telling our paying customers they were\ngoing to have to rewrite all of their client code using something new\nlike gRPC<\/a> was not an acceptable answer.<\/p>\n

Solutions<\/h2>\n

Spoiler alert: we were, in the end, able to solve all of these problems\nand successfully migrate to .NET 6. It was not always easy, however.<\/p>\n

Incompatible NuGet dependencies<\/h3>\n

Handling the problem of missing or incompatible NuGet packages took\ntime, but ultimately we were able to solve all such issues.<\/p>\n

In a number of cases we had to get creative and essentially \u201crepackage\u201d\nan existing NuGet package that was no longer supported, changing it to\nclaim it supported .NET Standard even when the actual assembly was some\nold .NET 4.6 binary. We would publish these modified packages to our own\ninternal package feed to unblock our migration.<\/p>\n

In a few cases we ended up decompiling old, no-longer-maintained\npackages for which no source code existed, and updated them to be .NET\nStandard compatible (for instance, removing use of features that only\nexist on Windows.) We would patch up the decompiled code and then,\nagain, make a new version of the package.<\/p>\n

Binding redirect issues<\/h3>\n

Moving from the\npackages.config<\/a>\nmechanism used in .NET Framework to the new\nPackageReference<\/a>\nmechanism used in the newer SDK tooling favored by .NET was a major\nfactor in finally getting the diamond dependency \/ binding redirect\nproblem under control.<\/p>\n

We decided to convert everything<\/em> in our tree to the new SDK style\nformat that supported PackageReference. When you have 600+ C# projects,\nthis is a non-trivial undertaking.<\/p>\n

Luckily, there are tools to help with this. I have a personal blog\npost<\/a>\nshowing how we did it.<\/p>\n

Subsequently, the .NET team released the\ntry-convert<\/a> tool to accomplish\nmuch the same thing.<\/p>\n

Once the large effort of converting all projects to the new SDK format\nwas accomplished, another technique that helped untangle the gnarled web\nof package dependencies was to move to centralized package\nversioning<\/a>.<\/p>\n

This forced all projects to use the same version of any referenced NuGet\npackages, with the version being specified in a single central location\ninstead of at the granularity of each individual project. This greatly\nsimplified moving to newer versions of NuGet packages that would allow\nmoving to .NET. The centralized package version can be overridden\nif necessary, so there is no significant downside to setting this up.<\/p>\n

WCF<\/h3>\n

Ultimately, Microsoft decided to produce a limited subset of WCF that\ntargeted .NET. This subset, which primarily focuses on SOAP-based\nweb services, was then donated to the community as the open source\nCoreWCF<\/a>\nproject.<\/p>\n

CoreWCF uses entirely new namespaces for the many existing types that\ncame from the System.ServiceModel namespace of traditional WCF, so\nconverting an existing service is not exactly trivial. Also, we had\ncommon code used so extensively in our codebase that we needed to\nsupport running that code in both CoreWCF services and .NET Framework\nservices while we were in the process of converting.<\/p>\n

We ultimately used\nmulti-targeting<\/a>\nto accomplish this.<\/p>\n

(Incidentally, a tricky problem resulting from multi-targeting caused by\na top-level exception handler led to perhaps the only time I have\nresorted to using the dynamic keyword in C# without feeling guilty about\nit.)<\/p>\n

We worked extensively with the extremely helpful Matt\nConnew<\/a> (a primary contributor to CoreWCF)\nto get our services ported, and while it was not an entirely smooth\ntransition, ultimately CoreWCF worked great as an alternative to\nrewriting our services to use something other than SOAP.<\/p>\n

If you need to host a SOAP service and you want the high performance\nthat comes with running on .NET 6, CoreWCF is a great answer.<\/p>\n

Results<\/h2>\n

Was all of the effort put into migrating to .NET 6 ultimately worth it?\nAbsolutely!<\/p>\n

The benefits we had outlined at the start of our journey all proved to\nbe true and then some.<\/p>\n

This graph shows the improvement in latency for one of our services that\nresulted simply from changing it to target .NET and recompiling.<\/p>\n

\"Graph<\/p>\n

You can tell where in the time axis we flipped to the new service\nrunning as .NET.<\/p>\n

That was without even attempting to optimize anything using the great\nnew features available in .NET. That was just recompiling and getting\nthe advantage of the many runtime improvements made by the .NET team.<\/p>\n

Here is another example where we looked at the memory usage both before\nand after moving to Core WCF for another service:<\/p>\n

\"Table<\/p>\n

We achieved a 40 to 50 percent reduction in memory usage simply by\nmaking the change to .NET.<\/p>\n

Modern cloud service infrastructure<\/h3>\n

In addition to these clear performance wins, getting onto .NET\nopened up the opportunity for us to migrate off of IIS and Windows and\nonto\nKestrel<\/a>\nservers running inside of containers on Linux, hosted in AKS.<\/p>\n

All of the modern tooling and resources available to manage and\nconfigure cloud services (such as Kubernetes) are now available to us.\nOur engineers get to use well supported, industry standard tooling\ninstead of the proprietary, internal systems that had developed over\ntime inside Microsoft to manage our custom, non-standard hosting and\ndeployment infrastructure.<\/p>\n

This ability to now use industry best practices technology to continue\nto evolve our platform is a huge win for everyone.<\/p>\n

Performance anecdote \u2013 hackathon<\/h2>\n

While not directly related to the Campaign Platform service migration,\ntwo devs from the Campaign Platform team (myself and a colleague)\nparticipated in Microsoft\u2019s annual hackathon and got a chance to\nshowcase how with good design and using high-performance techniques, you\ncan achieve impressive performance metrics.<\/p>\n

\"Hash<\/p>\n

The project involved optimizing the calculation of how similar two\n64-byte buffers were to each other, based on how many two-bit pairs were\nidentical. (This has applications in anti-malware research.)<\/p>\n

As you can see, the distributed cloud service we built on top of .NET 6\ncan check almost a billion 64-byte buffers totaling close to 60 GiB of\ndata in 40 to 60 milliseconds.<\/p>\n

Not too shabby!<\/p>\n

Summary<\/h2>\n

Migrating to .NET 6 was a large and at times painful engineering effort.\nIt was worth it in the end and going forward our team is looking forward\nto the continued improvements to .NET as it evolves.<\/p>\n

For others planning to migrate a large existing .NET Framework codebase\nto .NET 6 and beyond, lessons learned from our experience can be\napplied:<\/p>\n

    \n
  1. \n

    Get all existing code onto .NET 4.7 or 4.8 first<\/p>\n<\/li>\n

  2. \n

    Migrate all projects to the new SDK format so they are using\nPackageReference before doing anything else.<\/p>\n<\/li>\n

  3. \n

    Use .NET Standard as a bridge to allow sharing library code between\nboth .NET Framework and .NET projects while migration is in\nprogress.<\/p>\n<\/li>\n

  4. \n

    Use centralized package references to greatly ease the transition to\nnewer NuGet packages.<\/p>\n<\/li>\n<\/ol>\n

    Moving forward post-migration, we are looking forward to really drilling\ninto some of the new features in .NET that will allow us to take our\ncode to the next level.<\/p>\n

    Here is just one instance of such a feature: finding places we can use\nSpan\\<T><\/a>\nto reduce heap allocations and improve performance. For example, our\ncode has places where we check if two byte buffers are identical.\nInstead of looping over each byte and testing for equality we can make\nuse of the highly optimized\nSequenceEqual<\/a>\nmethod:<\/p>\n

    return bufferA.AsSpan().SequenceEqual(bufferB);<\/code><\/pre>\n
    As this\nbenchmark<\/a>\nshows, this is close to 25 times faster than the na\u00efve approach!<\/p>\n
    \"Benchmark<\/p>\n
    Rewriting some of our code to specifically take advantage of new\nlanguage and runtime features like this is going to continue to be a fun\nand highly rewarding exercise as we move forward.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Bing Ads Campaign Platform – our journey migrating a large .NET Framework codebase to .NET 6.<\/p>\n","protected":false},"author":102486,"featured_media":42796,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,7635,7257],"tags":[7239,3267],"class_list":["post-42658","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-developer-stories","category-wcf","tag-dotnet-6","tag-migration"],"acf":[],"blog_post_summary":"
    Bing Ads Campaign Platform – our journey migrating a large .NET Framework codebase to .NET 6.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/42658","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/102486"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=42658"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/42658\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/42796"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=42658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=42658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=42658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}