{"id":3748,"date":"2023-03-09T13:33:34","date_gmt":"2023-03-09T21:33:34","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/typescript\/?p=3748"},"modified":"2023-03-15T15:02:54","modified_gmt":"2023-03-15T23:02:54","slug":"typescripts-migration-to-modules","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/typescript\/typescripts-migration-to-modules\/","title":{"rendered":"TypeScript’s Migration to Modules"},"content":{"rendered":"

One of the most impactful things we’ve worked on in TypeScript 5.0 isn’t a feature, a bug fix, or a data structure optimization.\nInstead, it’s an infrastructure change.<\/p>\n

In TypeScript 5.0, we restructured our entire codebase to use ECMAScript modules, and switched to a newer emit target.<\/p>\n

What to Know<\/h2>\n

Now, before we dive in, we want to set expectations.\nIt’s good to know what this does and doesn’t mean for TypeScript 5.0.<\/p>\n

As a general user of TypeScript, you’ll need to be running Node.js 12 at a minimum.\nnpm install<\/code>s should go a little faster and take up less space, since the typescript<\/code> package size should be reduced by about 46%.\nRunning TypeScript will get a nice bit faster – typically cutting down build times of anywhere between 10%-25%.<\/p>\n

As an API consumer of TypeScript, you’ll likely be unaffected.\nTypeScript won’t be shipping its API as ES modules yet, and will still provide a CommonJS-authored API.\nThat means existing build scripts will still work.\nIf you rely on TypeScript’s typescriptServices.js<\/code> and typescriptServices.d.ts<\/code> files, you’ll be able to rely on typescript.js<\/code>\/typescript.d.ts<\/code> instead.\nIf you’re importing protocol.d.ts<\/code>, you can switch to tsserverlibrary.d.ts<\/code> and leverage ts.server.protocol<\/code>.<\/p>\n

Finally, as a contributor of TypeScript, your life will likely become a lot easier.\nBuild times will be a lot faster, incremental check times should be faster, and you’ll have a more familiar authoring format if you already write TypeScript code outside of our compiler.<\/p>\n

Some Background<\/h2>\n

Now that might sound surprising – modules?\nLike, files with import<\/code>s and export<\/code>s?\nIsn’t almost all modern JavaScript and TypeScript using modules?<\/p>\n

Exactly!\nBut the current TypeScript codebase predates ECMAScript’s modules – our last rewrite started in 2014, and modules were standardized in 2015.\nWe didn’t know how (in)compatible they’d be with other module systems like CommonJS, and to be frank, there wasn’t a huge benefit for us at the time for authoring in modules.<\/p>\n

Instead, TypeScript leveraged namespace<\/code>s – formerly called internal modules<\/em>.<\/p>\n

Namespaces had a few useful features.\nFor example, their scopes could merge across files, meaning it was easy to break up a project across files and expose it cleanly as a single variable.<\/p>\n

\/\/ parser.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n\r\n\/\/ program.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n\r\n\/\/ user.ts<\/span>\r\n\r\n\/\/ Can easily access both functions from 'ts'.<\/span>\r\nconst<\/span> <\/span>sourceFile<\/span> = <\/span>ts<\/span>.<\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\nconst<\/span> <\/span>program<\/span> = <\/span>ts<\/span>.<\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n<\/code><\/pre>\n
It was also easy for us to reference exports across files at a time when auto-import didn’t exist.\nCode in the same namespace could access each other’s exports without needing to write import<\/code> statements.<\/p>\n
\/\/ parser.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n\r\n\/\/ program.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/\/ We can reference 'createSourceFile' without writing<\/span>\r\n        <\/span>\/\/ 'ts.createSourceFile' or writing any sort of 'import'.<\/span>\r\n        <\/span>let<\/span> <\/span>file<\/span> = <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
In retrospect, these features from namespaces made it difficult for other tools to support TypeScript;\nhowever, they were very useful for our codebase.<\/p>\n
Fast-forward several years, and we were starting to feel more of the downsides of namespaces.<\/p>\n
Issues with Namespaces<\/h2>\n
TypeScript is written in TypeScript.\nThis occasionally surprises people, but it’s a common practice for compilers to be written in the language they compile<\/a>.\nDoing this really helps us understand the experience we’re shipping to other JavaScript and TypeScript developers.\nThe jargon-y way to say this is: we bootstrap<\/a> the TypeScript compiler so that we can dog-food<\/a> it.<\/p>\n
Most modern JavaScript and TypeScript code is authored using modules.\nBy using namespaces, we weren’t using TypeScript the way most of our users are.\nSo many<\/em> of our features are focused around using modules, but we weren’t using them ourselves.\nSo we had two issues here: we weren’t just missing out on these features – we were missing a ton of the experience in using those features.<\/p>\n
For example, TypeScript supports an incremental<\/code> mode for builds.\nIt’s a great way to speed up consecutive builds, but it’s effectively useless in a codebase structured with namespaces.\nThe compiler can only effectively do incremental builds across modules<\/em>, but our namespaces just sat within the global scope (which is usually where namespaces will reside).\nSo we were hurting our ability to iterate on TypeScript itself, along with properly testing out our incremental<\/code> mode on our own codebase.<\/p>\n
This goes deeper than compiler features – experiences like error messages and editor scenarios are built around modules too.\nAuto-import completions and the "Organize Imports" command are two widely used editor features that TypeScript powers, and we weren’t relying on them at all.<\/p>\n
Runtime Performance Issues with Namespaces<\/h2>\n
Some of the issues with namespaces are more subtle.\nUp until now, most of the issues with namespaces might have sounded like pure infrastructure issues – but namespaces also have a runtime performance impact.<\/p>\n
First, let’s take a look at our earlier example:<\/p>\n
\/\/ parser.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n\r\n\/\/ program.ts<\/span>\r\nnamespace<\/span> <\/span>ts<\/span> {<\/span>\r\n    <\/span>export<\/span> <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n    }<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
Those files will be rewritten to something like the following JavaScript code:<\/p>\n
\/\/ parser.js<\/span>\r\nvar<\/span> <\/span>ts<\/span>;<\/span>\r\n(<\/span>function<\/span> (<\/span>ts<\/span>) {<\/span>\r\n    <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    }<\/span>\r\n    <\/span>ts<\/span>.<\/span>createSourceFile<\/span> = <\/span>createSourceFile<\/span>;<\/span>\r\n})(<\/span>ts<\/span> || (<\/span>ts<\/span> = {}));<\/span>\r\n\r\n\/\/ program.js<\/span>\r\n(<\/span>function<\/span> (<\/span>ts<\/span>) {<\/span>\r\n    <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>ts<\/span>.<\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n    }<\/span>\r\n    <\/span>ts<\/span>.<\/span>createProgram<\/span> = <\/span>createProgram<\/span>;<\/span>\r\n})(<\/span>ts<\/span> || (<\/span>ts<\/span> = {}));<\/span>\r\n<\/code><\/pre>\n
The second, more subtle, and more significant issue is that our reference to createSourceFile<\/code> had to be rewritten to ts.createSourceFile<\/code>.\nRecall that this was actually something we liked<\/em> – it made it easy to reference exports across files.<\/p>\n
However, there is a runtime cost.\nUnfortunately, there are very few zero-cost abstractions in JavaScript, and invoking a method off of an object is more costly than directly invoking a function that’s in scope.\nSo running something like ts.createSourceFile<\/code> is more costly than createSourceFile<\/code>.<\/p>\n
The performance difference between these operations is usually negligible.\nOr at least, it’s negligible until you’re writing a compiler, where these operations occur millions of times over millions of nodes.\nWe realized this was a huge opportunity for us to improve a few years ago when Evan Wallace<\/a> pointed out this overhead on our issue tracker<\/a>.<\/p>\n
But namespaces aren’t the only construct that can run into this issue – the way most bundlers emulate scopes hits the same problem.\nFor example, consider if the TypeScript compiler were structured using modules like the following:<\/p>\n
\/\/ parser.ts<\/span>\r\nexport<\/span> <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n    <\/span>\/*...*\/<\/span>\r\n}<\/span>\r\n\r\n\/\/ program.ts<\/span>\r\nimport<\/span> { <\/span>createSourceFile<\/span> } <\/span>from<\/span> <\/span>".\/parser"<\/span>;<\/span>\r\n\r\nexport<\/span> <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n    <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n}<\/span>\r\n<\/code><\/pre>\n
A naive bundler might always create a function to establish scope for every module, and place exports on a single object.\nIt might look something like the following:<\/p>\n
\/\/ Runtime helpers for bundle:<\/span>\r\nfunction<\/span> <\/span>register<\/span>(<\/span>moduleName<\/span>, <\/span>module<\/span>) { <\/span>\/*...*\/<\/span> }<\/span>\r\nfunction<\/span> <\/span>customRequire<\/span>(<\/span>moduleName<\/span>) { <\/span>\/*...*\/<\/span> }<\/span>\r\n\r\n\/\/ Bundled code:<\/span>\r\nregister<\/span>(<\/span>"parser"<\/span>, <\/span>function<\/span> (<\/span>exports<\/span>, <\/span>require<\/span>) {<\/span>\r\n    <\/span>exports<\/span>.<\/span>createSourceFile<\/span> = <\/span>function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>\/*...*\/<\/span>\r\n    };<\/span>\r\n});<\/span>\r\n\r\nregister<\/span>(<\/span>"program"<\/span>, <\/span>function<\/span> (<\/span>exports<\/span>, <\/span>require<\/span>) {<\/span>\r\n    <\/span>var<\/span> <\/span>parser<\/span> = <\/span>require<\/span>(<\/span>"parser"<\/span>);<\/span>\r\n\r\n    <\/span>exports<\/span>.<\/span>createProgram<\/span> = <\/span>function<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n        <\/span>parser<\/span>.<\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n    };<\/span>\r\n});<\/span>\r\n\r\nvar<\/span> <\/span>parser<\/span> = <\/span>customRequire<\/span>(<\/span>"parser"<\/span>);<\/span>\r\nvar<\/span> <\/span>program<\/span> = <\/span>customRequire<\/span>(<\/span>"program"<\/span>);<\/span>\r\nmodule<\/span>.<\/span>exports<\/span> = {<\/span>\r\n    <\/span>createSourceFile:<\/span> <\/span>parser<\/span>.<\/span>createSourceFile<\/span>,<\/span>\r\n    <\/span>createProgram:<\/span> <\/span>program<\/span>.<\/span>createProgram<\/span>,<\/span>\r\n};<\/span>\r\n<\/code><\/pre>\n
Each reference of createSourceFile<\/code> now has to go through parser.createSourceFile<\/code>, which would still have more runtime overhead compared to if createSourceFile<\/code> was declared locally.\nThis is partially necessary to emulate the "live binding" behavior of ECMAScript modules – if someone modifies createSourceFile<\/code> within parser.ts<\/code>, it will be reflected in program.ts<\/code> as well.\nIn fact, the JavaScript output here can get even worse, as re-exports often need to be defined in terms of getters<\/a> – and the same is true for every intermediate re-export too!\nBut for our purposes, let’s just pretend bundlers always write properties and not getters.<\/p>\n
So if bundled modules can also run into these issues, why did we even mention the issues around boilerplate and indirection with namespaces?<\/p>\n
Well because the ecosystem around modules is rich, and bundlers have gotten surprisingly good at optimizing some of this indirection away!\nA growing number of bundling tools are able to not just aggregate multiple modules into one file, but they’re able to perform something called scope hoisting<\/em>.\nScope hoisting attempts to move as much code as possible into the fewest possible shared scopes.\nSo a bundler which performs scope-hoisting might be able to rewrite the above as<\/p>\n
function<\/span> <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n    <\/span>\/*...*\/<\/span>\r\n}<\/span>\r\n\r\nfunction<\/span> <\/span>createProgram<\/span>(<\/span>\/*...*\/<\/span>) {<\/span>\r\n    <\/span>createSourceFile<\/span>(<\/span>\/*...*\/<\/span>);<\/span>\r\n}<\/span>\r\n\r\nmodule<\/span>.<\/span>exports<\/span> = {<\/span>\r\n    <\/span>createSourceFile<\/span>,<\/span>\r\n    <\/span>createProgram<\/span>,<\/span>\r\n};<\/span>\r\n<\/code><\/pre>\n
Putting these declarations in the same scope is typically a win simply because it avoids adding boilerplate code to simulate scopes in a single file – lots of those scope setups and teardowns can be completely eliminated.\nBut because scope hoisting colocates declarations, it also<\/em> makes it easier for engines to optimize our uses of different functions.<\/p>\n
So moving to modules was not just an opportunity to build empathy and iterate more easily – it was a chance for us to make things faster!<\/p>\n
The Migration<\/h2>\n
Unfortunately there’s not a clear 1:1 translation for every codebase using namespaces to modules.<\/p>\n
We had some specific ideas of what we wanted our codebase to look like with modules.\nWe definitely wanted to avoid too much disruption to the codebase stylistically, and didn’t want to run into too many "gotchas" through auto-imports.\nAt the same time, our codebase had implicit cycles and that presented its own set of issues.<\/p>\n
To perform the migration, we worked on some tooling specific to our repository which we nicknamed the "typeformer".\nWhile early versions<\/a> used the TypeScript API directly, the most up-to-date version<\/a> used David Sherret<\/a>‘s fantastic ts-morph<\/a> library.<\/p>\n
Part of the approach that made this migration tenable was to break each transformation into its own step and its own commit.\nThat made it easier to iterate on specific steps without having to worry about trivial but invasive differences like changes in indentation.\nEach time we saw something that was "wrong" in the transformation, we could iterate.<\/p>\n
A small (see: very annoying<\/em>) snag on this transformation was how exports across modules are implicitly resolved.\nThis created some implicit cycles that were not always obvious, and which we didn’t really want to reason about immediately.<\/p>\n
But we were in luck – TypeScript’s API needed to be preserved through something called a "barrel" module – a single module that re-exports all the stuff from every other module.\nWe took advantage of that and applied an "if it ain’t broke, don’t fix it (for now)" approach when we generated imports.\nIn other words, in cases where we couldn’t create direct imports from each module, the typeformer simply generated imports from that barrel module.<\/p>\n
\/\/ program.ts<\/span>\r\nimport<\/span> { <\/span>createSourceFile<\/span> } <\/span>from<\/span> <\/span>".\/_namespaces\/ts"<\/span>; <\/span>\/\/ <- not directly importing from '.\/parser'.<\/span>\r\n<\/code><\/pre>\n
Picking a Bundler<\/h2>\n
There are some phenomenal new bundlers out there – so we thought about our requirements.\nWe wanted something that<\/p>\n
We figured eventually, we could (and thanks to a proposed change from Oleksandr Tarasiuk<\/a>, we will<\/a>), switch to direct imports across files.<\/p>\n
The first thing to notice is that each namespace is wrapped in an IIFE<\/a>.\nEach occurrence of a ts<\/code> namespace has the same setup\/teardown that’s repeated over and over again – which in theory<\/em> could be optimized away when producing a final output file.<\/p>\n