{"id":3985,"date":"2018-11-15T17:23:59","date_gmt":"2018-11-16T01:23:59","guid":{"rendered":"http:\/\/blogs.msdn.microsoft.com\/commandline\/?p=3985"},"modified":"2019-02-25T16:59:47","modified_gmt":"2019-02-26T00:59:47","slug":"windows-command-line-unicode-and-utf-8-output-text-buffer","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/commandline\/windows-command-line-unicode-and-utf-8-output-text-buffer\/","title":{"rendered":"Windows Command-Line: Unicode and UTF-8 Output Text Buffer"},"content":{"rendered":"

In this post, we’ll discuss the improvements we’ve been making to the Windows Console’s internal text buffer, enabling it to better store and handle Unicode and UTF-8 text.<\/p>\n

<\/p>\n

Posts in the Windows Command-Line series:<\/h2>\n

This list will be updated as more posts are published:<\/p>\n

    \n
  1. Command-Line Backgrounder<\/a><\/li>\n
  2. The Evolution of the Windows Command-Line<\/a><\/li>\n
  3. Inside the Windows Console<\/a><\/li>\n
  4. Introducing the Windows Pseudo Console (ConPTY) API<\/a><\/li>\n
  5. Unicode and UTF-8 Output Text Buffer<\/strong> [this post]<\/em><\/li>\n<\/ol>\n
    \n

    \"Study<\/a><\/p>\n

    [Source: David Farrell\u2019s \u201cBuilding a UTF-8 encoder in Perl\u201d]<\/a><\/p>\n

    The most visible aspect of a Command-Line Terminal is that it displays the text emitted from your shell and\/or Command-Line tools and apps, in a grid of mono-spaced cells \u2013 one cell per character\/symbol\/glyph. Great, that\u2019s simple. How hard can it be, right \u2013 it\u2019s just letters? Noooo! Read-on!<\/p>\n

    Representing Text<\/h2>\n

    Text is text is text. Or is it?<\/p>\n

    If you\u2019re someone who speaks a language that originated in Western Europe (e.g. English, French, German, Spanish, etc.), chances are that your written alphabet is pretty homogenous \u2013 10 digits, 26 separate letters \u2013 upper & lower case = 62 symbols in total. Now add around 30 symbols for punctuation and you\u2019ll need around 95 symbols in total. But if you\u2019re from East Asia (e.g. Chinese, Japanese, Korean, Vietnamese, etc.) you\u2019ll likely read and write text with a few more symbols \u2026 more than 7000 in total!<\/p>\n

    Given this complexity, how do computers represent, define, store, exchange\/transmit, and render these various forms of text in an efficient, and standardized\/commonly-understood manner?<\/p>\n

    In the beginning was ASCII<\/h2>\n

    The dawn of modern digital computing<\/a> was centralized around the UK and the US, and thus English was the predominant language and alphabet used.<\/p>\n

    As we saw above, the ~95 characters of the English alphabet (and necessary punctuation) can be individually represented using 7-bit values (0-127), with room left-over for additional non-visible control codes.<\/p>\n

    In 1963, the American National Standards Institute (ANSI) published the X3.4-1963 standard for the American Standard Code for Information Interchange (ASCII) \u2013 this became the basis of what we now know as the ASCII standard.<\/p>\n

    \n

    \u2026 and Microsoft gets a bad rap for naming things \ud83d\ude09\n The initial X3.4-1963 standard left 28 values undefined and reserved for future use. Seizing the opportunity, the International Telegraph and Telephone Consultative Committee (CCITT, from French: Comit\u00e9 Consultatif International T\u00e9l\u00e9phonique et T\u00e9l\u00e9graphique) proposed a change to the ANSI layout which caused the lower-case characters to differ in bit pattern from the upper-case characters by just a single bit. This simplified character case detection\/matching and the construction of keyboards and printers.<\/p>\n<\/blockquote>\n

    Over time, additional changes were made to some of the characters and control codes, until we ended up with the now well-established ASCII table of characters which is supported by practically every computing device in use today.<\/p>\n

    \"7-bit
    ]4<\/a> 7-bit ASCII Table<\/figcaption><\/figure><\/p>\n

    \u00a0<\/p>\n

    The rapid adoption of Computers in Europe presented a new challenge though: How to represent text in languages other than English. For example, how should letters with accents, umlauts, and additional symbols be represented?<\/p>\n

    To accomplish this, the ASCII table was extended with the addition of an extra bit, making characters 8-bits long, adding 127 \u201cextended characters\u201d:<\/p>\n

    \"Extended
    ]5<\/a> Extended 8-bit ASCII characters<\/figcaption><\/figure><\/p>\n

    \u00a0<\/p>\n

    But that that still didn\u2019t provide enough room to represent all the characters, glyphs and symbols required by computer users across the globe, many of whom needed to represent and display additional characters \/ glyphs.<\/p>\n

    So, Code pages<\/a> were introduced.<\/p>\n

    Code Pages \u2013 a partial solution<\/h2>\n

    Code pages define sets of characters for the \u201cextended characters\u201d from 0x80 \u2013 0xff (and, in some cases, a few of the non-displaying characters between 0x00 and 0x19). By selecting a different Code page, a Terminal can display additional glyphs for European languages and some block-symbols (see above), CJK<\/a> text, Vietnamese text, etc.<\/p>\n

    However, writing code to handle\/swap Code pages, and the lack of any standardization for Code pages in general, made text processing and rendering difficult, error prone, and presented major interop & user-experience challenges.<\/p>\n

    Worse still, 128 additional glyphs doesn\u2019t even come close to providing enough characters to represent some languages: For example, high-school level Chinese uses 2200 ideograms, with several hundred more in everyday use, and in excess of 7000 ideograms in total.<\/p>\n

    Clearly, code pages – additional sets of 128 chars \u2013 are not a scalable solution to this problem.<\/p>\n

    One approach to solving this problem was to add more bits \u2013 an extra 8-bits, in fact!<\/p>\n

    The Double Byte Character Set (DBCS)<\/a> code-page approach uses two bytes to represent a single character. This gives an addressable space of 2^16 \u2013 1 == 65,535 characters. However, despite attempts to standardize the Japanese Shift JIS<\/a> encoding, and the variable-length ASCII-compatible EUC-JP<\/a> encoding, DBCS code-page encodings were often riddled with issues<\/a> and did not deliver a universal solution to the challenge of encoding text.<\/p>\n

    What we really needed was a Universal Code for text data.<\/p>\n

    Enter, Unicode<\/h2>\n

    Unicode<\/a> is a set of standards that defines how text is represented and encoded.<\/p>\n

    The design of Unicode started in 1987 by engineers at Xerox and Apple. The initial Unicode-88 spec was published in February 1988, and has been continually refined and updated ever since, adding new character representations, additional language support, and even emoji \ud83d\ude0a<\/p>\n

    \n

    For a great history of Unicode, read this<\/a>!\n Today, Unicode supports up to 1,112,064 valid \u201ccodepoints\u201d each representing a single character \/ symbol \/ glyph \/ ideogram \/ etc. This should provide plenty of addressable codepoints for the future, especially considering that \u201cUnicode 11 currently defines 137,439 characters covering 146 modern and historic scripts<\/a>, as well as multiple symbol sets and emoji\u201d [source: Wikipedia<\/a>, Oct 2018]<\/p>\n

    \u201c1.2 million codepoints should be enough for anyone\u201d \u2013 source: Rich Turner, Oct 2018\n Unicode text data can be encoded in many ways, each with their strengths and weaknesses:<\/p>\n<\/blockquote>\n\n\n\n\n\n\n\n\n
    \n Encoding<\/b>\n <\/td>\n\n Notes<\/b>\n <\/td>\n\n # Bytes per codepoint<\/b>\n <\/td>\n\n Pros<\/b>\n <\/td>\n\n Cons<\/b>\n <\/td>\n<\/tr>\n
    \n UTF-32<\/a><\/b>\n <\/td>\n\n Each valid 32-bit value is a direct index to an individual Unicode codepoint\n <\/td>\n\n 4\n <\/td>\n\n No decoding required\n <\/td>\n\n Consumes a lot of space\n <\/td>\n<\/tr>\n
    \n UTF-16<\/a><\/b>\n <\/td>\n\n Variable-length encoding, requiring either one or two 16-bit values to represent each codepoint\n <\/td>\n\n 2\/4\n <\/td>\n\n Simple decoding\n <\/td>\n\n Consumes 2 bytes even for ASCII text. Can rapidly end-up requiring 4 bytes\n <\/td>\n<\/tr>\n
    \n UCS-2<\/b>\n <\/td>\n\n Precursor to UTF-16 Fixed-length 16-bit encoding used internally by Windows, Java, and JavaScript\n <\/td>\n\n 2\n <\/td>\n\n Simple decoding\n <\/td>\n\n Consumes 2 bytes even for ASCII text. Unable to represent some codepoints\n <\/td>\n<\/tr>\n
    \n UTF-8<\/a><\/b>\n <\/td>\n\n Variable-length encoding. Requires between one and four bytes to represent all Unicode codepoints\n <\/td>\n\n 1-4\n <\/td>\n\n Efficient, granular storage requirements\n <\/td>\n\n Moderate decoding cost\n <\/td>\n<\/tr>\n
    \n Others<\/b>\n <\/td>\n\n Other encodings<\/a> exist, but are not in widespread use\n <\/td>\n\n N\/A\n <\/td>\n\n N\/A\n <\/td>\n\n N\/A\n <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Due largely to its flexibility and storage\/transmission efficiency, UTF-8 has become the predominant text encoding mechanism on the Web: As of today (October 2018), 92.4% of all Web Pages are encoded in UTF-8<\/a>!<\/p>\n

    \"UTF-8
    ]16<\/a> UTF-8 encoding popularity for web pages (source: Wikipedia)<\/figcaption><\/figure><\/p>\n

    \u00a0<\/p>\n

    It\u2019s clear, therefore that anything that processes text should at least<\/em> be able to support UTF-8 text.<\/p>\n

    \n

    To learn more about text encoding and Unicode, read Joel Spolsky\u2019s great writeup here: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)<\/a><\/p>\n<\/blockquote>\n

    Console \u2013 built in a pre-Unicode dawn<\/h2>\n

    Alas, the Windows Console is not (currently) able to support UTF-8 text!<\/p>\n

    Windows Console was created way back in the early days of Windows, back before Unicode itself existed! Back then, a decision was made to represent each text character as a fixed-length 16-bit value (UCS-2). Thus, the Console\u2019s text buffer contains 2-byte wchar_t<\/code> values per grid cell, x<\/code> columns by y<\/code> rows in size.<\/p>\n

    While this design has supported the Console for more than 25 years, the rapid adoption of UTF-8 has started to cause problems:<\/p>\n

    One problem, for example, is that because UCS-2 is a fixed-width 16-bit encoding, it is unable to represent all Unicode codepoints<\/a>.<\/p>\n

    Another related but separate problem with the Windows Console is that because GDI is used to render Console\u2019s text, and GDI does not support font-fallback, Console is unable to display glyphs for codepoints that don\u2019t exist in the currently selected font!<\/p>\n

    \n

    Font-fallback is the ability to dynamically look-up and load a font that is similar-to the currently selected font, but which contains a glyph that\u2019s missing from the currently selected font\n These combined issues are why Windows Console cannot (currently) display many complex Chinese ideograms and cannot display emoji.<\/p>\n<\/blockquote>\n

    Emoji? SRSLY? This might at first sound trivial but is an issue since some tools now emit emoji to, for example, indicate test results, and some programming languages\u2019 source code supports\/requires Unicode, including emoji!<\/p>\n

    \"Emoji<\/a><\/p>\n

    [Source: You can use emoji characters in Swift variable, constant, function, and class names<\/a>]<\/p>\n

    But I digress \u2026<\/p>\n

    Text Attributes<\/h2>\n

    In addition to storing the text itself, a Console\/Terminal must store the foreground and background color, and any other per-cell information required.<\/p>\n

    These attributes must be stored efficiently and quickly \u2013 there\u2019s no need to store background and foreground information for each cell individually, especially since most Console apps\/tools output pretty uniformly colored text, but storing and retrieving attributes must not unnecessarily hinder rendering performance.<\/p>\n

    Let\u2019s dig in and find out how the Console handles all this! \ud83d\ude0a<\/p>\n

    Modernizing the Console\u2019s text buffer<\/h2>\n

    As discussed in the previous post in this series<\/a>, the Console team have been busy overhauling the Windows Console\u2019s internals for the last several Win10 releases, carefully modernizing, modularizing, simplifying, and improving the Console\u2019s code & features \u2026 while not noticeably sacrificing performance, and not changing current behaviors.<\/p>\n

    For each major change, we evaluate and prototype several approaches, and measure the Console\u2019s performance, memory footprint, power consumption, etc. to figure-out the best real-world solution. We took the same approach for the buffer improvements work which was started before 1803 shipped and continue beyond 1809.<\/p>\n

    The key issue to solve was that the Console previously stored each cell\u2019s text data as UCS-2 fixed-length 2-byte wchar_t<\/code> values.<\/p>\n

    To fully support all Unicode characters we needed a more flexible approach that added no noticeable processing or memory overhead for the general case, but was able to dynamically handle additional bytes of text data for cells that contain multi-byte Unicode characters.<\/p>\n

    We examined several approaches, and prototyped & measured a few, which helped us disqualify some potential approaches where turned-out to be ineffective in real-world use.<\/p>\n

    Adding Unicode Support<\/h2>\n

    Ultimately, we arrived at the following architecture:<\/p>\n

    \"Console
    ]21<\/a> Console Text Buffer Architecture<\/figcaption><\/figure><\/p>\n

    \u00a0<\/p>\n

    From the top (original buffer’s blue boxes):<\/p>\n