Recently there were some updates that changed the behavior of a national encoding standard, changing the behavior when mapping to and from Unicode.\u00a0 Those kinds of changes lead to data corruption.\u00a0 I figured I’d take this opportunity to remind folks of the benefits of “using Unicode!”<\/p>\n
Modern operating systems are Unicode, usually UTF-16 or UTF-8 internally.\u00a0 Converting between other codepages, particularly table-based ones, consumes resources and takes time.\u00a0 Even if there’s a difference between UTF-8 data and a UTF-16 platform, that conversion is algorithmic and heavily optimized to be fast.\u00a0 You can’t get that from other “national” codepages.\u00a0 Because they require tables, which have to be mapped into memory, or at least the CPU cache, it slows down the conversions.<\/p>\n
On Windows, it is much faster to encode or decode data using UTF-8 than with the legacy Windows-1252 encoding.<\/p>\n
Consistent use of Unicode is the most reliable way to protect the integrity of your user’s data.\u00a0 Every other codepage is subject to potential data corruption in various ways.<\/p>\n
Missing Codepoints<\/strong> – The most obvious source of corruption is when data is encountered that doesn’t fit in the encoding’s character set.\u00a0 Then they get replaced with ? or other replacement characters.\u00a0 Even seemingly simple scenarios like customer names can run afoul of this.\u00a0 A customer prefers Th\u00e9o to Theo, and the application was only expecting ASCII.\u00a0 Then we end up with Th?o instead, pleasing no one.<\/p>\n Encoding Stability<\/strong> – Over the years, nearly all of the encodings have undergone some churn.\u00a0 For some it is relatively minor.\u00a0 Others have added major blocks of codepoints, or corrected errors in the assignment of the codepoints.\u00a0 In such cases, data encoded in one version of an encoding may not decode the same way in a slightly different version.\u00a0 Such as after an OS update or some such.\u00a0 Especially in the modern world with computers distributed around the world, it’s usually impossible to guarantee consistent versions are being used.\u00a0 Even when confined to the same platform.<\/p>\n Platform Differences<\/strong> – As encodings evolved, platforms have adopted the updates at differencing paces.\u00a0 Some may choose the most recent and perceived “correct” encoding to keep up to date, at the expense of data compatibility.\u00a0 Others may update slowly, or never, choosing data stability for their customers.\u00a0 Additionally, there are the occasional implementation errors and other factors.\u00a0 The net result is that most platforms and browsers have minor differences in their character encodings.\u00a0 In particular for the larger character sets.\u00a0 This leads to the occasional random unexpected glyph or ? in decoded data.<\/p>\n Most applications have to interact with other software libraries, platforms, or operating systems, so these sorts of corruptions are unfortunately common.<\/p>\n Encoding (Mis)tagging<\/strong> – Similarly, to properly exchange encoded data, the sender and recipient must agree on the encoding.\u00a0 Common errors are using Windows-1252, ISO-8859-1, &\/or ASCII nearly interchangeably for Latin.\u00a0 Perhaps writing a document on a system that uses one encoding and then hosting it on server that uses a different default.\u00a0 Since they’re all Latin based and the ASCII range is similar, this can be difficult to notice.\u00a0 I’m sure we’ve all seen this type of corruption, commonly showing up with error such as “quoted” text, where the quotes are replaced with gibberish characters.\u00a0 Or a trademark or other symbol being mis-mapped.<\/p>\n A related problem is protocols that don’t really provide for the labeling of the encoding at all.\u00a0 Or applications that don’t provide an encoding even when permitted.<\/p>\n Unicode has become the most common encoding to store data, particularly on the Internet.\u00a0 Web metrics show that over 98% of web pages are encoded in UTF-8.\u00a0 Much of the remainder is in one of the common Latin based defaults (ASCII, ISO-8859-1, or Windows-1252).\u00a0 Even in the Asian regions that had a historical need for national character sets to support their large character repertoire, over 95% of the web data in those regions is in UTF-8.<\/p>\n ASCII is a special case, because it’s pretty common.\u00a0 And sometimes it’s mixed up with Windows-1252 or ISO-8859-1.\u00a0 There’s no advantage to using ASCII over UTF-8.\u00a0 UTF-8 is a superset of ASCII, so you get all of the benefits of ASCII, plus the ability to encode any other character that the computer can use.\u00a0 If you are looking for a single byte character set to support Latin scripts, then pick UTF-8.\u00a0 It reduces ambiguity, is about the same size, and will still allow encoding every character you might unexpectedly encounter.<\/p>\n Where you really want to Use Unicode! is in protocols.\u00a0 Anywhere where you’re exchanging data with another system or component, you want a clean Unicode interface.\u00a0 Oftentimes this might be UTF-8 for data on the wire, or UTF-16 for internal API calls on a UTF-16 system.\u00a0 Specifying Unicode for the interfaces removes any ambiguity about which encoding is necessary.\u00a0 And it reduces the risk of mis-tagged data sneaking through one or more layers of the system.\u00a0 It also allows any character that users might need to use, even if you initially intend the software to only be used in a Latin script market.<\/p>\n The legacy Windows “ANSI” (a misnomer) APIs have severe limitations compared to their Unicode counterparts.\u00a0 Historically, these used the system codepage prior to Windows support of Unicode.\u00a0 And they continue to be used by apps that don’t use Unicode.<\/p>\n These legacy APIs have different behavior, depending on the system codepage settings.\u00a0 Those differences can even lead to data loss when shared between systems in different regions.\u00a0 They usually internally convert to Unicode, then back to the system encoding, making them slower than the native “W” Unicode APIs.\u00a0 And they often don’t provide the functionality of their preferred Unicode counterparts.<\/p>\n Now that I’ve convinced everyone to Use Unicode!, I do recognize that there is a lot of legacy data that isn’t stored in Unicode.\u00a0 And that many processes may depend on older systems that were designed for a different encoding.\u00a0 Eventually nearly all of that data will move to Unicode – because the entire industry is continuing to move in that direction.\u00a0 In the meantime, try to confine those legacy data structures behind Unicode enabled interfaces.\u00a0 That way at least the problem is contained, and when it does eventually get updated the number of additional pieces that need to be updated are smaller.<\/p>\n Recently there were some updates that changed the behavior of a national encoding standard, changing the behavior when mapping to and from Unicode.\u00a0 Those kinds of changes lead to data corruption.\u00a0 I figured I’d take this opportunity to remind folks of the benefits of “using Unicode!” Unicode Benefits All modern operating systems are natively Unicode.\u00a0 […]<\/p>\n","protected":false},"author":17042,"featured_media":6,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-190","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-i18n"],"acf":[],"blog_post_summary":" Recently there were some updates that changed the behavior of a national encoding standard, changing the behavior when mapping to and from Unicode.\u00a0 Those kinds of changes lead to data corruption.\u00a0 I figured I’d take this opportunity to remind folks of the benefits of “using Unicode!” Unicode Benefits All modern operating systems are natively Unicode.\u00a0 […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/posts\/190","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/users\/17042"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/comments?post=190"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/posts\/190\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/media\/6"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/media?parent=190"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/categories?post=190"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/i18n\/wp-json\/wp\/v2\/tags?post=190"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Usage on the Internet<\/h2>\n
UTF-8 Versus ASCII<\/h2>\n
Protocols and Data Interchange<\/h2>\n
Legacy “A” APIs<\/h2>\n
Legacy Data<\/h2>\n
Use Unicode!<\/h2>\n","protected":false},"excerpt":{"rendered":"