{"id":106401,"date":"2022-03-28T07:00:00","date_gmt":"2022-03-28T14:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=106401"},"modified":"2022-06-06T20:33:09","modified_gmt":"2022-06-07T03:33:09","slug":"20220328-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20220328-00\/?p=106401","title":{"rendered":"Why is every other line of my file mysteriously changed to nonsense Chinese characters?"},"content":{"rendered":"

A customer found that every other line of their file mysteriously changed to nonsense Chinese characters:<\/p>\n

\/\/ Microsoft Visual C++ generated resource script.\r\n\u2300\u6900\u6e00\u6300\u6c00\u7500\u6400\u6500\u2000\u2200\u7200\u6500\u7300\u6f00\u7500\u7200\u6300\u6500\u2e00\u6800\u2200\u0d00\r\n#define APSTUDIO_READONLY_SYMBOLS\r\n\u2f00\u2f00\u2000\u4700\u6500\u6e00\u6500\u7200\u6100\u7400\u6500\u6400\u2000\u6600\u7200\u6f00\u6d00\u2000\u7400\u6800\u6500\u2000\u5400\u4500\u5800\u5400\u4900\u4e00\u4300\u4c00\u5500\u4400\u4500\u2000\u7200\u6500\u7300\u6f00\u7500\u7200\u6300\u6500\u2e00\u0d00\r\n<\/pre>\n
My goodness, that does indeed look mysterious.<\/p>\n
But it’s less mysterious if you look at it in a hex editor:<\/p>\n
FF FE 2F 00 2F 00 4D 00-69 00 63 00 72 00 6F 00  ..\/.\/.M.i.c.r.o.\r\n73 00 6F 00 66 00 74 00-20 00 56 00 69 00 73 00  s.o.f.t. .V.i.s.\r\n75 00 61 00 6C 00 20 00-43 00 2B 00 2B 00 20 00  u.a.l. .C.+.+. .\r\n67 00 65 00 6E 00 65 00-72 00 61 00 74 00 65 00  g.e.n.e.r.a.t.e.\r\n64 00 20 00 72 00 65 00-73 00 6F 00 75 00 72 00  d. .r.e.s.o.u.r.\r\n63 00 65 00 20 00 73 00-63 00 72 00 69 00 70 00  c.e. .s.c.r.i.p.\r\n74 00 2E 00 0D 00 0D<\/span> 0A-00 23 00 69 00 6E 00 63<\/span>  t........#.i.n.c\r\n00 6C 00 75 00 64 00 65-00 <\/span>20 00 22 00 72 00 65<\/span>  .l.u.d.e. .\".r.e\r\n00 73 00 6F 00<\/span> 75 00 72-00 63 00 65 00 2E 00 68<\/span>  .s.o.u.r.c.e...h\r\n00 22 00 0D 00 0D<\/span> 0A 00<\/span>-23 00 64 00 65 00 66 00  .\"......#.d.e.f.\r\n69 00 6E 00 65 00 20 00-41 00 50 00 53 00 54 00  i.n.e. .A.P.S.T.\r\n55 00 44 00 49 00 4F 00-5F 00 52 00 45 00 41 00  U.D.I.O._.R.E.A.\r\n44 00 4F 00 4E 00 4C 00-59 00 5F 00 53 00 59 00  D.O.N.L.Y._.S.Y.\r\n4D 00 42 00 4F 00 4C 00-53 00 0D 00 0D<\/span> 0A 00 2F<\/span>  M.B.O.L.S......\/\r\n00 2F 00 20 00 47 00 65-00 6E 00 65 00 72 00 <\/span>61<\/span>  .\/. .G.e.n.e.r.a\r\n00 74 00 65 00 64 00 20-00 66 00 72 00 6F 00 6D<\/span>  .t.e.d. .f.r.o.m\r\n00 20 00 74 00 68 00 65-00 20 00 54 00 45 00 58<\/span>  . .t.h.e. .T.E.X\r\n00 54 00 49 00 4E 00 43-00 4C 00 55 00 44 00 45<\/span>  .T.I.N.C.L.U.D.E\r\n00 20 00 72 00 65 00 73-00 6F<\/span> 00 75 00 72 00 63<\/span>  . .r.e.s.o.u.r.c\r\n00 65 00 2E 00 0D 00 0D<\/span>-0A 00<\/span> 2F 00 2F 00 0D 00  .e........\/.\/...\r\n<\/pre>\n
The first few bytes of the file are clearly recognizable as encoded in UTF-16LE, complete with U+FEFF BOM.<\/p>\n
And then something weird happens at the end of the first line of text: You would expect the line to end with the byte sequence 0D 00 0A 00<\/tt>, corresponding to the UTF-16LE encoding of U+000D CARRIAGE RETURN followed by U+000A LINE FEED. But instead, there’s this extra 0D<\/tt> byte stuck into the stream before the 0A<\/tt>, and that messes things up.<\/p>\n
Since UTF-16LE encoding uses pairs of bytes, inserting a single byte throws off the synchronization. All the even bytes become odd, and all odd bytes become even. This is why the second line (outlined) comes out as Chinese nonsense: Putting ASCII alphabetics in the high-order byte of UTF-16 code units  results in nonsense Chinese characters<\/a>.<\/p>\n
When we get to the end of the (now garbled) line, again there is a mystery 0D<\/tt> interloper byte immediately before the 0A<\/tt>. This second unwanted byte restores parity, which means that the bytes of the third line of text are properly paired again, and they appear normal.<\/p>\n
And then when we get to the end of the third line, the cycle repeats, with a rogue 0D<\/tt> throwing off the synchronization and causing the fourth line to become gibberish. This continues for the remainder of the file: The corrupted line ending causes every other line to turn into nonsense.<\/p>\n
Okay, so now we know what’s going on, but how did the file get corrupted in this way?<\/p>\n
The insertion of 0D<\/tt> before 0A<\/tt> is coming from CRLF\/LF conversion. Windows text files  end with carriage return and line feed<\/a>, but Unix text files end with just line feed. The usual algorithm for converting from Windows format to Unix format is to remove carriage returns if they are immediately followed by a line feed. And the usual algorithm for converting from Unix format to Windows format is to insert carriage returns before every line feed.<\/p>\n
Of course, if you “insert” and “remove” the carriage return, you have to do so with the correct encoding.<\/p>\n
What happened here is that the original file was encoded in UTF-16LE and used Windows line endings. My guess is that this file was then stored in a system that uses Unix line endings, so the line endings were converted, but the conversion code interprets the file as UTF-8, which encodes a CRLF as the byte sequence 0D 0A<\/tt>. That sequence does not occur in the file, so the conversion is a nop, and the file goes into storage unchanged.<\/p>\n
Some time later, the file is retrieved from storage, and the line endings are now converted from Unix-style to Windows-style. Again, the conversion code assumes the file is UTF-8 encoded, so it inserts a carriage return 0D<\/tt> before every line feed 0A<\/tt>.<\/p>\n
And that is why the 0A<\/tt> bytes are being inserted and messing up the file.<\/p>\n
There is a German word for this: ver\u00adschlimm\u00adbessern<\/i>: To make something worse in a well-intentioned but failed attempt to improve it.<\/p>\n
A common place you’ll see this type of corruption is when you ask Visual Studio to create a new project with a git repo.<\/p>\n
Some Visual Studio project templates create source files encoded as UTF-16LE, and it’s common to add a .gitattributes<\/tt> file that says that source files are text eol=crlf<\/tt>. If you do that, then git follows your instructions and converts files from Windows line endings to Unix line endings when committing, and performs the reverse conversion when checking files out.<\/p>\n
However, git assumes text files are encoded as UTF-8.<\/p>\n
To avoid Verschlimmbesserung<\/i>, you should re-encode your UTF-16LE files as UTF-8 before adding them to the git repo. Note that Resource Compiler *.rc files  default to the ANSI code page<\/a>, even if the file begins with a UTF-8 BOM. You need to explicitly inform the Resource Compiler that the file is encoded as UTF-8 by saying<\/p>\n
#pragma code_page(65001) \/\/ UTF-8\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
Decoding another type of mojibake.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[26],"class_list":["post-106401","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-other"],"acf":[],"blog_post_summary":"
Decoding another type of mojibake.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/106401","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=106401"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/106401\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=106401"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=106401"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=106401"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}