{"id":110692,"date":"2024-12-30T07:00:00","date_gmt":"2024-12-30T15:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=110692"},"modified":"2024-12-30T07:46:00","modified_gmt":"2024-12-30T15:46:00","slug":"20241230-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20241230-00\/?p=110692","title":{"rendered":"How various git diff viewers represent file encoding changes in pull requests"},"content":{"rendered":"

In addition to the git command line tool, there are other tools or services that let you view changes in git history. The most interesting cases are those which present changes as part of a pull request, since those are changes you are reviewing and approving. But a common problem is that what they show you might not be what actually changed.<\/p>\n

I’ll limit my discussion to services and tools I have experience with, which means that it’s the git command line, Azure DevOps, GitHub, and Visual Studio. You are welcome to share details for other services that you use, particularly those used for code reviews.<\/p>\n

First, let’s consider a commit that changes the encoding of a file. For concreteness, let’s say that the file is this:<\/p>\n

I just checked.\r\nIt costs A3<\/span>1.\r\n<\/pre>\n
where A3<\/span><\/tt> represents a single byte with hex value 0xA3<\/tt>. This is the representation of \u00a3 in the Windows 1252 code page.<\/p>\n
Suppose you change the encoding of this file to UTF-8:<\/p>\n
It costs C2<\/span>A3<\/span>1.\r\n<\/pre>\n
If you view this in the command line with git show<\/tt> you get<\/p>\n
\u00a0\u00a0I just checked.<\/tt><\/p>\n
- It costs <A3><\/span>1.<\/tt><\/div>\n
+ It costs \u00a31.<\/tt><\/div>\n<\/div>\n
The command line version shows you that there used to be a byte 0xA3<\/tt> but now there is a \u00a3 character.<\/p>\n
Next up is GitHub. Its diff says<\/p>\n
\n
\u00a0\u00a0I just checked.<\/tt><\/div>\n
- It costs \ufffd1.<\/tt><\/div>\n
- It costs \u00a31.<\/tt><\/div>\n<\/div>\n
GitHub assumes that all files are in UTF-8, so it interprets the A3<\/span> as an illegal UTF-8 code unit sequence and represents it with U+FFFD REPLACEMENT CHARACTER.<\/p>\n
Next up is Team Foundation Services<\/span> Visual Studio Online<\/span> Visual Studio Team Services<\/span> Azure DevOps. Azure DevOps. That’s the name. Azure DevOps.<\/p>\n
Here’s what Azure DevOps shows:<\/p>\n
\u26a0 The file differs only in whitespace.<\/div>\n
And if you expand the file and enable “Show whitespace changes”, it shows you no changes, not even whitespace changes!<\/p>\n
\n
I just checked.<\/tt><\/div>\n
It costs \u00a31.<\/tt><\/div>\n<\/div>\n
This is quite concerning, because it means that if you made a change to the text of a file and also changed the encoding, Azure DevOps highlights the text changes, but does not give any indication that the encoding changed!<\/p>\n
For example, maybe somebody changed the first line of text and accidentally changed the encoding from 1252 to UTF-8. Azure DevOps shows this as<\/p>\n
\n
I just checked<\/span>.<\/tt><\/div>\n
I just looked<\/span>.<\/tt><\/div>\n
It costs \u00a31.<\/tt><\/div>\n
It happily shows you the text change, but completely ignores the encoding change.<\/p>\n
That encoding change might have caused you to  inadvertently change a bunch of strings in a Resource Script<\/a>, resulting in mojibake.<\/p>\n
If you ask Visual Studio to view the diff, it indicates that the file has been modified (M), but when you ask to see the diff, it says “0 changes”, and nothing is highlighted.<\/p>\n
Now let’s consider a commit that inserted a UTF-8 BOM at the start of a file.<\/p>\n
From the command line with git<\/tt>, you get this:<\/p>\n
\n
- I just checked.<\/tt><\/div>\n
+\u00a0 I just checked.<\/tt><\/div>\n<\/div>\n
The BOM displays as a space. Not great, but at least there is a +\/\u2212 to show you that something<\/i> changed, and if the first line is not otherwise blank, the shifted contents tell you that something<\/i> got inserted at the start of the file.<\/p>\n
For GitHub, the diff shows up like this:<\/p>\n
\n
- I just checked.<\/tt><\/div>\n
- I just checked.<\/tt><\/div>\n<\/div>\n
The highlights tell you that something changed on that line, but squint all you want, you don’t see any change. The change must be invisible, but at least you’re told that there’s a change somewhere<\/i> on that line; you just can’t see it.<\/p>\n
And finally, we have Azure DevOps:<\/p>\n
\u26a0 The file differs only in whitespace.<\/div>\n
As before, even if you expand the file and enable “Show whitespace changes”, you get no changes.<\/p>\n
\n
I just checked.<\/tt><\/div>\n
It costs \u00a31.<\/tt><\/div>\n<\/div>\n
So Azure DevOps tells you that the file changed in whitespace, but when you ask to see it, you are shown no changes.<\/p>\n
If you ask Visual Studio to view the diff, it once again indicates that the file has been modified (M), but when you ask to see the diff, it says “0 changes”, and nothing is highlighted.<\/p>\n
I suspect that in the cases where GitHub, Azure DevOps, or Visual Studio show no visible changes, most users will just conclude, “Must be a bug,” and not realize that no really, there’s a change in there that you can’t see.<\/p>\n
So let’s summarize these results in a table.<\/p>\n\n\n\n\n\n\n
\u00a0<\/th>\ngit command line<\/th>\nGitHub<\/th>\nAzure DevOps<\/th>\nVisual Studio<\/th>\n<\/tr>\n
Code page<\/th>\nUTF-8<\/td>\nUTF-8<\/td>\nGuess<\/td>\nGuess<\/td>\n<\/tr>\n
Encoding changes<\/th>\nShown in diff<\/td>\nShown in diff<\/td>\nNo change shown<\/td>\nNo change shown<\/td>\n<\/tr>\n
BOM change<\/th>\nShow as space<\/td>\nInvisible<\/td>\nNo change shown<\/td>\nNo change shown<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
My take-away from this table is that if you do your work with any of these systems, you need to pay close attention when dealing with files that contain characters outside the 7-bit ASCII set because changes to encoding or the presence of a BOM can be hard to spot, or even become outright invisible, even though it drastically changes what the contents of the file mean.<\/p>\n","protected":false},"excerpt":{"rendered":"
The invisible UTF-8 BOM, and sometimes invisible encoding changes.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[26],"class_list":["post-110692","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-other"],"acf":[],"blog_post_summary":"
The invisible UTF-8 BOM, and sometimes invisible encoding changes.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/110692","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=110692"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/110692\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=110692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=110692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=110692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}