{"id":44613,"date":"2015-02-23T07:00:00","date_gmt":"2015-02-23T22:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2015\/02\/23\/further-adventures-in-trying-to-guess-what-encoding-a-file-is-in\/"},"modified":"2019-03-13T12:13:11","modified_gmt":"2019-03-13T19:13:11","slug":"20150223-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20150223-00\/?p=44613","title":{"rendered":"Further adventures in trying to guess what encoding a file is in"},"content":{"rendered":"

The Is­Text­Unicode<\/code> function tries to guess the encoding of a block of memory purporting to contain text, but it can only say “Looks like Unicode” or “Doesn’t look like Unicode”, and there some notorious examples<\/a> of where it guesses wrong<\/a>. <\/p>\n

A more flexible alternative is IMulti­Language2::Detect­Code­page­In­IStream<\/code><\/a> and its buffer-based equivalent IMulti­Language2::Detect­Input­Code­page<\/code><\/a>. Not only can these methods detect a much larger range of code pages, they also can report multiple code pages, each with a corresponding confidence level. <\/p>\n

Here’s a Little Program that takes the function out for a spin. (Remember, Little Programs do little to no error checking.) <\/p>\n

\n#define UNICODE\n#define _UNICODE\n#include <windows.h>\n#include <shlwapi.h>\n#include <ole2.h>\n#include <mlang.h>\n#include <shlwapi.h>\n#include <atlbase.h>\n#include <stdio.h>\n\nbool IsHtmlFile(PCWSTR pszFile)\n{\n PCWSTR pszExtension = PathFindExtensionW(pszFile);\n return\n  CompareStringOrdinal(pszExtension, -1,\n                       L\".htm\", -1, TRUE) == CSTR_EQUAL ||\n  CompareStringOrdinal(pszExtension, -1,\n                        L\".html\", -1, TRUE) == CSTR_EQUAL;\n}\n\nint __cdecl wmain(int argc, wchar_t **argv)\n{\n if (argc < 2) return 0;\n CCoInitialize<\/a> init;\n CComPtr<IStream> spstm;\n SHCreateStreamOnFileEx(argv[1], STGM_READ, 0, FALSE, nullptr, &spstm);\n\n CComPtr<IMultiLanguage2> spml;\n CoCreateInstance(CLSID_CMultiLanguage, NULL,\n     CLSCTX_ALL, IID_PPV_ARGS(&spml));\n\n DetectEncodingInfo info[10];\n INT cInfo = ARRAYSIZE(info);\n\n DWORD dwFlag = IsHtmlFile(argv[1]) ? MLDETECTCP_HTML\n                                    : MLDETECTCP_NONE;\n HRESULT hr = spml->DetectCodepageInIStream(\n     dwFlag, 0, spstm, info, &cInfo);\n if (hr == S_OK) {\n  for (int i = 0; i < cInfo; i++) {\n   wprintf(L\"info[%d].nLangID = %d\\n\", i, info[i].nLangID);\n   wprintf(L\"info[%d].nCodePage = %d\\n\", i, info[i].nCodePage);\n   wprintf(L\"info[%d].nDocPercent = %d\\n\", i, info[i].nDocPercent);\n   wprintf(L\"info[%d].nConfidence = %d\\n\", i, info[i].nConfidence);\n  }\n } else {\n  wprintf(L\"Cannot determine the encoding (error: 0x%08x)\\n\", hr);\n }\n return 0;\n}\n<\/pre>\n
Run the program with a file name as the command line argument, and the program will report all the detected code pages. <\/p>\n
One thing that may not be obvious is that the program passes the MLDETECTCP_HTML<\/code> flag if the file extension is .htm<\/code> or .html<\/code>. That is a hint to the detector that it shouldn’t get faked out by text like <body><\/code> and think it found an English word. <\/p>\n
Here’s the output of the program when run on its own source code: <\/p>\n
\ninfo[0].nLangID = 9\ninfo[0].nCodePage = 20127\ninfo[0].nDocPercent = 100\ninfo[0].nConfidence = 83\ninfo[1].nLangID = -1\ninfo[1].nCodePage = 65001\ninfo[1].nDocPercent = -1\ninfo[1].nConfidence = -1\n<\/pre>\n
This says that its first guess is that the text is in language 9, which is LANG_ENGLISH<\/code><\/a>, code page 20127, which is US-ASCII<\/a>, That text occupies 100% of the file, and the confidence level is 83. <\/p>\n
The second guess is that the text is in code page 65001, which is UTF-8, but the confidence level for that is low. <\/p>\n
The language-guessing part of the function is not very sophisticated. For a higher-quality algorithm for guessing what language some text is in, use Extended Linguistic Services<\/a>. I won’t bother writing a sample application because MSDN already contains one<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":"
Today’s candidate: IMultiLanguage2::DetectCodepageInIStream<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-44613","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Today’s candidate: IMultiLanguage2::DetectCodepageInIStream<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/44613","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=44613"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/44613\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=44613"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=44613"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=44613"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}