{"id":34973,"date":"2005-07-11T10:00:14","date_gmt":"2005-07-11T10:00:14","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2005\/07\/11\/converting-from-traditional-to-simplified-chinese-part-1-loading-the-dictionary\/"},"modified":"2005-07-11T10:00:14","modified_gmt":"2005-07-11T10:00:14","slug":"converting-from-traditional-to-simplified-chinese-part-1-loading-the-dictionary","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20050711-14\/?p=34973","title":{"rendered":"Converting from traditional to simplified Chinese, part 1: Loading the dictionary"},"content":{"rendered":"

\nOne step we had glossed over in our haste to get something\ninteresting on the screen in our\nChinese\/English dictionary program\nwas the conversion from traditional to simplified Chinese\ncharacters.\n<\/P>\n

\nThe format of the hcutf8.txt<\/CODE> file is a series of lines,\neach of which is a UTF-8 encoded string consisting of a simplified\nChinese character followed by its traditional equivalents.\nOften, multiple traditional characters map to a single\nsimplified character.\nMuch more rarely—only twice in our data set—multiple\nsimplified characters map to a single traditional character.\nUnfortunately, one of the cases is the common syllable\n麼, which has two simplifications, either\n么 or 麽, the first of which is far more productive.\nWe’ll have to keep an eye out for that one.\n<\/P>\n

\n(Note also that in real life,\n\nthe mapping is more complicated\nthan a character-for-character substitution<\/A>,\nbut I’m willing to forego that level of complexity\nbecause this is just for my personal use and people will have\nrealized I’m not a native speaker long before I get caught up\nin language subtleties like that.)\n<\/P>\n

\nOne could try to work out a fancy data structure to represent\nthis mapping table compactly, but it turns out that simple is\nbetter here: an array of 65536 WCHAR<\/CODE>s, each producing\nthe corresponding simplification.\nMost of the array will lie unused,\nsince the characters we are interested in lie in the range\nU+4E00 to U+9FFF.\nConsequently, the active part of the table is only about 40Kb,\nwhich easily fits inside the L2 cache.\n<\/P>\n

\nIt is important to know when\na simple data structure is better than a complex one.\n<\/P>\n

\nThe hcutf8.txt<\/CODE> file contains a lot of fluff that we\naren’t interested in. Let’s strip that out ahead of time so that\nwe don’t waste our time parsing it at run-time.\n<\/P>\n

\n#!perl\n$_ = <> until \/^# Start zi\/; # ignore uninteresting characters\nwhile (<>) {\n s\/\\r\/\/g;\n next if length($_) == 7 &&\n         substr($_, 0, 3) eq substr($_, 3, 3); # ignore NOPs\n print;\n}\n<\/PRE>\n
\nRun the hcutf8.txt<\/CODE> file through this filter to clean\nit up a bit.\n<\/P>\n
\nNow we can write our “traditional to simplified” dictionary.\n<\/P>\n
\nclass Trad2Simp\n{\npublic:\n Trad2Simp();\n WCHAR Map(WCHAR chTrad) const { return _rgwch[chTrad]; }<\/p>\n
private:\n WCHAR _rgwch[65536]; \/\/ woohoo!\n};<\/p>\n
Trad2Simp::Trad2Simp()\n{\n ZeroMemory(_rgwch, sizeof(_rgwch));<\/p>\n
 MappedTextFile mtf(TEXT(“hcutf8.txt”));\n const CHAR* pchBuf = mtf.Buffer();\n const CHAR* pchEnd = pchBuf + mtf.Length();\n while (pchBuf < pchEnd) {\n  const CHAR* pchCR = std::find(pchBuf, pchEnd, ‘\\r’);\n  int cchBuf = (int)(pchCR – pchBuf);\n  WCHAR szMap[80];\n  DWORD cch = MultiByteToWideChar(CP_UTF8, 0, pchBuf, cchBuf,\n                                  szMap, 80);\n  if (cch > 1) {\n   WCHAR chSimp = szMap[0];\n   for (DWORD i = 1; i < cch; i++) {\n    if (szMap[i] != chSimp) {\n     _rgwch[szMap[i]] = chSimp;\n    }\n   }\n   pchBuf = std::find(pchCR, pchEnd, ‘\\n’) + 1;\n  }\n }\n _rgwch[0x9EBC] = 0x4E48;\n}\n<\/PRE>\n
\nWe read the file one line at a time, convert it from UTF-8,\nand for each nontrivial mapping, record it in our dictionary.\nAt the end, we do our little 么 special-case patch-up.\n<\/P>\n
\nNext time, we’ll use this mapping table to generate simplified\nChinese characters into our dictionary.\n<\/P><\/p>\n","protected":false},"excerpt":{"rendered":"
One step we had glossed over in our haste to get something interesting on the screen in our Chinese\/English dictionary program was the conversion from traditional to simplified Chinese characters. The format of the hcutf8.txt file is a series of lines, each of which is a UTF-8 encoded string consisting of a simplified Chinese character […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-34973","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
One step we had glossed over in our haste to get something interesting on the screen in our Chinese\/English dictionary program was the conversion from traditional to simplified Chinese characters. The format of the hcutf8.txt file is a series of lines, each of which is a UTF-8 encoded string consisting of a simplified Chinese character […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/34973","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=34973"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/34973\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=34973"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=34973"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=34973"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}