{"id":35633,"date":"2005-05-16T08:56:30","date_gmt":"2005-05-16T08:56:30","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2005\/05\/16\/loading-the-dictionary-part-4-character-conversion-redux\/"},"modified":"2005-05-16T08:56:30","modified_gmt":"2005-05-16T08:56:30","slug":"loading-the-dictionary-part-4-character-conversion-redux","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20050516-30\/?p=35633\/","title":{"rendered":"Loading the dictionary, part 4:  Character conversion redux"},"content":{"rendered":"<p>\nGetting rid of <code>getline<\/code> was a big help, but 480ms\nis still not quite peppy enough.  You need to respond to user\nactions within a tenth of a second for thing to seem responsive.\n<\/p>\n<p>\nProfiling the latest endeavor reveals that 40% of our CPU time\nis spent in <code>codecvt::in<\/code>.  Some debugging reveals\nthat <code>codecvt::in<\/code> ultimately calls\n<code>MultiByteToWideChar<\/code> but uses it to convert\nonly one or two characters at a time, even though we handed it a\nwhole line.\n<\/p>\n<p>\nLet&#8217;s get rid of <code>codecvt::in<\/code> and convert the\ncharacters ourselves, calling\n<code>MultiByteToWideChar<\/code> exactly once to convert the\nentire line at a single go.\n<\/p>\n<pre>\n<font COLOR=\"blue\">#define CP_BIG5 950<\/font>\nDictionary::Dictionary()\n{\n MappedTextFile mtf(TEXT(\"cedict.b5\"));\n <font COLOR=\"blue\"><strike>\/\/ typedef std::codecvt&lt;wchar_t, char, mbstate_t&gt; widecvt;\n \/\/ std::locale l(\".950\");\n \/\/ const widecvt&amp; cvt = _USE(l, widecvt); \/\/ use_facet&lt;widecvt&gt;(l);<\/strike><\/font>\n const CHAR* pchBuf = mtf.Buffer();\n const CHAR* pchEnd = pchBuf + mtf.Length();\n while (pchBuf &lt; pchEnd) {\n  const CHAR* pchEOL = std::find(pchBuf, pchEnd, '\\n');\n  if (*pchBuf != '#') {\n   size_t cchBuf = pchEOL - pchBuf;\n   wchar_t* buf = new wchar_t[cchBuf];\n   <font COLOR=\"blue\">DWORD cchResult = MultiByteToWideChar(CP_BIG5, 0,\n                          pchBuf, cchBuf, buf, cchBuf);\n   if (cchResult) {<\/font>\n    wstring line(buf, <font COLOR=\"blue\">cchResult<\/font>);\n    DictionaryEntry de;\n    if (de.Parse(line)) {\n     v.push_back(de);\n    }\n   }\n   delete[] buf;\n  }\n  pchBuf = pchEOL + 1;\n }\n}\n<\/pre>\n<p>\nInstead of using the <code>codecvt::in<\/code> method to perform\ncharacter conversion, we go straight to the\n<code>MultiByteToWideChar<\/code> function.\nNotice that we assume that the Big5 string will not generate\nmore Unicode characters than its length in bytes.\nThis happens to be a safe assumption based on our external knowledge\nof the Big5 encoding.  (If the encoding were something else,\nthe assumption may no longer be valid.)\n<\/p>\n<p>\nWith this change, the dictionary load time has dropped to 240ms\n(or 300ms if you include the time it takes to destroy the\ndictionary).  That&#8217;s twice as fast the previous version, but still\nnot quite close enough to the 100ms goal.\nWe still have some work ahead of us.\n<\/p>\n<p>\n[Raymond is currently on vacation; this message was pre-recorded.]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Getting rid of getline was a big help, but 480ms is still not quite peppy enough. You need to respond to user actions within a tenth of a second for thing to seem responsive. Profiling the latest endeavor reveals that 40% of our CPU time is spent in codecvt::in. Some debugging reveals that codecvt::in ultimately [&hellip;]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-35633","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"<p>Getting rid of getline was a big help, but 480ms is still not quite peppy enough. You need to respond to user actions within a tenth of a second for thing to seem responsive. Profiling the latest endeavor reveals that 40% of our CPU time is spent in codecvt::in. Some debugging reveals that codecvt::in ultimately [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/35633","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=35633"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/35633\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=35633"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=35633"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=35633"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}