\nThe first thing we’ll need to do in our little dictionary program\nis to load the dictionary into memory. The format of the dictionary\nfile is as a plain text file, each line of which is of the form\n<\/p>\n
\nChinese [pinyin] \/English 1\/English 2\/...\/\n舉例 [ju3 li4] \/to give an example\/\n<\/pre>\n\nSince it was the Big5 dictionary we downloaded,\nthe Chinese characters are in Big5 format,\nknown to Windows as code page 950.\nOur program will be Unicode, so we’ll have to convert it as we load\nthe dictionary. Yes, I could’ve used the Unicode version of the\ndictionary, but it so happens that when I set out to write this program,\n\nthere was no Unicode version available<\/a>.\nFortunately, this oversight opened up the opportunity to illustrate\nsome other programming decisions and techniques.\n<\/p>\n
\nThe first stage in our series of exercises will be loading the dictionary\ninto memory.\n<\/p>\n
\n#define UNICODE\n#define _UNICODE<\/a>\n#include <windows.h>\n#include <string>\n#include <fstream>\n#include <iostream> \/\/ for cin\/cout\n#include <vector>\nusing std::string;\nusing std::wstring;\nusing std::vector;\nstruct DictionaryEntry\n{\n bool Parse(const wstring& line);\n wstring trad;\n wstring simp;\n wstring pinyin;\n wstring english;\n};\nbool DictionaryEntry::Parse(const wstring& line)\n{\n wstring::size_type start = 0;\n wstring::size_type end = line.find(L' ', start);\n if (end == wstring::npos) return false;\n trad.assign(line, start, end);\n start = line.find(L'[', end);\n if (start == wstring::npos) return false;\n end = line.find(L']', ++start);\n if (end == wstring::npos) return false;\n pinyin.assign(line, start, end - start);\n start = line.find(L'\/', end);\n if (start == wstring::npos) return false;\n start++;\n end = line.rfind(L'\/');\n if (end == wstring::npos) return false;\n if (end <= start) return false;\n english.assign(line, start, end-start);\n return true;\n}\nclass Dictionary\n{\npublic:\n Dictionary();\n int Length() { return v.size(); }\n const DictionaryEntry& Item(int i) { return v[i]; }\nprivate:\n vector<DictionaryEntry> v;\n};\nDictionary::Dictionary()\n{\n std::wifstream src;\n src.imbue(std::locale(\".950\"));\n src.open(\"cedict.b5\");\n wstring s;\n while (getline(src, s)) {\n if (s.length() > 0 && s[0] != L'#') {\n DictionaryEntry de;\n if (de.Parse(s)) {\n v.push_back(de);\n }\n }\n }\n}\nint __cdecl main(int argc, const char* argv[])\n{\n DWORD dw = GetTickCount();\n {\n Dictionary dict;\n std::cout << dict.Length() << std::endl;\n std::cout << GetTickCount() - dw << std::endl;\n }\n std::cout << GetTickCount() - dw << std::endl;\n return 0;\n}\n<\/pre>\n\nOur dictionary is just a list of words with their English definitions.\nThe Chinese words are written in three forms\n(traditional Chinese,\nsimplified Chinese, and\nPinyin romanization).\nFor those who are curious, there are two writing systems\nfor the Mandarin Chinese language and two phonetic systems.\nWhich one a particular Mandarin-speaking population follows depends\non whether they fell under the influence of China’s\nlanguage reform of 1956.\nTraditional Chinese characters and the Bopomo\nphonetic system\n(also called Bopomofo)\nare used on Taiwan; simplified Chinese characters\nand the Pinyin system are used in China.\nConverting Pinyin to Bopomo isn’t interesting,\nso I’ve removed that part from the program I’m presenting here.\n<\/p>\n
(The schism in the spelling of the English language follows a similar\npattern.\nUnder the leadership of Noah Webster,\nthe United States underwent its own spelling reform,\nbut countries which were under the influence of the British crown\nretained the traditional spellings.\n\nSpelling reform continues in other languages even today<\/a>,\nand the subject is almost always highly contentious,\nwith traditionalists and reformists pitted against each other\nin a battle over a language’s—and by proxy,\na culture’s—identity.)<\/p>\n
\nThe program itself is fairly straightforward.\nIt creates a Unicode file stream
wifstream<\/code>\nand “imbues” it with code page 950 (Big5).\nThis instructs the runtime to interpret the bytes of the file\ninterpreted in the specified code page.\nWe read strings out of the file, ignore the comments,\nand parse the rest, appending them to ourvector<\/code>\nof dictionary entries.\n<\/p>\n\nParsing the line consists of finding the spaces, brackets,\nand slashes, and splitting the line into the traditional Chinese,\nPinyin, and English components. (We’ll deal with simplified\nChinese later.)\n<\/p>\n
\nWhen I run this program on my machine, the dictionary loads in 2080ms\n(or 2140ms if you include the time to run the destructor).\nThis is an unacceptably long startup time, so the first order\nof business is to make startup faster. That will be the focus of\nthis stage.\n<\/p>\n
\nNotice that as a sanity check, I print the total number of words in the\ndictionary. The number should match the number of lines in the\n
cedict.b5<\/code> file (minus the one comment line).\nIf not, then I know that something went wrong.\nThis is an important sanity check<\/strong>:\nYou might make a performance optimization that looks great\nwhen you run it past a stopwatch,\nonly to discover that your “optimization” actually introduced a\nbug. For example, one of my attempted optimizations of this program\nresulted in a phenomenal tenfold speedup,\nbut only because of a bug that caused it to think it was finished\nwhen it had in reality processed only 10% of the dictionary!\n<\/p>\n