It seems that a popular way of converting a string to uppercase or lowercase is to do it letter by letter.<\/p>\n
std::wstring name;\r\n\r\nstd::transform(name.begin(), name.end(), name.begin(),\r\n std::tolower);\r\n<\/pre>\nThis is wrong for many reasons.<\/p>\n
First of all,
std::tolower<\/code> is not an addressible function<\/i>. This means, among other things, that you are not allowed to take the function’s address,\u00b9 like we’re doing here when we pass a pointer to the function to std::transform<\/code>. So we’ll have to use a lambda.<\/p>\n std::wstring name;\r\n\r\nstd::transform(name.begin(), name.end(), name.begin(),\r\n [](auto c) { return std::tolower(c); });\r\n<\/pre>\nThe next mistake is a copy-pasta: The code is using
std::tolower<\/code> to convert wide characters ( wchar_t<\/code>) even thoughstd::tolower<\/code> works only for narrow characters (even more restrictive than that: it works only for unsigned<\/i> narrow characters unsigned char<\/code>). There is no compile-time error becausestd::tolower<\/code> accepts an int<\/code>, and on most systems,wchar_t<\/code> is implicitly promotable toint<\/code>, so the compiler accepts the value without complaint even though over 99% of the potential values are out of range.<\/p>\nEven if we fix the code to use
std::towlower<\/code>:<\/p>\n std::wstring name;\r\n\r\nstd::transform(name.begin(), name.end(), name.begin(),\r\n [](auto c) { return std::towlower(c); });\r\n<\/pre>\nit’s still wrong because it assumes that case mapping can be performed
char<\/code>-by-char<\/code> orwchar_t<\/code>-by-wchar_t<\/code> in a context-free manner.<\/p>\nIf the
wchar_t<\/code> encoding is UTF-16, then characters outside the basic multilingual plane (BMP) are represented by pairs ofwchar_t<\/code> values. For example, the Unicode character OLD HUNGARIAN CAPITAL LETTER A\u00b2 (U+10C80) is represented by two UTF-16 code units: D803<\/kbd> followed by DC80<\/kbd>.<\/p>\nPassing these two code units to
towlower<\/code> one at a time preventstowlower<\/code> from understanding how they interact with each other. If you calltowlower<\/code> with DC80<\/kbd>, it recognizes that you passed only half of a character, but it doesn’t know what the other half is, so it has to just shrug its shoulders and say, “Um, DC80<\/kbd>?” Too bad, because the lowercase version of OLD HUNGARIAN CAPITAL LETTER A (U+10C80) is OLD HUNGARIAN SMALL LETTER A (U+10CC0), so it should have returned DCC0<\/kbd>. Of coursetowlower<\/code> doesn’t have psychic powers, so you can’t really expect it to have known that the DC80<\/kbd> was the partner of an unseen D803<\/kbd>.<\/p>\nAnother problem (which applies even if
wchar_t<\/code> is UTF-32) is that the uppercase and lowercase versions of a character might have different lengths. For example, LATIN SMALL LETTER SHARP S (“\u00df” U+00DF) uppercases to the two-character sequence “SS”:\u00b3 Stra\u00dfe \u21d2 STRASSE, and LATIN SMALL LIGATURE FL (“\ufb02” U+FB02) uppercases to the two-character sequence “FL”. In both examples, converting the string to uppercase causes the string to get longer. And in certain forms of the French language, capitalizing an accented character causes the accent to be dropped: \u00e0 Paris \u21d2 A PARIS. If the accented character \u00e0 were encoded as LATIN SMALL LETTER A (U+0061) followed by COMBINING GRAVE ACCENT (U+0300), then converting to uppercase causes the string to get shorter.<\/p>\nSimilar issues apply to the
std::string<\/code> version:<\/p>\nstd::string name;\r\n\r\nstd::transform(name.begin(), name.end(), name.begin(),\r\n [](auto c) { return std::tolower(c); });\r\n<\/pre>\nIf the string potentially contains characters outside the 7-bit ASCII range, then this triggers undefined behavior when those characters are encountered. And for UTF-8 data, you have the same issues discussed before: Multibyte characters will not be converted properly, and it breaks for case mappings that alter string lengths.<\/p>\n
Okay, so those are the problems. What’s the solution?<\/p>\n