{"id":8183,"date":"2012-03-02T07:00:00","date_gmt":"2012-03-02T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2012\/03\/02\/this-isnt-highlights-magazine-sort-keys-and-why-they-change\/"},"modified":"2012-03-02T07:00:00","modified_gmt":"2012-03-02T07:00:00","slug":"this-isnt-highlights-magazine-sort-keys-and-why-they-change","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20120302-00\/?p=8183\/","title":{"rendered":"This isn't Highlights magazine: Sort keys and why they change"},"content":{"rendered":"

\nSome time ago,\n\nRy Jones<\/a>\ngave some examples of\n\nQuotable Raymond<\/a>,\nincluding the following:\n<\/p>\n

\n

\nHow to make a good doc bug report:\n<\/p>\n

\n1. Don’t embed pictures. … This isn’t Highlights magazine.\n<\/p>\n<\/blockquote>\n

\nWhat Ry didn’t realize is that his “…” totally misrepresented\nthe message.\nThere were actually two separate items,\nbut he combined them into one and replaced the missing parts\nwith “…”.\nI already ranted some time ago about\n\nembedding pictures<\/a>.\nToday I’ll rant about the second item, which is turning a bug\nreport into the\n\nSpot the difference between these two pictures<\/i> game<\/a>\nin\n\nHighlights<\/i><\/a> magazine.\n<\/p>\n

\nA customer reported that the LCMap­String<\/code>\nfunction was returning incorrect results on Windows 7.\nTo illustrate, they included a sample program with\nthe Windows XP and Windows 7 output.\n<\/p>\n

\n
\n#include <windows.h>\nchar bad[] = \"... a very long string...<\/i>\";\nchar dest[10000];\nint main()\n{\n    int nChr, i;\n    nChr = LCMapStringA(1033,  \/* English (US) *\/\n                  LCMAP_SORTKEY,\n                  bad,\n                  -1,    \/* source is NULL terminated *\/\n                  dest,  \/* destination *\/\n                  sizeof(dest));  \/* dest size *\/\n    \/* print out returned number of bytes as hexadecimal *\/\n    for (i = 0; i < nChr; i++)\n        printf(\"%02x \", (unsigned char)dest[i]);\n    printf(\"\\n\");\n}\n<\/pre>\n
\nOn Windows XP, we get the correct sort key:\n<\/p>\n
\n\n0e 48 0e 7c 0e 8a 0e 21 0e 51 07 02 0e 32 0e 7e 0e 91 0e\n9f 0e 51 07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 07 02 0e 91\n0e 32 0e 99 07 02 0e 02 0e 51 0e 21 0e 99 07 2f 07 02 0e\n0a 0e 7c 0e 70 0e 91 0e 21 0e 0a 0e 99 0e 21 0e 99 0e 9f\n0e 8a 07 02 0e 02 0e 1a 0e 32 0e 7e 0e 32 0e 91 0e 32 0e\n0a 0e 32 0e 70 0e 25 07 02 0e 21 0e 48 0e 32 0e 99 07 2f\n07 02 0e 91 0e 21 0e 1a 07 02 0e 1a 0e 7c 07 02 0e 21 0e\n32 0e 9f 0e 91 0e 51 0e 7c 0e 1a 07 02 0e 99 0e 21 0e 51\n0e 7e 0e 7c 0e 8a 07 02 0e 32 0e 70 0e 0a 0e 32 0e 1a 0e\n32 0e 1a 0e 9f 0e 70 0e 99 07 02 0e 9f 0e 99 07 02 0e 48\n0e 02 0e 09 0e 7c 0e 8a 0e 21 07 02 0e 21 0e 99 07 02 0e\n1a 0e 7c 0e 48 0e 7c 0e 8a 0e 21 07 02 0e 51 0e 02 0e 25\n0e 70 0e 02 07 02 0e 02 0e 48 0e 32 0e 89 0e 9f 0e 02 07\n33 07 02 08 14 07 02 0e 9f 0e 99 07 02 0e 21 0e 70 0e 32\n0e 51 07 02 0e 02 0e 1a 07 02 0e 51 0e 32 0e 70 0e 32 0e\n51 07 02 0e a2 0e 21 0e 70 0e 32 0e 02 0e 51 07 2f 07 02\n0e 89 0e 9f 0e 32 0e 91 07 02 0e 70 0e 7c 0e 91 0e 99 0e\n8a 0e 9f 0e 1a 07 02 0e 21 0e a6 0e 21 0e 8a 0e 0a 0e 32\n0e 99 0e 02 0e 99 0e 32 0e 7c 0e 70 07 02 0e 9f 0e 48 0e\n48 0e 02 0e 51 0e 0a 0e 7c 07 02 0e 48 0e 02 0e 09 0e 7c\n0e 8a 0e 32 0e 91 07 02 0e 70 0e 32 0e 91 0e 32 07 02 0e\n9f 0e 99 07 02 0e 02 0e 48 0e 32 0e 89 0e 9f 0e 32 0e 7e\n07 02 0e 21 0e a6 07 02 0e 21 0e 02 07 02 0e 0a 0e 7c 0e\n51 0e 51 0e 7c 0e 1a 0e 7c 07 02 0e 0a 0e 7c 0e 70 0e 91\n0e 21 0e 89 0e 9f 0e 02 0e 99 07 33 07 02 0e 1a 0e 9f 0e\n32 0e 91 07 02 0e 02 0e 9f 0e 99 0e 21 07 02 0e 32 0e 8a\n0e 9f 0e 8a 0e 21 07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 07\n02 0e 32 0e 70 07 02 0e 8a 0e 21 0e 7e 0e 8a 0e 21 0e 2c\n0e 21 0e 70 0e 1a 0e 21 0e 8a 0e 32 0e 99 07 02 0e 32 0e\n70 07 02 0e a2 0e 7c 0e 48 0e 9f 0e 7e 0e 99 0e 02 0e 99\n0e 21 07 02 0e a2 0e 21 0e 48 0e 32 0e 99 07 02 0e 21 0e\n91 0e 91 0e 21 07 02 0e 0a 0e 32 0e 48 0e 48 0e 9f 0e 51\n07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 0e 21 07 02 0e 21 0e\n9f 07 02 0e 23 0e 9f 0e 25 0e 32 0e 02 0e 99 07 02 0e 70\n0e 9f 0e 48 0e 48 0e 02 07 02 0e 7e 0e 02 0e 8a 0e 32 0e\n02 0e 99 0e 9f 0e 8a 07 33 07 02 0e 21 0e a6 0e 0a 0e 21\n0e 7e 0e 99 0e 21 0e 9f 0e 8a 07 02 0e 91 0e 32 0e 70 0e\n99 07 02 0e 7c 0e 0a 0e 0a 0e 02 0e 21 0e 0a 0e 02 0e 99\n07 02 0e 0a 0e 9f 0e 7e 0e 32 0e 1a 0e 02 0e 99 0e 02 0e\n99 07 02 0e 70 0e 7c 0e 70 07 02 0e 7e 0e 8a 0e 7c 0e 32\n0e 1a 0e 21 0e 70 0e 99 07 2f 07 02 0e 91 0e 9f 0e 70 0e\n99 07 02 0e 32 0e 70 07 02 0e 0a 0e 9f 0e 48 0e 7e 0e 02\n07 02 0e 89 0e 9f 0e 32 07 02 0e 7c 0e 23 0e 23 0e 32 0e\n0a 0e 32 0e 02 07 02 0e 1a 0e 21 0e 91 0e 21 0e 8a 0e 9f\n0e 70 0e 99 07 02 0e 51 0e 7c 0e 48 0e 48 0e 32 0e 99 07\n02 0e 02 0e 70 0e 32 0e 51 07 02 0e 32 0e 1a 07 02 0e 21\n0e 91 0e 99 07 02 0e 48 0e 02 0e 09 0e 7c 0e 8a 0e 9f 0e\n51 07 33 01 01 12 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 12\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 12 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 12 01 01 81 fb 06 82 00\n<\/tt>\n<\/p>\n
\nbut on Windows 7 the output is wrong:\n<\/p>\n
\n\n0e 48 0e 7c 0e 8a 0e 21 0e 51 07 02 0e 32 0e 7e 0e 91 0e\n9f 0e 51 07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 07 02 0e 91\n0e 32 0e 99 07 02 0e 02 0e 51 0e 21 0e 99 07 2f 07 02 0e\n0a 0e 7c 0e 70 0e 91 0e 21 0e 0a 0e 99 0e 21 0e 99 0e 9f\n0e 8a 07 02 0e 02 0e 1a 0e 32 0e 7e 0e 32 0e 91 0e 32 0e\n0a 0e 32 0e 70 0e 25 07 02 0e 21 0e 48 0e 32 0e 99 07 2f\n07 02 0e 91 0e 21 0e 1a 07 02 0e 1a 0e 7c 07 02 0e 21 0e\n32 0e 9f 0e 91 0e 51 0e 7c 0e 1a 07 02 0e 99 0e 21 0e 51\n0e 7e 0e 7c 0e 8a 07 02 0e 32 0e 70 0e 0a 0e 32 0e 1a 0e\n32 0e 1a 0e 9f 0e 70 0e 99 07 02 0e 9f 0e 99 07 02 0e 48\n0e 02 0e 09 0e 7c 0e 8a 0e 21 07 02 0e 21 0e 99 07 02 0e\n1a 0e 7c 0e 48 0e 7c 0e 8a 0e 21 07 02 0e 51 0e 02 0e 25\n0e 70 0e 02 07 02 0e 02 0e 48 0e 32 0e 89 0e 9f 0e 02 07\n33 07 02 08 14 07 02 0e 9f 0e 99 07 02 0e 21 0e 70 0e 32\n0e 51 07 02 0e 02 0e 1a 07 02 0e 51 0e 32 0e 70 0e 32 0e\n51 07 02 0e a2 0e 21 0e 70 0e 32 0e 02 0e 51 07 2f 07 02\n0e 89 0e 9f 0e 32 0e 91 07 02 0e 70 0e 7c 0e 91 0e 99 0e\n8a 0e 9f 0e 1a 07 02 0e 21 0e a6 0e 21 0e 8a 0e 0a 0e 32\n0e 99 0e 02 0e 99 0e 32 0e 7c 0e 70 07 02 0e 9f 0e 48 0e\n48 0e 02 0e 51 0e 0a 0e 7c 07 02 0e 48 0e 02 0e 09 0e 7c\n0e 8a 0e 32 0e 91 07 02 0e 70 0e 32 0e 91 0e 32 07 02 0e\n9f 0e 99 07 02 0e 02 0e 48 0e 32 0e 89 0e 9f 0e 32 0e 7e\n07 02 0e 21 0e a6 07 02 0e 21 0e 02 07 02 0e 0a 0e 7c 0e\n51 0e 51 0e 7c 0e 1a 0e 7c 07 02 0e 0a 0e 7c 0e 70 0e 91\n0e 21 0e 89 0e 9f 0e 02 0e 99 07 33 07 02 0e 1a 0e 9f 0e\n32 0e 91 07 02 0e 02 0e 9f 0e 99 0e 21 07 02 0e 32 0e 8a\n0e 9f 0e 8a 0e 21 07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 07\n02 0e 32 0e 70 07 02 0e 8a 0e 21 0e 7e 0e 8a 0e 21 0e 2c\n0e 21 0e 70 0e 1a 0e 21 0e 8a 0e 32 0e 99 07 02 0e 32 0e\n70 07 02 0e a2 0e 7c 0e 48 0e 9f 0e 7e 0e 99 0e 02 0e 99\n0e 21 07 02 0e a2 0e 21 0e 48 0e 32 0e 99 07 02 0e 21 0e\n91 0e 91 0e 21 07 02 0e 0a 0e 32 0e 48 0e 48 0e 9f 0e 51\n07 02 0e 1a 0e 7c 0e 48 0e 7c 0e 8a 0e 21 07 02 0e 21 0e\n9f 07 02 0e 23 0e 9f 0e 25 0e 32 0e 02 0e 99 07 02 0e 70\n0e 9f 0e 48 0e 48 0e 02 07 02 0e 7e 0e 02 0e 8a 0e 32 0e\n02 0e 99 0e 9f 0e 8a 07 33 07 02 0e 21 0e a6 0e 0a 0e 21\n0e 7e 0e 99 0e 21 0e 9f 0e 8a 07 02 0e 91 0e 32 0e 70 0e\n99 07 02 0e 7c 0e 0a 0e 0a 0e 02 0e 21 0e 0a 0e 02 0e 99\n07 02 0e 0a 0e 9f 0e 7e 0e 32 0e 1a 0e 02 0e 99 0e 02 0e\n99 07 02 0e 70 0e 7c 0e 70 07 02 0e 7e 0e 8a 0e 7c 0e 32\n0e 1a 0e 21 0e 70 0e 99 07 2f 07 02 0e 91 0e 9f 0e 70 0e\n99 07 02 0e 32 0e 70 07 02 0e 0a 0e 9f 0e 48 0e 7e 0e 02\n07 02 0e 89 0e 9f 0e 32 07 02 0e 7c 0e 23 0e 23 0e 32 0e\n0a 0e 32 0e 02 07 02 0e 1a 0e 21 0e 91 0e 21 0e 8a 0e 9f\n0e 70 0e 99 07 02 0e 51 0e 7c 0e 48 0e 48 0e 32 0e 99 07\n02 0e 02 0e 70 0e 32 0e 51 07 02 0e 32 0e 1a 07 02 0e 21\n0e 91 0e 99 07 02 0e 48 0e 02 0e 09 0e 7c 0e 8a 0e 9f 0e\n51 07 33 01 01 12 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 12\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 12 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02 02\n02 12 01 01 ff 82 82 12 00\n<\/tt>\n<\/p>\n<\/blockquote>\n
\nThis is what I meant by the\nSpot the difference between these two pictures<\/i> game.\nWho wants to sit there and compare two 900-byte sequences\nbyte-by-byte?\n(On top of that,\nthe customer liaison forwarded the question to an internal\npeer-to-peer discussion group.\n\nPeer-to-peer discussion groups do not come with a service level agreement<\/a>,\nand if you ask a question which makes heavy demands on the reader\njust to understand the question<\/i>,\ndon’t be surprised if nobody answers.)\n<\/p>\n
\nIf you want an explanation of why two things are different,\nyou need to point out where the difference is.\nThis often happens with screen shots,\nwhere a customer includes two different screen shots and\nasks for an explanation of why they are different,\nwithout indicating where the difference is<\/i>.\nAre they complaining that the font is different?\nThat ClearType is enabled in one screen shot but not the other?\nThe hard drive in the first screen shot has more free\nspace than the one in the second screen shot?\n<\/p>\n
\nAnyway, back to the sort keys.\n(As a refresher, you can read\n\nMichael Kaplan’s explanation of sort keys<\/a>.)\nThe documentation for LCMap­String<\/code>\nsays\n<\/p>\n
\n
\ndwMapFlags<\/i> [in]<\/i>\n<\/p>\n
\nFlags specifying the type of transformation to use during string mapping\nor the type of sort key to generate.\nFor detailed definitions, see the dwMapFlags<\/i>\nparameter of LCMapStringEx<\/b>.\n<\/p>\n<\/blockquote>\n
\nIf you follow the instructions and consult the\ndwMapFlags<\/code> parameter of the\nLCMapStringEx<\/code> function, you find\n<\/p>\n
\n\n\n
LCMAP_SORTKEY<\/td>\nProduce a normalized sort key.\n        If the LCMAP_SORTKEY flag is not specified,\n        the function performs string mapping.\n        For details of sort key generation and string mapping,\n        see the Remarks section.<\/td>\n<\/tr>\n<\/table>\n<\/blockquote>\n
\nIf you follow the instructions and go to the Remarks section\nlooking for LCMAP_SORTKEY<\/code>,\nyou find\n<\/p>\n
\nThe application can use LCMapString<\/b>\nor LCMapStringEx<\/b>\nto generate a sort key.\nTo do this, the application specifies\nLCMAP_SORTKEY for the dwMapFlags<\/i> parameter.\nFor more information, see\n\nHandling Sorting in Your Applications<\/a>.\n<\/p><\/blockquote>\n
\n(A link to Handling Sorting in Your Applications<\/i> also\nexists on the LCMapString<\/code> page, but let’s say you\ndidn’t see that.)\n<\/p>\n
\nIf you follow the instructions and read the\nHandling Sorting in Your Applications<\/i> page,\n<\/p>\n
\n
\nUse Sort Versioning<\/b>\n<\/p>\n
\nA sorting table has two numbers that identify its version:\nthe defined version and the NLS version.\nBoth numbers are DWORD values,\ncomposed of a major value and a minor value.\n…\n<\/p>\n
\nNote<\/b>\nFor a major version,\none or more code points are changed so that the application\nmust re-index all data for comparisons to be valid.\nFor a minor version,\nnothing is moved but code points are added.\nFor this type of version,\nthe application only has to re-index strings\nwith previously unsortable values.\n<\/p>\n<\/blockquote>\n
\nAnd there you have your answer.\nSort keys generated by different major versions are not compatible.\nI don’t know what NLS version for English was included\nwith Windows XP, though I did check that on Windows Vista SP2\nit is major version 0x0405 (minor version 0)\nand on Windows 7 it is major version 0x601 (minor version 1).\nSo whatever it was on Windows XP, it was presumably less than or\nequal to 0x0405,\nwhich means that it’s definitely different from the major version\non Windows 7.\n<\/p>\n
\nAnd like the documentation says, when the major version changes,\nyou need to regenerate your sort keys\nbecause sort keys from different major versions are not compatible.\n<\/p>\n
\nMichael Kaplan discussed\n\nother issues arising from changing the major version<\/a>\nas well as various\n\nchanges to the sort keys over time<\/a>.\n<\/p>\n
\nIf you find yourself using\na program that relies on sort keys remaining stable\neven across major versions,\nyou can try applying\n\nthe EmulateSorting compatibility fix<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Some time ago, Ry Jones gave some examples of Quotable Raymond, including the following: How to make a good doc bug report: 1. Don’t embed pictures. … This isn’t Highlights magazine. What Ry didn’t realize is that his “…” totally misrepresented the message. There were actually two separate items, but he combined them into one […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-8183","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Some time ago, Ry Jones gave some examples of Quotable Raymond, including the following: How to make a good doc bug report: 1. Don’t embed pictures. … This isn’t Highlights magazine. What Ry didn’t realize is that his “…” totally misrepresented the message. There were actually two separate items, but he combined them into one […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/8183","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=8183"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/8183\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=8183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=8183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=8183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}