The sad history of Unicode printf-style format specifiers in Visual C++
Windows adopted Unicode before most other operating systems. As a result, Windows’s solutions to many problems differ from solutions adopted by those who waited for the dust to settle.¹ The most notable example of this is that Windows used UCS-2 as the Unicode encoding. This was the encoding recommended by the Unicode Consortium because Unicode 1.0 supported only 65536 characters.² The Unicode Consortium changed their minds five years later, but by then it was far too late for Windows, which had already shipped Win32s, Windows NT 3.1, Windows NT 3.5, Windows NT 3.51, and Windows 95, all of which used UCS-2.³
But today we’re going to talk about
printf-style format strings.
Windows adopted Unicode before the C language did. This meant that Windows had to invent Unicode support in the C runtime. The result was functions like
wprintf. As for
printf-style format strings, here’s what we ended up with:
%sformat specifier represents a string in the same width as the format string.
%Sformat specifier represents a string in the opposite width as the format string.
%hsformat specifier represents a narrow string regardless of the width of the format string.
%lsformat specifiers represent a wide string regardless of the width of the format string.
The idea behind this pattern was so that you could write code like this:
TCHAR buffer; GetSomeString(buffer, 256); _tprintf(TEXT("The string is %s.\n"), buffer);
If the code is compiled as ANSI, the result is
char buffer; GetSomeStringA(buffer, 256); printf("The string is %s.\n", buffer);
And if the code is compiled as Unicode, the result is⁴
wchar_t buffer; GetSomeStringW(buffer, 256); wprintf(L"The string is %s.\n", buffer);
By following the convention that
%s takes a string in the same width as the format string itself, this code runs properly when compiled either as ANSI or as Unicode. It also makes converting existing ANSI code to Unicode much simpler, since you can keep using
%s, and it will morph to do what you need.
When Unicode support formally arrived in C99, the C standard committee chose a different model for
printf format strings.
%hsformat specifiers represent an narrow string.
%lsformat specifier represents a wide string.
This created a problem. There were six years and untold billions of lines of code in the Windows ecosystem that used the old model. What should the Visual C and C++ compiler do?
They chose to stick with the existing nonstandard model, so as not to break every Windows program on the planet.
If you want your code to work both on runtimes that use the Windows classic
printf rules as well as those that use C standard
printf rules, you can limit yourself to
%hs for narrow strings and
%ls for wide strings, and you’ll get consistent results regardless of whether the format string was passed to
#ifdef UNICODE #define TSTRINGWIDTH TEXT("l") #else #define TSTRINGWIDTH TEXT("h") #endif TCHAR buffer; GetSomeString(buffer, 256); _tprintf(TEXT("The string is %") TSTRINGWIDTH TEXT("s\n"), buffer); char buffer; GetSomeStringA(buffer, 256); printf("The string is %hs\n", buffer); wchar_t buffer; GetSomeStringW(buffer, 256); wprintf("The string is %ls\n", buffer);
TSTRINGWIDTH separately lets you do things like
_tprintf(TEXT("The string is %10") TSTRINGWIDTH TEXT("s\n"), buffer);
Since people like tables, here’s a table.
|Format||Windows classic||C standard|
I highlighted the rows where the C standard agrees with the Windows classic format.⁵ If you want your code to work the same under either format convention, you should stick to those rows.
¹ You’d think that adopting Unicode early would give Windows the first-mover advantage, but at least with respect to Unicode, it ended up being a first-mover disadvantage, because everybody else could sit back and wait for better solutions to emerge (such as UTF-8) before beginning their Unicode adoption efforts.
² I guess they thought that 65536 characters should be enough for anyone.
³ This was later upgraded to UTF-16. Fortunately, UTF-16 is backward compatible with UCS-2 for the code points that are representable in both.
⁴ Technically, the Unicode version was
unsigned short buffer; GetSomeStringW(buffer, 256); wprintf(L"The string is %s.\n", buffer);
because there was not yet a
wchar_t as an independent type. Prior to the introduction of
wchar_t to the standard, the
wchar_t type was just a synonym for
unsigned short. The changing fate of the
wchar_t type has its own story.
⁵ The Windows classic format came first, so the question is whether the C standard chose to align with the Windows classic format, rather than vice versa.