{"id":20045,"date":"2018-07-06T14:30:26","date_gmt":"2018-07-06T21:30:26","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vcblog\/?p=20045"},"modified":"2019-04-18T09:31:03","modified_gmt":"2019-04-18T09:31:03","slug":"msvc-preprocessor-progress-towards-conformance","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/msvc-preprocessor-progress-towards-conformance\/","title":{"rendered":"MSVC Preprocessor Progress towards Conformance"},"content":{"rendered":"

Why re-write the preprocessor?\n<\/span><\/strong><\/span><\/p>\n

Recently, we published a blog post on C++ conformance completion.<\/a> As mentioned in the blog post, the preprocessor in MSVC is currently getting an overhaul. We are doing this to improve its language conformance, address some of the longstanding bugs that were difficult to fix due to its design and improve its usability and diagnostics. In addition to that, there are places in the standard where the preprocessor behavior is undefined or unspecified and our traditional behavior diverged from other major compilers. In some of those cases, we want to move closer to the ecosystem to make it easier for cross platform libraries to come to MSVC.\n<\/span><\/p>\n

If there is an old library you use or maintain that depends on the non-conformant traditional behavior of the MSVC preprocessor, you don’t need to worry about these breaking changes as we will still support this behavior. The updated preprocessor is currently under the switch \/experimental:preprocessor until it is fully implemented and production-ready, at which point it will be moved to a \/Zc: switch which is enabled by default under \/permissive- mode.\n<\/span><\/p>\n

How do I use it?\n<\/span><\/strong><\/span><\/p>\n

The behavior of the traditional preprocessor is maintained and continues to be the default behavior of the compiler. The conformant preprocessor can be enabled by using the \/experimental:preprocessor switch on the command line starting with the Visual Studio 2017 15.8 Preview 3 release<\/a>.<\/p>\n

We have introduced a new predefined macro in the compiler called “_MSVC_TRADITIONAL<\/em>” to indicate the traditional preprocessor is being used. This macro is set unconditionally, independent of which preprocessor is invoked. Its value is “1” for the traditional preprocessor, and “0” for the conformant experimental preprocessor.\n<\/span><\/p>\n

#if <\/span>defined(_MSVC_TRADITIONAL) && _MSVC_TRADITIONAL\n<\/span><\/span><\/span><\/span>\/\/ Logic using the traditional preprocessor\n<\/span><\/span>#else\n<\/span><\/span>\/\/ Logic using cross-platform compatible preprocessor\n<\/span><\/span>#endif<\/span>\n<\/span><\/p>\n

\"\"\n<\/span><\/p>\n

What are the behavior changes?\n<\/span><\/strong><\/span><\/p>\n

This first experimental release is focused on getting conformant macro expansions done to maximize adoption of new libraries by the MSVC compiler. Below is a list of some of the more common breaking changes that were run into when testing the updated preprocessor with real world projects.<\/p>\n

Behavior 1 [macro comments]\n<\/span><\/strong><\/span><\/p>\n

The traditional preprocessor is based on character buffers rather than preprocessor tokens. This allows unusual behavior such as this preprocessor comment trick which will not work under the conforming preprocessor:<\/p>\n

#if DISAPPEAR\n<\/span><\/span>#define DISAPPEARING_TYPE \/##\/\n<\/span><\/span>#else\n<\/span><\/span>#define <\/span>DISAPPEARING_TYPE <\/span>int\n<\/span><\/span><\/span><\/span><\/span><\/span>#endif\n<\/span><\/span><\/p>\n

\/\/ myVal disappears when DISAPPEARING_TYPE is turned into a comment\n<\/span><\/span>\/\/ To make standard compliant wrap the following line with the appropriate \/\/ \/\/ #if\/#endif\n<\/span><\/span>DISAPPEARING_TYPE myVal;\n<\/span><\/span><\/p>\n

Behavior 2 [L#val]\n<\/span><\/strong><\/span><\/p>\n

The traditional preprocessor incorrectly combines a string prefix to the result of the # operator:\n<\/span><\/p>\n

\u00a0<\/span>#define <\/span>DEBUG_INFO(val) L”debug prefix:” L#val\n<\/span><\/span><\/span><\/span><\/span><\/span>\/\/ \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ^\n<\/span><\/span>\/\/ \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 this prefix\n<\/span><\/span><\/p>\n

const <\/span>wchar_t *info = DEBUG_INFO(hello world);\n<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

In this case the L<\/span> prefix is unnecessary because the adjacent string literals get combined after macro expansion anyway. The backward compatible fix is to change the definition to:\n<\/span><\/p>\n

#define DEBUG_INFO(val) L”debug prefix:” #val\n<\/span><\/span><\/span><\/span><\/span><\/span>\/\/ \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ^\n<\/span><\/span>\/\/ \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 no prefix<\/span>\n<\/span><\/p>\n

This issue is also found in convenience macros that ‘stringize’ the argument to a wide string literal:<\/p>\n

\u00a0<\/span>\/\/ The traditional preprocessor creates a single wide string literal token\n<\/span>#define <\/span><\/span>STRING(str) L#str<\/span><\/span><\/span><\/span><\/p>\n

\/\/ Potential fixes:\n<\/span><\/span>\/\/ Use string concatenation of L”” and #str to add prefix\n<\/span><\/span>\/\/ This works because adjacent string literals are combined after macro expansion\n<\/span><\/span>#define <\/span>STRING1(str) L””#str<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

\/\/ Add the prefix after #str is stringized with additional macro expansion\n<\/span><\/span>#define <\/span>WIDE(str) L##str\n<\/span><\/span><\/span><\/span>#define <\/span>STRING2(str) WIDE(#str)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

\/\/ Use concatenation operator ## to combine the tokens.\n<\/span><\/span>\/\/ The order of operations for ## and # is unspecified, although all compilers\n<\/span><\/span>\/\/ I checked perform the # operator before ## in this case.\n<\/span><\/span>#define <\/span>STRING3(str) L## #str<\/span><\/span><\/span><\/span>\n<\/span><\/span><\/p>\n

Behavior 3 [warning on invalid ##]\n<\/span><\/strong><\/span><\/p>\n

When the ##<\/span> operator does not result in a single valid preprocessing token, the behavior is undefined. The traditional preprocessor will silently fail to combine the tokens. The new preprocessor will match the behavior of most other compilers and emit a diagnostic.<\/p>\n

\/\/ The ## is unnecessary and does not result in a single preprocessing token.\n<\/span><\/span>#define <\/span>ADD_STD(x) std::##x<\/span><\/span><\/span><\/span><\/p>\n

\/\/ Declare a std::string\n<\/span><\/span>ADD_STD(string) s;<\/span><\/span>\n<\/span><\/span><\/p>\n

Behavior 4 [comma elision in variadic macros]\n<\/span><\/strong><\/span><\/p>\n

Consider the following example:\n<\/span><\/p>\n

void func(int, int = 2, int = 3);\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\/\/ This macro replacement list has a comma followed by __VA_ARGS__\n<\/span><\/span>#define <\/span>FUNC(a, …) func(a, __VA_ARGS__)\n<\/span><\/span><\/span><\/span>int main()\n<\/span><\/span>{\n<\/span>\u00a0\u00a0\u00a0\u00a0\/\/ The following macro is replaced with:\n<\/span><\/span><\/span>\u00a0\u00a0\u00a0\u00a0\/\/ func(10,20,30)\n<\/span><\/span><\/span>\u00a0\u00a0\u00a0\u00a0FUNC(10, 20, 30);<\/span><\/span><\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\/\/ A conforming preprocessor will\u00a0\u00a0\u00a0\u00a0 replace the following macro with: \u00a0\u00a0\u00a0\u00a0func(1, );\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u00a0\u00a0\u00a0\u00a0\/\/ Which will result in a syntax error. \n<\/span><\/span><\/span>\u00a0\u00a0\u00a0\u00a0FUNC(1, );\n<\/span><\/span><\/span>}<\/span>\n<\/span><\/p>\n

All major compilers have a preprocessor extension that helps address this issue. The traditional MSVC preprocessor always removes commas before empty __VA_ARGS__<\/span> replacements. In the updated preprocessor we have decided to more closely follow the behavior of other popular cross platform compilers. For the comma to be removed, the variadic argument must be missing (not just empty) and it must be marked with a ##\n<\/span><\/span>operator.<\/span><\/p>\n

#define <\/span>FUNC2(a, …) func(a , ## __VA_ARGS__)\n<\/span><\/span><\/span><\/span>int main()\n<\/span><\/span>{\n<\/span>\u00a0\u00a0\u00a0\u00a0\/\/ The variadic argument is missing in the macro being evoked\n<\/span><\/span>\/\/ Comma will be removed and replaced with:\n<\/span>\/\/ func(1)\n<\/span><\/span>FUNC2(1);<\/span><\/span><\/span><\/p>\n

\u00a0 \u00a0 \/\/ The variadic argument is empty, but not missing (notice the\n<\/span><\/span><\/span>\/\/ comma in the argument list). The comma will not be removed\n<\/span><\/span>\/\/ when the macro is replaced.\n<\/span><\/span><\/span>\/\/ func(1, )\n<\/span><\/span><\/span>FUNC2(1, );\n<\/span><\/span><\/span>}<\/span>\n<\/span><\/p>\n

In the upcoming C++2a standard this issue has been addressed by adding __VA_OPT__<\/span>, which is not yet implemented.\n<\/span><\/p>\n

Behavior 5 [macro arguments are ‘unpacked’]\n<\/span><\/strong><\/span><\/p>\n

In the traditional preprocessor, if a macro forwards one of its arguments to another dependent macro then the argument does not get “unpacked” when it is substituted. Usually this optimization goes unnoticed, but it can lead to unusual behavior:<\/p>\n

\/\/ Create a string out of the first argument, and the rest of the arguments.\n<\/span><\/span>#define <\/span>TWO_STRINGS( first, … ) #first, #__VA_ARGS__\n<\/span><\/span><\/span><\/span>#define <\/span>A( … ) TWO_STRINGS(__VA_ARGS__)<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

const <\/span>char* c[2] = { A(1, 2) };\n<\/span><\/span><\/span><\/span><\/span><\/span>\/\/ Conformant preprocessor results:\n<\/span><\/span>\/\/ const char c[2] = { “1”, “2” };\n<\/span><\/span>\/\/ Traditional preprocessor results, all arguments are in the first string:\n<\/span><\/span>\/\/ const char c[2] = { “1, 2”, };<\/span>\n<\/span><\/p>\n

When expanding A(), the traditional preprocessor forwards all of the arguments packaged in __VA_ARGS__<\/span> to the first argument of TWO_STRINGS<\/span>, which leaves the variadic argument of TWO_STRINGS<\/span> empty. This causes the result of #first to be “1, 2” rather than just “1”. If you are following along closely, then you may be wondering what happened to the result of #__VA_ARGS__<\/span> in the traditional preprocessor expansion: if the variadic parameter is empty it should result in an empty string literal “”. Due to a separate issue, the empty string literal token was not generated.\n<\/span><\/p>\n

Behavior 6 [rescanning replacement list for macros]\n<\/span><\/strong><\/span><\/p>\n

After a macro is replaced, the resulting tokens are rescanned for additional macro identifiers that need to be replaced. The algorithm used by the traditional preprocessor for doing the rescan is not conformant as shown in this example based on actual code:<\/p>\n

#define CAT(a,b) a ## b\n<\/span><\/span><\/span><\/span>#define <\/span>ECHO(…) __VA_ARGS__<\/span><\/span><\/span><\/span><\/p>\n

\/\/ IMPL1 and IMPL2 are implementation details\n<\/span><\/span>#define <\/span>IMPL1(prefix,value) do_thing_one( prefix, value)\n<\/span><\/span><\/span><\/span>#define IMPL2(prefix,value) do_thing_two( prefix, value)\n<\/span><\/span><\/span><\/span>\/\/ MACRO chooses the expansion behavior based on the value passed to macro_switch\n<\/span><\/span>#define <\/span>DO_THING(macro_switch, b) CAT(IMPL, macro_switch) ECHO(( “Hello”, b))<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

DO_THING(1, “World”);\n<\/span><\/span><\/span><\/span>\/\/ Traditional preprocessor:\n<\/span><\/span>\/\/ do_thing_one( “Hello”, “World”);\n<\/span><\/span>\/\/ Conformant preprocessor:\n<\/span><\/span>\/\/ IMPL1 ( “Hello”,”World”);<\/span>\n<\/span><\/p>\n

Although this example is a bit contrived, we have run into this issue few times when testing the preprocessor changes against real world code. To see what is going on we can break down the expansion starting with DO_THING<\/span>:\n<\/span><\/p>\n

DO_THING(1, “World”)<\/span><\/span><\/span><\/span>— >\nCAT(IMPL, 1) ECHO((“Hello”, “World”))\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

Second, CAT is expanded:\n<\/span><\/p>\n

CAT(IMPL, 1)– > IMPL ## 1 — > IMPL1\n<\/span><\/span><\/span><\/span><\/p>\n

Which puts the tokens into this state:\n<\/span><\/p>\n

IMPL1 ECHO((“Hello”, “World”))\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

The preprocessor finds the function-like macro identifier IMPL1, but it is not followed by a “(“, so it is not considered a function-like macro invocation. It moves on to the following tokens and finds the function-like macro ECHO being invoked:\n<\/span><\/p>\n

ECHO((“Hello”, “World”))– > (“Hello”, “World”)\n<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

IMPL1 is never considered again for expansion, so the full result of the expansions is:\n<\/span><\/p>\n

IMPL1(“Hello”, “World”);\n<\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

The macro can be modified to behave the same under the experimental preprocessor and the traditional preprocessor by adding in another layer of indirection:\n<\/span><\/p>\n

#define <\/span>CAT(a,b) a##b\n<\/span><\/span><\/span><\/span>#define <\/span>ECHO(…) __VA_ARGS__<\/span><\/span><\/span><\/span><\/p>\n

\/\/ IMPL1 and IMPL2 are macros implementation details\n<\/span><\/span>#define <\/span>IMPL1(prefix,value) do_thing_one( prefix, value)\n<\/span><\/span><\/span><\/span>#define <\/span>IMPL2(prefix,value) do_thing_two( prefix, value)<\/span><\/span><\/span><\/span><\/p>\n

#define <\/span>CALL(macroName, args) macroName args\n<\/span><\/span><\/span><\/span>#define DO_THING_FIXED(a,b) CALL( CAT(IMPL, a), ECHO(( “Hello”,b)))<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n

DO_THING_FIXED(1, “World”);\n<\/span><\/span><\/span><\/span>\/\/ macro expanded to:\n<\/span><\/span>\/\/ do_thing_one( “Hello”, “World”);<\/span>\n<\/span><\/p>\n

What’s next\u2026\n<\/span><\/strong><\/span><\/p>\n

The preprocessor overhaul is not yet complete; we will continue to make changes under the experimental mode and fix bugs from early adopters. \n<\/span><\/p>\n