C++17/20 Features and Fixes in Visual Studio 2019

Stephan T. Lavavej - MSFT

April 19th, 201932 0

Visual Studio 2019 version 16.0 is now available and is binary compatible with VS 2015/2017. In this first release of VS 2019, we’ve implemented more compiler and library features from the C++20 Working Paper, implemented more <charconv> overloads (C++17’s “final boss”), and fixed many correctness, performance, and throughput issues. Here’s a list of the C++17/20 compiler/library feature work and the library fixes. (As usual, many compiler bugs were also fixed, but they aren’t listed here; compiler fixes tend to be specific to certain arcane code patterns. We recently blogged about compiler optimization and build throughput improvements in VS 2019, and we maintain a documentation page about compiler conformance improvements in VS 2019.)

New Features:

  • Implemented P1164 from C++20 unconditionally. This changes std::create_directory to check whether the target was already a directory on failure. Previously, all ERROR_ALREADY_EXISTS type errors were turned into success-but-directory-not-created codes.
  • The iterator debugging feature has been taught to properly unwrap std::move_iterator. For example, std::copy(std::move_iterator<std::vector<int>::iterator>, std::move_iterator<std::vector<int>::iterator>, int*) can now engage our memcpy fast path.
  • The standard library’s macroized keyword enforcement <xkeycheck.h> was fixed to emit the actual problem keyword detected rather than a generic message, look for C++20 keywords, and avoid tricking IntelliSense into saying random keywords were macros.
  • We added LWG 2221‘s operator<<(std::ostream, nullptr_t) for writing nullptrs to streams.
  • Parallel versions of is_sorted, is_sorted_until, is_partitioned, set_difference, set_intersection, is_heap, and is_heap_until were implemented by Miya Natsuhara.
  • P0883 “Fixing atomic initialization”, which changes std::atomic to value-initialize the contained T rather than default-initializing it, was implemented (also by Miya) when using Clang/LLVM with our standard library. This is currently disabled for C1XX as a workaround for a bug in constexpr processing.
  • Implemented the “spaceship” three-way comparison operator from P0515 “Consistent comparison”, with partial support for the C++20 <compare> header as specified in P0768 (specifically, the comparison category types and common_comparison_category type trait, but not the comparison algorithms which are undergoing some redesign in WG21). (Implemented by Cameron DaCamara in the compiler.)
  • Implemented the new C++20 P1008 rules for aggregates: a type with a user-declared constructor – even when defaulted or deleted so as not to be user-provided – is not an aggregate. (Implemented by Andrew Marino in the compiler.)
  • Implemented the remove_cvref and remove_cvref_t type traits from P0550, which are handy for stripping reference-ness and cv-qualification but without decaying functions and arrays to pointers (which std::decay and std::decay_t do).
  • C++17 <charconv> floating-point to_chars() has been improved: shortest chars_format::fixed is 60-80% faster (thanks to Ulf Adams at Google for suggesting long division), and shortest/precision chars_format::hex is complete. Further performance improvements for shortest fixed notation have been implemented and will ship in a future VS 2019 update, along with the decimal precision overloads that will complete the <charconv> implementation.
  • C++20 P0941 feature-test macros are now completely supported in the compiler and STL, including __has_cpp_attribute implemented by Phil Christensen. As a reminder, the feature-test macros are always active (i.e. defined or not defined, depending on the availability of the feature in question) regardless of the Standard mode option selected, because making them conditional on /std:c++latest would largely defeat their purpose.

Correctness Fixes:

  • std::allocator<void>, std::allocator::size_type, and std::allocator::difference_type have been un-deprecated.
  • A spurious static_cast not called for by the standard that accidentally suppressed C4244 narrowing warnings was removed from std::string. Attempting to call std::string::string(const wchar_t*, const wchar_t*) will now properly emit C4244 “narrowing a wchar_t into a char.”
  • Fixed std::filesystem::last_write_time failing when attempting to change a directory’s last write time.
  • std::filesystem::directory_entry‘s constructor was changed to store a failed result, rather than throwing an exception, when supplied a nonexistent target path.
  • std::filesystem::create_directory‘s 2-parameter version was changed to call the 1-parameter version, as the underlying CreateDirectoryExW function would perform copy_symlink when the existing_p was a symlink.
  • std::filesystem::directory_iterator no longer fails when encountering a broken symlink.
  • std::filesystem::space now accepts relative paths.
  • std::filesystem::path::lexically_relative is no longer confused by trailing slashes, reported as LWG 3096.
  • Worked around CreateSymbolicLinkW rejecting paths with forward slashes in std::filesystem::create_symlink.
  • Worked around the POSIX deletion mode delete function existing on Windows 10 LTSB 1609 but not actually being capable of deleting files.
  • std::boyer_moore_searcher and std::boyer_moore_horspool_searcher‘s copy constructors and copy assignment operators now actually copy things.
  • The parallel algorithms library now properly uses the real WaitOnAddress family on Windows 8 and later, rather than always using the Windows 7 and earlier fake versions.
  • std::system_category::message() now trims trailing whitespace from the returned message.
  • Some conditions that would cause std::linear_congruential_engine to trigger divide by 0 have been fixed.
  • The iterator unwrapping machinery we first exposed for programmer-user integration in VS 2017 15.8 (as described in https://devblogs.microsoft.com/cppblog/stl-features-and-fixes-in-vs-2017-15-8/ ) no longer unwraps iterators derived from standard library iterators. For example, a user that derives from std::vector<int>::iterator and tries to customize behavior now gets their customized behavior when calling standard library algorithms, rather than the behavior of a pointer.
  • The unordered container reserve function now actually reserves for N elements, as described in LWG 2156.
  • Many STL internal container functions have been made private for an improved IntelliSense experience. Additional fixes to mark members as private are expected in subsequent releases of MSVC.
  • Times passed to the concurrency library that would overflow (e.g. condition_variable::wait_for(seconds::max())) are now properly dealt with instead of causing overflows that changed behavior on a seemingly random 29-day cycle (when uint32_t milliseconds accepted by underlying Win32 APIs overflowed).
  • Exception safety correctness problems wherein the node-based containers like list, map, and unordered_map would become corrupted were fixed. During a propagate_on_container_copy_assignment or propagate_on_container_move_assignment reassignment operation, we would free the container’s sentinel node with the old allocator, do the POCCA/POCMA assignment over the old allocator, and then try to acquire the sentinel node from the new allocator. If this allocation failed, the container is corrupted and can’t even be destroyed, as owning a sentinel node is a hard data structure invariant. This was fixed to allocate the new sentinel node from the source container’s allocator before destroying the existing sentinel node.
  • The containers were fixed to always copy/move/swap allocators according to propagate_on_container_copy_assignment, propagate_on_container_move_assignment, and propagate_on_container_swap, even for allocators declared is_always_equal.
  • std::basic_istream::read was fixed to not write into parts of the supplied buffer temporarily as part of \r\n => \n processing. This gives up some of the performance advantage we gained in VS 2017 15.8 for reads larger than 4k in size, but efficiency improvements from avoiding 3 virtual calls per character are still present.
  • std::bitset‘s constructor no longer reads the ones and zeroes in reverse order for large bitsets.
  • When implementing P0083 “Splicing Maps And Sets”, we managed to overlook the fact that the merge and extract members of the associative containers should have overloads that accept rvalue containers in addition to the overloads that accept lvalue containers. We’ve rectified this oversight by implementing the rvalue overloads.
  • With the advent of the Just My Code stepping feature, we no longer need to provide bespoke machinery for std::function and std::visit to achieve the same effect. Removing that machinery largely has no user-visible effects, except that the compiler will no longer produce diagnostics that indicate issues on line 15732480 or 16707566 of <type_traits> or <variant>.
  • The <ctime> header now correctly declares timespec and timespec_get in namespace std in addition to declaring them in the global namespace.
  • We’ve fixed a regression in pair‘s assignment operator introduced when implementing LWG 2729 “Missing SFINAE on std::pair::operator=“; it now correctly accepts types convertible to pair again.
  • Fixed a minor type traits bug, where add_const_t etc. is supposed to be a non-deduced context (i.e. it needs to be an alias for typename add_const<T>::type, not const T).

Header Inclusion Restructuring:

The standard library’s physical design was substantially overhauled to avoid including headers when they are not necessary. A large number of customers want to use standard library containers but don’t want to use the iostreams and locales. However, the C++ standard library has a circular dependency among components:

  • Systems that depend on <locale> facets want to use std::string as part of their underlying implementations.
  • std::string wants a stream insertion operator, which depends on std::ostream, which depends on <locale>.
  • Historically our standard library worked around this problem by introducing a lower level header <xstring>, which defined std::string, but none of the other contents of <string>. <xstring> would be included by both <locale> components, and by <string>, restoring the directed dependency graph. However, this approach had a number of problems:
  • #include <string> needed to drag in all of the iostreams machinery to provide the stream insertion operator, even though most translation units wouldn’t use the stream insertion operator.
  • If someone included only <ostream> they got std::basic_string and std::ostream, but they did not get std::basic_string‘s stream insertion operator, the std::string typedef, or the string literals. Customers found this extremely confusing. For example, if one tried to stream insert a std::basic_string after including only <ostream>, the compiler would print an incredibly long diagnostic saying operator<< couldn’t be found, listing 26 unrelated overloads. Also, attempts to use std::string_literals, std::to_string, or other <string> components, would fail, which is confusing when std::basic_string was otherwise available.

In VS 2019, we resolve the circular reference completely differently. The stream insertion operator now finds the necessary ostream components using argument-dependent lookup, allowing us to provide it in the same place as string. This restores appropriate layering (of std::string below <locale> components), and makes it possible to use <string> without dragging in the entire mass of iostreams machinery.

If you have lots of .cpp files that include string and do something simple, for example:

#include <stdio.h>
#include <string>

void f(const std::string& s) {
  puts(s.c_str());
}

In VS 2017 15.9 this program takes 244 milliseconds to compile on a 7980XE test machine (average of 5 runs), while in VS 2019 16.0 it takes only 178 milliseconds (or about 73% of the time).

Moreover, seemingly unrelated headers like <vector> were pulled into this mess. For example, vector wants to throw std::out_of_range, which derives from std::runtime_error, which has a constructor that takes a std::string. We already had out-of-line functions for all throw sites, so the spurious include of <stdexcept> in <vector> was unnecessary and has been removed. The following program used to take 177 milliseconds to compile in VS 2017 15.9, but now only needs 151 milliseconds (85% of the time):

#include <vector>

void f(std::vector<int>& v) {
  v.push_back(42);
}

The one downside of this change is that several programs that were getting away with not including the correct headers may need to add #includes. If you were saying std::out_of_range before, you may need to #include <stdexcept>. If you were using a stream insertion operator, you may now need to #include <ostream>. This way, only translation units actually using <stdexcept> or <ostream> components pay the throughput cost to compile them.

Performance and Throughput Improvements:

  • if constexpr was applied in more places in the standard library for improved throughput and reduced code size in the copy family, permutations like reverse and rotate, and in the parallel algorithms library.
  • The STL now internally uses if constexpr to reduce compile times even in C++14 mode.
  • The runtime dynamic linking detection for the parallel algorithms library no longer uses an entire page to store the function pointer array, as marking this memory read-only was deemed no longer relevant for security purposes.
  • std::thread‘s constructor no longer waits for the thread to start, and no longer inserts so many layers of function calls between the underlying C library _beginthreadex and the supplied callable object. Previously std::thread put 6 functions between _beginthreadex and the supplied callable object, which has been reduced to only 3 (2 of which are just std::invoke). This also resolves an obscure timing bug where std::thread‘s constructor would hang if the system clock changed at the exact moment a std::thread was being created.
  • Fixed a performance regression in std::hash that we introduced when implementing std::hash<std::filesystem::path>.
  • Several places the standard library used to achieve correctness with catch blocks now use destructors instead. This results in better debugger interaction — exceptions you throw through the standard library in the affected locations will now show up as being thrown from their original throw site, rather than our rethrow. Not all standard library catch blocks have been eliminated; we expect the number of catch blocks to be reduced in subsequent releases of MSVC.
  • Suboptimal codegen in std::bitset caused by a conditional throw inside a noexcept function was fixed by factoring out the throwing path.
  • The std::list and std::unordered_meow family use non-debugging iterators internally in more places.
  • Several std::list members were changed to reuse list nodes where possible rather than deallocating and reallocating them. For example, given a list<int> that already has a size of 3, a call to assign(4, 1729) will now overwrite the ints in the first 3 list nodes, and allocate one new list node with the value 1729, rather than deallocating all 3 list nodes and then allocating 4 new list nodes with the value 1729.
  • All locations the standard library was calling erase(begin(), end()) were changed to call clear() instead.
  • std::vector now initializes and erases elements more efficiently in certain cases.
  • <variant> has been refactored to make it more optimizer-friendly, resulting in smaller and faster generated code. Most notably, std::visit and the inliner have now become good friends.
  • We’ve applied clang-format to the STL’s headers for improved readability. (There were additional manual changes, e.g. adding braces to all control flow.)

Reporting Bugs:

Please let us know what you think about VS 2019. You can report bugs via the IDE’s Report A Problem and also via the web, at the Developer Community’s C++ tab.

Billy O’Neal, Casey Carter, and Stephan T. Lavavej

Stephan T. Lavavej - MSFT Programmer-Archaeologist, Visual C++ Libraries

Read next

32 comments

Discussion is closed. Login to edit/delete existing comments.

  • Hristo Hristov 0

    When will start more C++20 features to  pop in? Like ranges, concepts, and especially this thing here:{ auto fsTime = std::filesystem::last_write_time(filename); return decltype(fsTime)::clock::to_time_t(fsTime); }

    • Billy O'NealMicrosoft employee 0

      Calendars and Time Zones is near the top of the list precisely because folks have explicitly asked to move it to the top, but no firm ETA at this time, sorry. It’s probably an even split between that feature and ranges for “biggest and most expensive library feature to implement” voted into C++20.

      One point to correct here though, C++20 does not add a to_time_t member to file_time_clock. What it does add is clock_cast, which lets you easily convert from any clock type back into system_clock, which has the to_time_t member.

    • unituniverse2@yahoo.com 0

      I got a bad feeling about concepts… Hoping I may be wrong.

  • Alex Rodionov 0

    Does std::unordered_meow still randomly push items off its table?

    • Billy O'NealMicrosoft employee 0

      I’m unfamilar with unordered_meow randomly doing anything (other than the usual probilistic nature of hashes). Have you filed a repro on Developer Community?

      • Trevor Talbot 0

        Developer Community does not seem to provide a suitable input method for a repro in this case. What is the appropriate address to send a sample feline?

        • Billy O'NealMicrosoft employee 0

          Sorry, you had me terrified I had broken unordered_Xxx in the shipping product for a moment. There was a period where I broke the heck out of them after I implemented LWG 2156 because we didn’t do as good a job internally documenting the container’s invariants as we should have.

  • Sunil Joshi 0

    Can you say what the impact of binary compatability for Visual Studio 2019 has? What features and improvements (performance or conformance) are waiting for the next binary incompatible release?

    • Stephan T. Lavavej - MSFTMicrosoft employee 0

      We’ve figured out how to deliver almost all new features in a binary-compatible manner (including Filesystem and other things that need separately compiled code), but some major improvements to existing features are blocked. In a separate branch (that we need to migrate from Team Foundation Version Control to git), we have:

      * A rewrite of the multithreading library, fixing many conformance and performance bugs

      * A rewrite of iterator debugging machinery throughout the STL, avoiding dynamic memory allocation and quadratic wall-clock complexity (almost complete; deque and vector<bool> remain to be overhauled)

      * Removals of old machinery (hash_map/hash_set, tr1, broken and unused floating-point conversion code, /clr:pure support, more in the future)

      We’ll be able to further improve the multithreading machinery if we can simultaneously drop support for XP and Vista targeting (not yet decided; we will support Win7 targeting for a very very long time, and being able to assume Win7+ will allow us to unconditionally use the OS’s condition variable support).

      There are also many potential compiler improvements that would break ABI, including universal support for the empty base class optimization, potential improvements to RTTI size, fixing name mangling issues, etc.

      To be clear: the entire VS 2019 release series (RTW/16.0 and all future 16.x updates) will contain the binary-compatible 19.2x toolset that can be freely mixed with VS 2015/2017. Any binary-incompatible toolset work will appear in a different major version of the IDE, or as an opt-in additional toolset (no specific timeframe planned right now) – you will not be switched over automatically.

      • Owens, Bowie (Data61, Clayton) 0

        So I can compile a static library with vs2019 and deliver to someone who will link with other code compiled by vs2017 and that will work? Any other steps required?

        • Stephan T. Lavavej - MSFTMicrosoft employee 0

          Yes, it will work, with the following restrictions: (1) Link-Time Code-Gen (LTCG, controlled by the /GL compiler option) is incompatible with mixing-and-matching compiler versions, and (2) the linker that performs the final link of object files and static libraries into the output executable can’t be older than the compiler versions used to generate those object files and static libraries. That is, if your static library is built with VS 2017 15.0, and it’s linked into an application that’s otherwise built with VS 2019 16.0, and the final linker is 16.0, that’s fine. Also fine if the static library is compiled with 16.0 and the application is compiled with 15.0, but the final linker is 16.0. However, if your static library is 16.0 and the application is both compiled and linked with 15.0, that is not supported (and will fail to link in certain scenarios). This binary compatibility is mostly meant for applications to be able to upgrade their toolsets without simultaneously upgrading all compiled dependencies, but it isn’t symmetric for third-party library authors. (Note that even though some mixing-and-matching is supported, it is still best for the whole program to be built with the newest version when possible, as that will give you the most correctness and performance fixes.)

          This is documented at https://docs.microsoft.com/en-us/cpp/porting/binary-compat-2015-2017?view=vs-2019 (the behavior is the same for 2015/2017/2019).

      • Hristo Hristov 0

        I really wish you keep the IDE for longer (4-5-6 years) and to update the toolsets. One problem I experienced is that third party extensions don’t get updated as often or at all and as a resault if we cannot upgrade to a newer VS IDE for the life of the project (which may span a decade).

      • Nikolay Baklicharov 0

        You should really include the binary incompatible toolset before the next major release of VS. There are many projects that can migrate without considering backward compatibility.

    • Jonathan Wilson 0

      I seem to recall statements in the past from the VS dev team (probably in posts or comments on the Visual C++ blog) that said “we cant do xyz in VS 2017 because it is ABI breaking but the next version will be one where we can break the ABI” or something similar. What changed and why did the VS dev team not go ahead with the original plan to make the ABI breaking changes in VS 2019 in the way the dev team had originally indicated was the plan?

      • Stephan T. Lavavej - MSFTMicrosoft employee 0

        Short explanation: the plan wasn’t that simple (and we clearly miscommunicated the subtleties), we had to put this work on the back burner for several reasons, and C++17/20 work has been higher priority.

        The plan: we initially envisioned shipping the binary-incompatible branch (“vNext”, aka “WCFB02”) as an opt-in preview alongside the compatible toolset. I’ve personally tried to make this clear by talking about the “next binary-incompatible major version”, but it’s really easy to read that as “the next major version”.

        The back burner: We had been spending a fair amount of time on the vNext branch (Billy overhauled atomics/threading, Steve overhauled iostreams and ABI stability, I mostly overhauled iterator debugging), but then other work items took priority (e.g. non-STL library work, the infinite expanse of C++17 charconv, etc.) We also switched from TFS to Git for source control, which has been a massive productivity boon, but also stranded our vNext changes in TFS (extensive manual work will be required to port them to Git, not because of TFS/Git themselves, but because we’ve significantly changed the underlying code).

        C++17/20: We’ve gotten better at delivering features in a binary-compatible way, even when they have separately compiled components. The Committee has been relentlessly developing new features, and by spending dev time on keeping up with the pace of Standardization, we can deliver immediate value to customers. This comes at the cost of working on vNext (which is long-range work without an immediate payoff).

        While binary compatibility has proven to be extremely valuable to our customers, management is definitely hearing increased customer interest in the significant improvements that require a bincompat break, so the vNext work hasn’t been cancelled in any way – it will simply take longer to arrive than we initially thought.

  • Nikita Kniazev 0

    > P0883 “Fixing atomic initialization”, which changes std::atomic to value-initialize the contained T rather than default-initializing it, was implemented (also by Miya) when using Clang/LLVM with our standard library. This is currently disabled for C1XX as a workaround for a bug in constexpr processing.

    This is about statics constant initialization by constexpr constructors bug, right? Do you know when it will be fixed? There is not even a comment on the corresponding ticket tha was open more than a half year ago, and `boost::atomic` also suffering because of it.

    • Billy O'NealMicrosoft employee 0

      Rest assured that your standard library implementers are more frustrated than anyone by this bug 🙂

  • Jan Ringoš 0

    Just a tiniest correction: “Windows 10 LTSB 1609” doesn’t really exist. It’s either 1607, or LTSB 2016 😉

  • greg.hickman@lmco.com 0

    Just curious: Why require braces for all control flow?

    • Stephan T. Lavavej - MSFTMicrosoft employee 0

      It prevents bugs from creeping in during maintenance.

  • Hristo Hristov 0

    Will there be a concepts enabled standard library? Will it come as a preview first or it will only come as a final product?

    • Stephan T. Lavavej - MSFTMicrosoft employee 0

      We’re going to implement the C++20 Ranges library (designed by Eric Niebler at Facebook and Casey Carter on the Visual C++ Libraries team), which is powered by concepts (and will be the first consumer of the compiler’s concepts implementation). C++20 is nearing completion, so I don’t expect major new concept-powered machinery to appear beyond the <concepts> and <ranges> headers, but C++23 and beyond will certainly be adding more.

      When we implement features from the C++ Working Paper, we generally ship them in a complete form, at production quality, but with the understanding that we may need to make source breaking and even binary breaking changes if they evolve before final Standardization (that is, our binary compatibility is “frozen” when a Standard ships, but not for /std:c++latest features). That said, sometimes we do ship partial implementations, as is happening with char8_t and spaceship.

      • Hristo Hristov 0

        I guess I was wrong. I was under the impression after Concepts are voted in there would be also a concepts implementation of the C++ containers. So there aren’t any such proposals/TSs or proof-of-concept implementations beyound <ranges>.

  • Me Gusta 0

    Is there any news on the preprocessor? There hasn’t been any news since last July.

    • Stephan T. Lavavej - MSFTMicrosoft employee 0

      No news at this time. (I recently asked the compiler front-end dev lead about /experimental:preprocessor.)

  • Jesse Pepper 0

    Hey there STL, I’m using Visual Studio Enterprise 2019 v16.1.0, Visual C++ 2019 – 00435-60000-00000-AA453, and CMake reports: 
    [CMake] — The CXX compiler identification is MSVC 19.21.27702.2

    here’s the contents of the file to reproduce it.  It seems to be some interaction between std::variant and a templatised function.  Note the template parameter is note even used in the template function but making it non templatised stops the ICE.  I removed main for brevity because the ICE happens before link time so it will still fail:
    — file below —

    // [CMake] — The CXX compiler identification is MSVC 19.21.27702.2#include <variant>inline void f(int i){}template <typename T>void g_templatised(){std::variant<int> v;f(std::get<int>(v));}
    /*Produces:
    C:\Users\JessePepper\source\repos\mi-geometry-framework\miAutomaticUVMapping\main.cpp(9): fatal error C1001: An internal error has occurred in the compiler.(compiler file ‘d:\agent\_work\2\s\src\vctools\Compiler\CxxFE\sl\p1\c\convert.cpp’, line 674)To work around this problem, try simplifying or changing the program near the locations listed above.Please choose the Technical Support command on the Visual C++Help menu, or open the Technical Support help file for more information
    */
    — end of file —

    By the way it would be nice to be able to have a bit of backtick code markups in here.

      • Jesse Pepper 0

        Thanks… wasn’t being lazy just didn’t really know where to submit it.

        • Stephan T. Lavavej - MSFTMicrosoft employee 0

          No problem. You can also report bugs through the IDE (Report A Problem).

  • brawl dummy 0

    Hallo ich möchte für meine Schule ein Programm schreiben hat jemand so was vielleicht schonmal gemacht und kann mir Tipps geben ich würde mich über antworten freuen.

Feedback usabilla icon