{"id":6871,"date":"2015-07-14T17:38:09","date_gmt":"2015-07-14T17:38:09","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/vcblog\/2015\/07\/14\/stl-fixes-in-vs-2015-part-2\/"},"modified":"2019-02-18T18:05:01","modified_gmt":"2019-02-18T18:05:01","slug":"stl-fixes-in-vs-2015-part-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/stl-fixes-in-vs-2015-part-2\/","title":{"rendered":"STL Fixes In VS 2015, Part 2"},"content":{"rendered":"

In addition to implementing a whole bunch of features<\/span><\/a> in VS 2015’s C++ Standard Library, we’ve also fixed a whole bunch of bugs.  A year ago, I listed the STL fixes in VS 2015 CTP1<\/span><\/a>.  Now, it’s time to list the STL fixes between CTP1 and RTM. <\/span>\n  \n<functional> Overhaul <\/strong><\/span>\n  \nWe originally shipped bind()\/function\/mem_fn()\/reference_wrapper\/etc. as part of TR1 in VS 2008 SP1.  As TR1 evolved into C++0x and then into C++11, we updated our implementation to use rvalue references, decltype, and variadic templates.  While this mostly worked, the TR1-era machinery was extremely complicated, and it accumulated a large number of bugs over the years.  We don’t want the Standard Library to “mostly work” – we want crystalline perfection.  So, I rewrote <functional> almost entirely from scratch, keeping only C++03’s stuff and std::function’s skeleton.  This significantly reduced the size of the STL’s source code (by 24 KB, 935 editor lines).  Except for one limitation (Expression SFINAE in function\/result_of), overhauling <functional> fixed all known bugs: <\/span>\n  \n* In certain scenarios, bind() wouldn’t compile with a function object inheriting operator()() from a base class (DevDiv#617421\/Connect#779061<\/span><\/a>). <\/span>\n  \n* bind() wasn’t perfectly forwarding unbound arguments passed through placeholders like _1 (DevDiv#343411\/Connect#717188<\/span><\/a>, DevDiv#410033\/Connect#737872<\/span><\/a>, DevDiv#862588\/Connect#814028<\/span><\/a>). <\/span>\n  \n* bind()’s function call operator wasn’t const-overloaded (DevDiv#535246\/Connect#773565<\/span><\/a>, DevDiv#1034816\/Connect#981289<\/span><\/a>).  My original fix accidentally tried to use Expression SFINAE, but I was able to fix that for RTM (DevDiv#1160769, DevDiv#1165732\/Connect#1298009<\/span><\/a>). <\/span>\n  \n* bind()\/function\/etc. had difficulty with PMFs\/PMDs (pointers to member functions\/data).  The Standard specifies that bind()\/function\/etc. call an imaginary function named INVOKE(), which does magical things (N4527<\/span><\/a> 20.9.2 [func.require]\/1).  INVOKE() knows that function objects, PMFs, and PMDs all require different syntax, and it knows how to use PMFs\/PMDs with references\/raw pointers\/smart pointers to base\/derived objects.  In C++17, this is a real function invoke() that you can call directly.  My <functional> overhaul implemented invoke(), following the Standard exactly, and therefore fixed all of the bugs with PMFs\/PMDs (DevDiv#294051\/Connect#694400<\/span><\/a>, DevDiv#477804\/Connect#759806<\/span><\/a>, DevDiv#487679\/Connect#763571<\/span><\/a>, DevDiv#505570\/Connect#768903<\/span><\/a>, DevDiv#505572\/Connect#768899<\/span><\/a>, DevDiv#789899\/Connect#802299<\/span><\/a>). <\/span>\n  \n* bind() and std::function had several space inefficiencies, which could lead to dramatic time inefficiencies.  An example was reported where storing a bound functor in a std::function worked fine on x86, but was almost 10x slower on x64 (DevDiv#490878).  This was triggered by x64’s larger pointers exceeding the limit of our Small Functor Optimization (SFO), assisted by unexpected bloat in bind() and std::function’s representations.  Fixing this involved several things.  First, I reworked std::function’s representation, eliminating unnecessary bloat and making more space available for the SFO.  Second, I retuned the SFO, which previously had a very small limit.  The limit is officially undocumented and we reserve the right to change it in the future, but I can tell you what it is.  In VS 2015, we consider a function object to be “small” (and can therefore avoid dynamic memory allocations) if it’s as big as a basic_string or smaller.  (James McNellis suggested this heuristic.)  Our basic_string is fairly large due to the Small String Optimization, so this means that functors can store several integers\/pointers while remaining within the SFO.  Finally, as part of an STL-wide overhaul to use compressed pairs, bind() compresses empty functors (like stateless lambdas or less<>) and std::function compresses empty allocators (like std::allocator, the default).  This avoids wasting the SFO’s precious bits on storing empty classes.  With all of these changes, we expect the SFO to be activated much more frequently.  It’ll still be possible for 64-bit architectures to push user functors over the limit, but far fewer functors will be affected. <\/span>\n  \nThere are minor limitations, which I’ll mention for completeness.  First, std::function doesn’t attempt to optimize empty functors with stateful allocators.  (This doesn’t matter in practice, because a stateful allocator would have to be just below the SFO’s limit in order to be affected.)  Second, bind() doesn’t attempt to optimize stateful functors bound to zero arguments.  (This is extremely unlikely to matter in practice, because the only reason to bind zero arguments would be for nested bind().  bind(pmf\/pmd) isn’t a reason, because that’s better written with mem_fn() or a stateless lambda.)  Finally, bind() stores bound arguments in a tuple, which isn’t compressed, so binding placeholders or other empty arguments will pay 1 byte per argument.  (This isn’t a big deal because the size of a bound functor rarely matters except when it exceeds the SFO’s limit, which we’ve dramatically increased.  We may compress tuple in the future.) <\/span>\n  \n* function::swap() is required to be noexcept.  To achieve this, I’ve given the SFO an additional requirement.  A functor must be small and is_nothrow_move_constructible must be true in order to activate the SFO. <\/span>\n  \n* std::function was generating larger object\/executable files than necessary, due to how the SFO was metaprogrammed (DevDiv#1174895).  I reworked this, reducing the size of an artificial test case by 12% on x86 and 30% on x64.  This was reported by the Office team, and the fix reduced the size of their x64 executable by 9% (they had many std::functions). <\/span>\n  \n* The constructor function(F f) is required to store move(f), but we were copying it (DevDiv#759096).  While I’ve fixed this, note that the Standard still requires F to be CopyConstructible, so you can’t store movable-only functors in std::function.  (Usually, STL classes have “on-demand” requirements.  For example, list<T> doesn’t require T to be less-than comparable, until you call list<T>::sort(), thanks to how templates work.  std::function behaves differently due to type erasure.  When you construct function<Signature> from UserFunctor, the std::function needs to generate all the operations on UserFunctor that could possibly be called, because its actual type is about to be erased through the magic of virtual functions, regardless of whether those operations will ultimately be needed by the whole program.  Because std::function is CopyConstructible, it must require UserFunctor to be CopyConstructible, regardless of whether you actually copy any std::functions.) <\/span>\n  \n* function<void (Args)> is now required to accept functors with non-void return types (LWG 2420<\/span><\/a>, DevDiv#1010027\/Connect#949899<\/span><\/a>).  If the functor returns anything, it will be ignored. <\/span>\n  \n* For certain types (with long mangled names, like set<wstring>), function<Ret (Args)> emitted warning C4503 “decorated name length exceeded, name was truncated” (DevDiv#1053579, DevDiv#1094949\/Connect#1052543<\/span><\/a>).  This was extremely annoying, although it didn’t affect runtime behavior.  During VS 2015’s development, this problem temporarily became worse, while I was reworking std::function to properly use allocators.  I tracked down the ultimate causes of C4503 and mitigated them as much as possible, so it’ll be very difficult to trigger unintentionally.  (C4503 can’t be completely avoided with the current name mangling system.  The compiler team may change this in the future.) <\/span>\n  \n* reference_wrapper had trouble with function objects that defined result_type (DevDiv#794227), especially function objects that were simultaneously unary and binary (DevDiv#864867). <\/span>\n  \n* reference_wrapper had trouble with function types, like reference_wrapper<int (int)>, where get() and assignment wouldn’t compile (DevDiv#535636, DevDiv#868374). <\/span>\n  \n* reference_wrapper<AbstractBase> didn’t work with pure virtual function call operators (DevDiv#391117\/Connect#734305<\/span><\/a>, DevDiv#897806\/Connect#828696<\/span><\/a>). <\/span>\n  \n* Everything in <functional> (except for the deprecated-in-C++11, removed-in-C++17 stuff like ptr_fun()) now respects arbitrary calling conventions, including __vectorcall, and the compiler options to change the default calling convention (\/Gd, \/Gr, \/Gv, \/Gz).  Unlike Boost, this is always supported, so you don’t need to define macros to enable it.  (DevDiv#553067\/Connect#774720<\/span><\/a>, DevDiv#563620\/Connect#775554<\/span><\/a>, DevDiv#793009\/Connect#804357<\/span><\/a>) <\/span>\n  \ncall_once() Overhaul <\/strong><\/span>\n  \nI rewrote call_once() and its associated once_flag from scratch, using Vista+’s InitOnceExecuteOnce()<\/span><\/a> with a handwritten fallback for XP.  This fixed all known bugs: <\/span>\n  \n* Exceptional executions weren’t handled as required by the Standard (DevDiv#637433\/Connect#781049<\/span><\/a>, DevDiv#1086953\/Connect#1038276<\/span><\/a>). <\/span>\n  \n* once_flag’s constructor wasn’t marked as constexpr as required by the Standard (DevDiv#497946).  (Note that while it’s marked as constexpr in VS 2015 RTM, it’s affected by the compiler bug DevDiv#1134662 “constexpr constructors are emitting dynamic initializers”, which we’re planning to fix in 2015 Update 1.) <\/span>\n  \n* call_once() previously used a global lock, which led to hangs in certain situations (DevDiv#840791\/Connect#811192<\/span><\/a>).  This global lock also prevented separate call_once() invocations with separate once_flags from executing simultaneously (DevDiv#1092852). <\/span>\n  \n* call_once() previously used bind(), which it wasn’t supposed to do.  (bind() has special cases for placeholders, reference_wrappers, and nested bind() expressions – none of which should be treated specially by call_once().) <\/span>\n  \n* The new implementation is much more efficient.  I measured call_once()’s performance on my dev box (4-core 8-thread Sandy Bridge i7-2600 3.4 GHz; Server 2008 R2, Win7-class) by having 8 threads hammer a single once_flag with an empty lambda.  To make the test take approximately one second, I needed 1M repeats per thread for VS 2013, and 35M for VS 2015.  This allowed me to measure call_once()’s overhead very precisely.  (For example, if it takes 1 s of wall clock time for 8 threads to perform 1M calls each, then I say that each call took 1 s \/ 8M = 125 ns.)  Compared to VS 2013, I observe that VS 2015 x86 is 37.2x faster<\/strong> (212.0 ns improved to 5.7 ns), and x64 is 87.7x faster<\/strong> (306.9 ns improved to 3.5 ns).  I believe that the difference between x86’s 5.7 ns and x64’s 3.5 ns is due to x64’s zero-overhead table-based exception handling (we need a try\/catch block to transport exceptions, which can’t be allowed to rampage through InitOnceExecuteOnce()). <\/span>\n  \nAtomic Fixes <\/strong><\/span>\n  \n* atomic<T> now static_asserts that T must be trivially copyable (DevDiv#798735\/Connect#805015<\/span><\/a>).  For example, atomic<string> will emit a compiler error. <\/span>\n  \n* The Standard forgot to say so (tracked by the open issue LWG 2426<\/span><\/a>), but the compare_exchange function family is supposed to read from “expected” before performing the atomic operation, and not after (DevDiv#1015776).  This permits certain algorithms to be written, e.g. manipulating list nodes. <\/span>\n  \n* shared_ptr’s atomic_compare_exchange function family could deadlock in certain situations (DevDiv#1066589\/Connect#1004269<\/span><\/a>).  We fixed this, in addition to other subtle correctness and performance problems. <\/span>\n  \n* <atomic> now supports the \/clr compiler option (DevDiv#1088751\/Connect#1041602<\/span><\/a>).  Note that the Clause 30 multithreading headers (<thread>, <future>, etc.) remain blocked; we’ll investigate lifting that restriction in the future. <\/span>\n  \nMultithreading Fixes (ConcRT Removal) <\/strong><\/span>\n  \nWe’ve reimplemented the STL’s multithreading primitives to avoid using the Concurrency Runtime (ConcRT).  Using ConcRT was a good idea at the time (2012), but it proved to be more trouble than it was worth.  Now we’re using the Windows API directly, which has fixed many bugs.  (The only exception is that ConcRT is still used for XP targeting, so XP hasn’t received these fixes.) <\/span>\n  \n* Using a mutex while constructing a global variable could hang (DevDiv#789979\/Connect#802344<\/span><\/a>, DevDiv#968936\/Connect#900741<\/span><\/a>). <\/span>\n  \n* async()\/mutex\/etc. would crash when operator new\/delete had been replaced (DevDiv#943284\/Connect#868994<\/span><\/a>, DevDiv#1019418\/Connect#962406<\/span><\/a>). <\/span>\n  \n* timed_mutex::try_lock_for()\/etc. leaked memory (DevDiv#957609\/Connect#886682<\/span><\/a>). <\/span>\n  \n* condition_variable improperly handled zero timeouts (DevDiv#1082065\/Connect#1029842<\/span><\/a>). <\/span>\n  \nMultithreading Fixes (Other) <\/strong><\/span>\n  \n* When constructing a std::thread, the Standard requires that “If the invocation of [the user’s callable object] terminates with an uncaught exception, std::terminate shall be called.” (N4527<\/span><\/a> 30.3.1.2 [thread.thread.constr]\/5).  We previously enforced this with try\/catch, but that made debugging difficult (DevDiv#918396\/Connect#845184<\/span><\/a>).  Now we enforce this by allowing user exceptions to slam into noexcept, which is friendly to debugging. <\/span>\n  \n* condition_variable’s predicate-wait functions were calling their predicates too often (DevDiv#1114006\/Connect#1098841<\/span><\/a>).  Now they’re following the Standard exactly. <\/span>\n  \n* std::thread’s constructor didn’t compile with movable-only arguments (DevDiv#377755\/Connect#729886<\/span><\/a>, DevDiv#1079684\/Connect#1026715<\/span><\/a>). <\/span>\n  \n* this_thread::get_id() was unnecessarily slow (DevDiv#1039430). <\/span>\n  \n* We’ve fixed all of the correctness issues reported by Anthony Williams<\/span><\/a> (DevDiv#482769), except for “the std::atomic<> class template cannot be used on types without a default constructor” which is tracked by the open issue LWG 2334<\/span><\/a>.  A couple of the performance issues remain to be investigated. <\/span>\n  \n* When called after main() exits (e.g. in a global destructor), thread::join() could hang (DevDiv#435439\/Connect#747145<\/span><\/a>). <\/span>\n  \n* packaged_task::make_ready_at_thread_exit() could hang when called more than once (DevDiv#803317). <\/span>\n  \n* packaged_task::reset() crashed when called on a packaged_task with no shared state (DevDiv#865726).  Now it throws future_error with future_errc::no_state as required by the Standard. <\/span>\n  \n* std::thread didn’t work with DLLs properly; it was possible for a DLL to be unloaded while a thread was still executing code in that DLL (DevDiv#895753).  This was especially problematic in Windows Store apps. <\/span>\n  \n* During VS 2015’s development, timed_mutex\/recursive_timed_mutex’s try_lock_for()\/try_lock_until() would spin a CPU core at 100% (DevDiv#1114242\/Connect#1099043<\/span><\/a>).  They’ve been rewritten to properly block. <\/span>\n  \n* When timing out, condition_variable’s wait_for()\/etc. could wake up slightly too early (DevDiv#1129370). <\/span>\n  \nFilesystem Fixes <\/strong><\/span>\n  \nIn addition to implementing the N4100<\/span><\/a> Filesystem “V3” Technical Specification, we fixed several bugs.  (Note that while we’re providing the TS’s header <experimental\/filesystem> and namespace std::experimental::filesystem, we’re also providing the old header <filesystem> and namespace std::tr2::sys for limited back-compat.) <\/span>\n  \n* The current_path() getter previously ignored failures (DevDiv#430113).  We now throw an exception or report an error_code as appropriate.  (In particular, this isn’t supported for Windows Store apps, so it’ll fail there.) <\/span>\n  \n* path::generic_string()\/etc. wasn’t returning forward slashes (DevDiv#982173\/Connect#916517<\/span><\/a>). <\/span>\n  \n* Using the compiler option \/Zc:wchar_t- with the filesystem library triggered linker errors (DevDiv#1004799).  \/Zc:wchar_t- is an abomination, but we grudgingly support it. <\/span>\n  \n* rename(“test.txt”, “test.txt”) is required to be a no-op, but we were deleting the file (DevDiv#1066931\/Connect#1006361<\/span><\/a>). <\/span>\n  \n* The last_write_time() getters and setters were truncating timestamps to whole seconds (DevDiv#1075324\/Connect#1018797<\/span><\/a>).  Now we’re preserving Windows’ 100 ns resolution. <\/span>\n  \n* The filesystem library has “throwers” like “bool create_directories(const path&)” and “non-throwers” like “bool create_directories(const path&, error_code&) noexcept”.  Several non-throwers were mistakenly implemented by calling throwers without any try\/catch logic (DevDiv#1180290\/Connect#1385031<\/span><\/a>).  This would lead to exceptions slamming into noexcept and triggering program termination.  We’ve fixed all known occurrences of this problem by having the non-throwers call other non-throwers.  In one case, the thrower is the core implementation, and the non-thrower wraps it in try\/catch.  (Note that there are several remaining occurrences of non-throwers calling throwers, but they’re innocuous because the “throwers” involved can’t actually fail in our implementation.  We may rework this in the future, but there was no time for VS 2015 RTM, as this was literally the last STL fix checked in.) <\/span>\n  \n* copy_file(src_path, dest_path, copy_options::update_existing) (optionally with an error_code) is a request to copy\/overwrite src_path to dest_path if and only if src_path is more recently modified (i.e. “newer overwrites older”).  Due to a reversed timestamp test, our implementation believed that update_existing meant “older overwrites newer”. <\/span>\n  \n* last_write_time(path, code), remove_all(path, code), and temp_directory_path(code) had incorrect return values for failure. <\/span>\n  \n* remove_all() previously returned either 0 or 1.  Now it correctly returns the number of files removed. <\/span>\n  \nVisualizer Improvements <\/strong><\/span>\n  \n* shared_ptr\/weak_ptr’s visualizers now display the original pointer stored in the control block (DevDiv#497336).  For example, this matters when shared_ptr<Derived> is converted to shared_ptr<Base>, or shared_ptr<Anything> is converted to shared_ptr<void>.  The control block remembers the original Derived * or Anything *. <\/span>\n  \n* We now visualize exception_ptr (DevDiv#716887).  This required magical custom machinery, implemented by Lukasz Chodorski and Eric Feiveson. <\/span>\n  \n* We now visualize atomic and atomic_flag (DevDiv#756373\/Connect#796725<\/span><\/a>). <\/span>\n  \n* reverse_iterator’s visualizer was technically correct but endlessly confusing (DevDiv#950042\/Connect#879130<\/span><\/a>).  Now it has special cases for vector\/array\/list iterators and raw pointers.  Also, the general case now displays the “current” data member as “base()” (which is the member function to retrieve it) and has a Synthetic child explaining “NOTE: *ri is equivalent to *prev(ri.base())”.  This should prevent further confusion. <\/span>\n  \n* We now visualize recursive_mutex (DevDiv#992590\/Connect#926641<\/span><\/a>). <\/span>\n  \n* We now visualize future and promise (DevDiv#1014393). <\/span>\n  \n* We now visualize initializer_list and chrono::duration. <\/span>\n  \n* The visualizers for tree iterators (map\/multimap\/set\/multiset) now detect end iterators. <\/span>\n  \n<random> Fixes <\/strong><\/span>\n  \n* mt19937_64 is supposed to take 64-bit seeds, but it was truncating them to 32-bit (DevDiv#619912\/Connect#779231<\/span><\/a>). <\/span>\n  \n* The output of piecewise_linear_distribution was completely incorrect (DevDiv#665556\/Connect#783628<\/span><\/a>). <\/span>\n  \n* The output of generate_canonical() was completely incorrect (DevDiv#844616\/Connect#811611<\/span><\/a>). <\/span>\n  \n* The output of binomial_distribution was incorrect for some parameters like (100, 0.991) (DevDiv#940047, DevDiv#1188781\/Connect#1444620<\/span><\/a>). <\/span>\n  \n* Distributions and their param_types are supposed to have matching constructors including explicitness, but many differed (DevDiv#977224, DevDiv#1076782\/Connect#1021566<\/span><\/a>). <\/span>\n  \nIostream\/Locale Fixes <\/strong><\/span>\n  \n* put_time() incorrectly treated %z (time zone offset) as a synonym of %Z (time zone name\/abbreviation) (DevDiv#821666\/Connect#808156<\/span><\/a>).  This was actually a bug in the CRT’s strftime(), which was fixed by James McNellis. <\/span>\n  \n* Parsing floating-point with iostream’s operator>>() had inaccurate results (DevDiv#938627\/Connect#866093<\/span><\/a>, DevDiv#961116\/Connect#890195<\/span><\/a>).  We’ve substantially improved the correctness here, although we’re still getting a few corner cases wrong, and there are significant performance issues.  In the next major version, we’re planning to switch back to the CRT for floating-point parsing, which has been overhauled for complete correctness (and is much faster than iostream’s separate codepaths).  Note that in VS 2015, the stod()\/to_string() family has been rewritten to use the CRT as required by the Standard. <\/span>\n  \n* We fixed a couple of read overruns in iostream’s floating-point and integer parsing (DevDiv#900436\/Connect#829931<\/span><\/a>, DevDiv#911845\/Connect#840103<\/span><\/a>). <\/span>\n  \n* collate::transform() misbehaved when given inputs consisting entirely of null characters, throwing an exception on x86 and allocating a huge string on x64 (DevDiv#869525\/Connect#814337<\/span><\/a>). <\/span>\n  \n* basic_istream::ignore(n, delim) was comparing n to INT_MAX, instead of numeric_limits<streamsize>::max() as required by the Standard (DevDiv#964171\/Connect#894605<\/span><\/a>).  This was a behavioral problem, not cosmetic (we have 32-bit int and 64-bit streamsize). <\/span>\n  \n* time_get::do_get_year() thought that the world will end in 2035 (DevDiv#640278).  Old behavior: <\/span>\n  <\/p>\n