Measurements<\/strong> – The primary thing you are looking for with Stress testing are systemic failures. It’s not functional testing – you aren’t trying to verify the results of the tests. You don’t even care if the data was correct. I see people get very confused over this point. You have to remember that every kind of testing you do has a purpose. All testing does not test everything. Let functional testing do what it does (determine correctness of operations) and let stress testing do what it does (find rarely occurring “catastrophic” problems, leaks, etc).<\/p>\n In stress testing, you monitor:<\/p>\n Failure resolution<\/strong> – One thing to look out for with stress testing is bug handling. There is strong tendency to resolve bugs as “no repro” because, they don’t actually reproduce at will. You don’t want to let go of an occurrence until you are certain you’ve exhausted every possibility to identify the cause. If you see a pattern of stress bugs being resolved “no repro”, then you have a problem. I’ve generally found that problem to be one of two things – not enough understanding on the part of developers of the importance of pursuing the cause relentlessly or not enough instrumentation to help identify the cause. Attaching a debugger before the run starts can sometimes help. Sometimes, when we find stress bugs we can’t isolate, we add specific instrumentation to the code aimed at helping identify it and run with that in future runs.<\/p>\n Load profile<\/strong> – We pick a load (# of simulated users) that we know runs the server at about 70% utilization and run a constant load profile at that level. The goal is not to drive the system under test to saturation, but rather just to keep it very busy the whole time. Unlike Load testing, we don’t ramp the load up over time because we are not measuring the increasing response times – in Stress testing, we don’t care about the response times.<\/p>\n Frequency\/Duration<\/strong> – We run “short haul” and “long haul” stress testing. A short haul run is an 8 hour run and starts at night and completes by morning – ready for analysis. We generally run short haul runs on every day\/build. A long haul run is 120 hours (5 days). We generally run long haul runs on builds that have good short haul pass rates. The reason for both is that you need the cycle time of being able to fix bugs and pick up new build that you get with short haul runs. But certain kinds of problems (particularly resource leaks) don’t always show up in an 8 hour run and 120 hours helps. In the past we’ve done some math to equate how much simulated calendar time a long haul run (120 hours at high load) represents for a “typical” team. With a reasonable set of assumptions – it generally comes out to be months but I don’t obsess on that question. We’ve just determined 120 hours works out to be a good number and more than that doesn’t help much. When I worked on the .NET Framework team, we experimented with “ultra long haul testing” – running for 5 weeks but it didn’t yield much new information.<\/p>\n Development cycle<\/strong> – We generally start short haul stress testing early in the cycle. In my opinion, the earlier the better. The kinds of problems found in stress testing are hard to isolate and debug and the closer you find them to their introduction the better you are. We generally don’t start long haul testing until around our final Beta. There’s no point doing it until short haul is passing at a very high rate – because it’s just going to find the same problems. Because of the cycle time, anything you can find using short haul is better – you get to try fixes much faster.<\/p>\n Execution infrastructure<\/strong> – You don’t need particularly high end hardware. You just need something that can run your tests and achieve a solid load. Whereas in Load testing, we run the system under test “pristinely”, using separate clients for the load agents, in Stress testing it’s different. We generally run the load agent on the same physical machine as the code under test. You don’t care if the load on the server is affected by the test infrastructure and combining things allows you to conserve hardware and run more instances; remember – let each kind of testing do what it’s designed to.<\/p>\n For short haul testing, we generally have 10-15 “pods” (collections of machines) that run stress tests. For long haul testing, we generally use 3-5. There are a mix of topologies among the pods – single server\/dual server, domain\/workgroup, etc. I’ll be talking about test matrices in a future post in this series. You want to make sure you have plenty of redundancy in your execution infrastructure. There are a few reasons:<\/p>\n Misc<\/strong> – There is some debate about the value of running debug vs release builds. The advantage of debug builds is that you get the benefits of the asserts in your code. A\n disadvantage is that the debugging code (asserts, etc) can affect the timing and cause you to miss failures. I am a fan of running both debug and release builds – but opinions vary on this topic.<\/p>\n At the highest level, we track the overall stress runs build by build, noting # of runs, # of new bugs found, # of passes and tests per second. This gives the 10,000 foot view of how you are progressing…<\/p>\n We have daily reports that include more information on the runs of that day…<\/p>\n And the status of bugs that have recently been found by stress testing…<\/p>\n\n
\n
Stress Testing Reports<\/h3>\n
![](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D%5B7%5D.png\")
<\/a> <\/p>\n<\/a> <\/p>\n