WPR fails to start, insufficient system resources?

Sowoon

From time to time, I get questions about insufficient system resource error (0x800705aa) when starting the trace using WPR. The error can be frustrating, especially when there is enough memory and storage space left on the system. Some people try to solve the issue by increasing the system resources such as killing some apps and services. It does not help. I agree. It is frustrating to see generic error messages like this which can mean many things. Let us find out what causes this error and how to resolve the situation.

Cases where WPR throws ERROR_NO_SYSTEM_RESOURCES error

This insufficient system resource error (0x800705aa) happens in two scenarios. The error can happen when we start the trace, or when we stop the trace.

c:\wpt>wpr -start GeneralProfile -start PerfTest.wprp
        Insufficient system resources exist to complete the requested service.
        Profile Id: GeneralProfile.Verbose.Memory, BootPerf.Verbose.Memory
        Error code: 0x800705aa
        Insufficient system resources exist to complete the requested service.

c:\wpt>wpr -stop test.etl
Press Ctrl+C to cancel the stop operation.
  0%  [                                                            ]
        Insufficient system resources exist to complete the requested service.
       Error code: 0x800705aa

What is causing the insufficient system resource(0x800705aa) error?

The error is returned by ETW API StartTrace(). MSDN actually lists out the common error codes of StartTrace API and one of them is ERROR_NO_SYSTEM_RESOURCES. The doc explains “The maximum number of logging sessions on the system has been reached. No new loggers can be created until a logging session has been stopped. This value defaults to 64 on most systems.

In this case “system resources” are ETW sessions. I have mentioned many times that ETW sessions are system resources, and they are expensive. By default, you can run up to 64 sessions simultaneously. Though the maximum number of the ETW sessions is configurable through the registry settings (also documented in the same section), increasing this limit only hides the real issue.

Then why would ‘wpr -stop’ command fail with ERROR_NO_SYSTEM_RESOURCES? That is because WPR does start another session to log extra events as well as the rundown session as necessary while saving the trace.

What you can do about it

Check how many sessions are running on the system

When you see Insufficient system resource error like above, the first thing you want to do is find out how many sessions are currently running. I introduced multiple ways to enumerate ETW sessions in the previous post. If you do ‘xperf -loggers’ and the number of running sessions is about 50 ~ 55, it is very likely that you are maxed out on the ETW sessions. Even though the number of sessions is less than 64, chances are the system reached the limit because tools like xperf do not show all the sessions due to the access issue(ACL).

Reduce unnecessary sessions

First, you should review your custom profile and find out how many sessions you are starting. The number of sessions is equal to the number of collectors nested within the chosen <profile>. Authoring custom profiles – part 1 explains about these elements. There is no need start multiple event sessions unless there are specific needs. Try merging multiple event sessions into one. Also, see If you have stale/unnecessary sessions still running around and stop them.

If reducing your own sessions does not apply or work, you can try to kill other sessions. Identifying what session to stop is the question here. There is no good answer but at least you can see the basic information (session properties) such as the session names, providers they are collecting events from and their logging modes. Use your judgement and caution in picking the candidate.

Once you identified the session to stop, stopping session is as easy as just running “xperf -stop sessionname”. Or you can use Computer Management control panel and stop a few sessions by right click -> Stop to resolve the imminent problem. If the problem comes back after the reboot, then deleting autologgers from “Startup Event Sessions” is effective across the reboots.

Stopping a session in Computer Management
Stopping a session in Computer Management

Temporary options

I do not recommend these options since there can be negative impacts. If you choose to do either or both options, do it only temporarily and last resort only.

  • Turn off “Optional diagnostic data” in Diagnostics & feedback settings. Since the session may not run at the time, the option may or may not reduce the number of sessions.
  • Increase the maximum number (EtwMaxLoggers) of ETW session through the registry setting. The registry key and value are documented in MSDN by ERROR_NO_SYSTEM_RESOURCES return value.

Why so many sessions are running on my system?

Good question. If your system is running a maximum number of sessions, you should investigate why.

On freshly installed OS, the number of ETW sessions should be less than half of the maximum number(64). Possible reasons for the logging session on the system would be you or someone including your company/IT

  • Installed applications/drivers that start the logging sessions
  • Agreed with the telemetry data submission when you installed software including OS, apps, and drivers.
  • Pushed a enterprise group policy to run logging sessions.
  • Started logging sessions for performance investigation, test, etc.

I expect reaching the maximum number of logging sessions would be a rare issue unless you are in the environment where software tracing is common like mine.

Software and diagnostics

Sometimes when you install a driver or application, and it asks for your agreement to provide feedback and diagnostic data. Among many ways to collect diagnostic data, the ETW logging session is one of the common ways. Without doubt, feedback and diagnostic are beneficial technologies for both users and software providers when used right.

The license agreement covers what data is collected and options to be in/out. You should be mindful about what you agree with and make good decisions. For information about how Windows diagnostic data is used as an example, see Diagnostics, feedback, and privacy in Windows 10.

To developers creating ETW sessions

If you are a developer who incorporates the ETW logging session in your software, it is always a good idea to keep the number of logging sessions minimal. Try merging multiple sessions into one. If you develop a user mode application and do not need to listen to your events outside of your application, consider using a private session that runs in the same process as its event trace providers. The private session does not count against the maximum session count.

Closing

WPR returns, or more precisely StartTrace API returns, the insufficient system resource error(ERROR_NO_SYSTEM_RESOURCES) when the system reached the maximum number of logging sessions. We have looked into what can cause the max logging sessions and what to do about it. Although Insufficient system resource error (0x800705aa) is not a common error, it can happen in an environment such as software OS testing. I hope this article to help you review ETW sessions that are running on your system.

 

2 comments

Comments are closed. Login to edit/delete your existing comments

  • Alois Kraus

    This can happen frequently when one opens 4-5 Visual Studio instances which all create some ETW sessions when one is e.g. debugging an application. Each VS instance where a debugger is used I get up to 7 ETW sessions which pretty quickly gets to that point. When I then want to use wpr or xperf then I get this cryptic error message which is indeed misleading. I have found that when using .NET Core applications I do not get JITed call stacks at all. Even when I do wpr -start CPU -start dotnet it is not able to collect stack traces from .NET Core processes. The setting used by WPR is “.NET Common Language Runtime”:0x20098:0x5 which looks good to me. But for some reason for .NET Core processes now ETW events show up from .NET Core processes. Is there some filtering during stop and merge happening inside wpr? The one thing which is different is that .NET Core processes have no clr.dll but coreclr.dll is wpr doing some magic there?

    • Sowoon PyoMicrosoft employee

      No WPR does not filter any events during stop or merging. I am not familiar with .NET providers but if “.NET Common Language Runtime”:0x20098:0x5″ should show you the events you are looking for, then you need to turn on the stack attribute for the provider. The built-in DotNet profile enables two providers with stacks and others without stacks.
      wpr -profiledetails DotNet

      Collector Name : WPR_initiated_WprApp_DotNETRuntime_Session
      Buffer Size (KB) : 1024
      Number of Buffers : 160
      Providers
      2e5dba47-a3d2-4d16-8ee0-6671ffdcd7b5: 0xffffffff: 0x05 : Stack
      47c3ba0c-77f1-4eb0-8d4d-aef447f16a85: 0x4007ccbd: 0x05 : Stack

      763fd754-7086-4dfe-95eb-c01a46faf4ca: 0x1: 0x04
      aa087e0e-0b35-4e28-8f3a-440c3f51eef1: 0xffffffff: 0x05
      aff081fe-0247-4275-9c4e-021f3dc1da35: 0xffffffff: 0x05
      de4649c9-15e8-4fea-9d85-1cdda520c334: : 0xff
      e13b77a8-14b6-11de-8069-001b212b5009: 0xffffffff: 0x11
      Microsoft-Windows-DotNETRuntime: 0x4007ccbd: 0x05
      CaptureState Providers on Save
      Microsoft-Windows-DotNETRuntime: 0x70138: 0x05
      Microsoft-Windows-DotNETRuntimeRundown: 0x70138: 0x05