{"id":103720,"date":"2020-05-01T07:00:00","date_gmt":"2020-05-01T14:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=103720"},"modified":"2020-05-16T09:03:53","modified_gmt":"2020-05-16T16:03:53","slug":"20200501-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20200501-00\/?p=103720","title":{"rendered":"Diagnosing a hang: Everybody stuck in Win­Http­Get­Proxy­For­Url<\/CODE>"},"content":{"rendered":"

A customer reported that their program eventually ground to a halt with over 750 threads stuck in Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>:<\/p>\n

ntdll!ZwWaitForSingleObject+0x14\r\nKERNELBASE!WaitForSingleObjectEx+0x8f\r\nwinhttp!OutProcGetProxyForUrl+0x160\r\nwinhttp!WinHttpGetProxyForUrl+0x349\r\ncontoso!submit_web_request+0x232\r\nntdll!TppWorkpExecuteCallback+0x35e\r\nntdll!TppWorkerThread+0x474\r\nkernel32!BaseThreadInitThunk+0x14\r\nntdll!RtlUserThreadStart+0x21\r\n<\/pre>\n
(I’ve simplified the stack trace for expository purposes.)<\/p>\n
What’s happening here is that you put some work on the thread pool, and that work called Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>. This function is synchronous, but it makes HTTP network requests which are asynchronous. To bridge the gap, the Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code> function performs a synchronous wait for the asynchronous work to complete.<\/p>\n
And my guess is that Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code> itself uses the thread pool to complete its asynchronous work.<\/p>\n
What’s happening is that the program flooded the thread pool with submit_<\/code>web_<\/code>request<\/code> work items. Those work items called Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>, which queues its own work item and waits for it to complete. But those work items can’t run because the thread pool’s threads are all busy handling submit_<\/code>web_<\/code>request<\/code> work items.<\/p>\n
Eventually, the thread pool may realize that it’s not making progress and spin up a new thread to deal with the work that has been piling up. Maybe that thread will finish the work begun by Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>, and that will allow one of the submit_<\/code>web_<\/code>request<\/code> threads to continue. Once that thread is finished with the Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code> work item, it will go pull another work item from the queue, and odds are that it’s going to get another submit_<\/code>web_<\/code>request<\/code> work item, so now we’re back where we started, except with one more stuck thread in the thread pool.<\/p>\n
If the submit_<\/code>web_<\/code>request<\/code> work items come in faster than Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code> can retire its own work items, the thread pool will fill up with threads blocked inside submit_<\/code>web_<\/code>request<\/code>, and eventually the thread pool will reach its thread limit, and everything stops.<\/p>\n
You’re basically starving the thread pool by hijacking it with requests that themselves require the thread pool. All of the thread pool threads are stuck handling your requests, and none are left to do the work that your requests generated.<\/p>\n
It’s like you have a lot of heavy equipment that you want to move, so you hire every moving company in the city to move them. Company A shows up, and they say, “Hm, this is too big for us to move by ourselves. Let me call Company B, maybe they can help us.” Company B says, “Sorry, I can’t help you now. I just got an order to move a heavy piece of equipment.” By starving out all of the available moving companies, you manage to prevent any of them from completing the job.<\/p>\n
I suspect that this system is running in a network environment where  WPAD<\/a> is slow, which makes Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>‘s work item take longer to finish its job, and that makes it more likely that submit_<\/code>web_<\/code>request<\/code> work items will arrive faster than Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code> work items can be retired.<\/p>\n
Now that we’ve diagnosed the problem, what can we do to fix it?<\/p>\n
One idea is to hire just one moving company and let them decide how many more moving companies they need. Put all your calls to submit_<\/code>web_<\/code>request<\/code> on a single thread and retire them one at a time. This clogs up just one thread, leaving the others available to assist. On the other hand, this means that the requests cannot be handled in parallel.<\/p>\n
A better fix is to change the way you use the thread pool so you don’t keep a thread hostage for a long time.<\/p>\n
I’m not an expert on Win\u00adHttp, but other people had some ideas on how to do this.<\/p>\n
You can switch to Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl\u00adEx<\/code>, which returns immediately and calls you back when it has an answer. The submit_<\/code>web_<\/code>request<\/code> function could call Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl\u00adEx<\/code> and return immediately. This releases the thread pool thread to do other work\u2014possibly even the work that Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl\u00adEx<\/code> needs to do in order to complete. When Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl\u00adEx<\/code> finishes its asynchronous work, it calls the callback, and the callback and do whatever work submit_<\/code>web_<\/code>request<\/code> was planning on doing after getting the proxy information.<\/p>\n
Basically, go asynchronous all the way. It’s not an unreasonable approach for this program, since the submit_<\/code>web_<\/code>request<\/code> itself models an asynchronous request: It initiates the request and will call some caller-provided callback with the response from the servber. Since it’s already behaving asynchronously, you may as well make it even more<\/i> asynchronous.<\/p>\n
Another suggestion was to skip Win\u00adHttp\u00adGet\u00adProxy\u00adFor\u00adUrl<\/code>\u00a0entirely and just pass the WIN\u00adHTTP_<\/code>ACCESS_<\/code>TYPE_<\/code>AUTOMATIC_<\/code>PROXY<\/code> flag to Win\u00adHttp\u00adOpen<\/code>. This defers the proxy work to the Win\u00adHttp\u00adOpen<\/code> function, and it can do that as part of its other asynchronous activities. This seems like a good idea because it gets you out of the proxy business entirely, and you still get the asynchronous behavior. It also gives you the satisfaction of fixing a bug by deleting code.<\/p>\n
The customer confirmed that switching to the WIN\u00adHTTP_<\/code>ACCESS_<\/code>TYPE_<\/code>AUTOMATIC_<\/code>PROXY<\/code> flag fixed the problem.<\/p>\n","protected":false},"excerpt":{"rendered":"
Putting on your thinking cap.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-103720","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Putting on your thinking cap.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/103720","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=103720"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/103720\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=103720"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=103720"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=103720"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}