{"id":11813,"date":"2011-01-06T07:00:00","date_gmt":"2011-01-06T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2011\/01\/06\/processes-commit-ram-threads-and-how-high-can-you-go\/"},"modified":"2011-01-06T07:00:00","modified_gmt":"2011-01-06T07:00:00","slug":"processes-commit-ram-threads-and-how-high-can-you-go","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20110106-00\/?p=11813","title":{"rendered":"Processes, commit, RAM, threads, and how high can you go?"},"content":{"rendered":"

\nBack in 2008,\nIgor Levicki\n\nmade a boatload of incorrect assumptions in an attempt to calculate\nthe highest a process ID can go on Windows NT<\/a>.\nLet’s look at them one at a time.\n<\/p>\n

\nSo if you can’t create more than 2,028 threads in one process\n(because of 2GB per process limit)\nand each process needs at least one thread,\nthat means you are capped by the amount of physical RAM available for stack.\n<\/p><\/blockquote>\n

\nOne assumption is that each process needs at least one thread<\/i>.\nReally?\nWhat about a process that has exited?\n(Some people call these zombie<\/i> processes.)\nThere are no threads remaining in this process,\n\nbut the process object hangs around until all handles are closed<\/a>.\n<\/p>\n

\nNext, the claim is that you are capped by the amount of physical\nRAM available for stack<\/i>.\nThis assumes that stacks are non-pageable,\nwhich is an awfully strange assumption.\nUser-mode stacks are most certainly pageable.\nIn fact, everything<\/i> in user-mode is pageable unless you take\nspecial steps to make it not pageable.\n<\/p>\n

\n

\nGiven that the smallest stack allocation is 4KB\nand assuming 32-bit address space:\n<\/p>\n

\n4,294,967,296 \/ 4,096 = 1,048,576 PIDs\n<\/p>\n<\/blockquote>\n

\nThis assumes that all the stacks live in the same address space,\nbut user mode stacks from different processes most certainly do not;\nthat’s the whole point of separate address spaces!\n(Okay, kernel stacks live in the same address space, but the discussion\nabout “initial stack commit” later makes it clear he’s talking about\nuser-mode stacks.)\n<\/p>\n

\n

\nSince they have to be a multiple of 4:\n<\/p>\n

\n1,048,576 \/ 4 = 262,144 PIDs\n<\/p>\n<\/blockquote>\n

\nIt’s not clear why we are dividing by four here.\nYes,\nprocess IDs are a multiple of four\n(implementation detail, not contractual, do not rely on it<\/a>),\nbut that doesn’t mean that three quarters of the stacks are no longer any good.\nIt just means that we can’t use more than 4,294,967,296\/4 of them\nsince we’ll run out of names after 1,073,741,824 of them.\nIn other words, this is not a division but rather a min<\/code> operation.\nAnd we already dropped below 1 billion when we counted kernel stacks,\nso this min<\/code> step has no effect.\n<\/p>\n

\nIt’s like saying, “This street is 80 meters long.\nThe minimum building line is 4 meters, which means that you can\nhave at most 20 houses on this side of the street.\nBut house numbers on this side of the street must be even,\nso the maximum number of houses is half that, or 10.”\nNo, the requirement that house numbers be even doesn’t cut the number\nof houses in half; it just means you have to be more careful how you\nassign the numbers.\n<\/p>\n

\nHaving 262,144 processes would consume 1GB of RAM\njust for the initial stack commit\nassuming that all processes are single-threaded.\nIf they commited 1MB of stack each you would need 256 GB of memory.\n<\/p><\/blockquote>\n

\nCommit does not consume RAM.\nCommit is merely a promise from the memory manager that the RAM will\nthere when you need it, but the memory manager doesn’t have to\nproduce it immediately (and certainly doesn’t have to keep the RAM\nreserved for you until you free it).\nIndeed, that’s the whole point of virtual memory,\nto decouple commit from RAM!\n(If commit consumed RAM, then what’s the page file for?)\n<\/p>\n

\nThis calculation also assumes that process IDs are allocated\n“smallest available first”, but it’s clear that it’s not as simple\nas that:\nFire up Task Manager and look at the highest process ID.\n(I’ve got one as high as 4040.)\nIf process IDs are allocated smallest-available-first, then a process ID\nof 4040 implies that at some point there were 1010 processes in the system\nsimultaneously!\nUnlikely.\n<\/p>\n

\nHere’s a much simpler demonstration that process IDs are not allocated\nsmallest-available-first:\nFire up Task Manager, tell it to Show processes from all users<\/i>,\ngo to the Processes tab, and\nenable the PID column if you haven’t already.\nNow launch Calc. Look for Calc in the process list and\nobserve that it was not assigned the lowest available PID.\nIf your system is like mine, you have PID zero assigned to the System\nIdle Process (not really a process but it gets a number anyway),\nand PID 4 assigned to the System process (again, not really a process\nbut it gets a number anyway), and then you have a pretty big gap before\nthe next process ID (for me, it’s 372).\nAnd yet Calc was given a process ID in the 2000’s.\nProof by counterexample that the system does not assign PIDs\nsmallest-available-first.\n<\/p>\n

\nSo if they aren’t assigned smallest-available-first,\nwhat’s to prevent one from having a process ID of 4000000000?\n<\/p>\n

\n(Advanced readers may note that kernel stacks do all share a single\naddress space, but even in that case, a thread that doesn’t exist\ndoesn’t have a stack.\nAnd it’s clear that Igor was referring to user-mode stacks since he\ntalked about 1MB stack commits, a value which applies to user mode\nand not kernel mode.)\n<\/p>\n

\nJust for fun, I tried to see how high I could get my process ID.\n<\/p>\n

\n#include <windows.h>\nint __cdecl _tmain(int argc, TCHAR **argv)\n{\n DWORD dwPid = 0;\n TCHAR szSelf[MAX_PATH];\n GetModuleFileName(NULL, szSelf, MAX_PATH);\n int i;\n for (i = 0; i < 10000; i++) {\n  STARTUPINFO si = { 0 };\n  PROCESS_INFORMATION pi;\n  if (!CreateProcess(szSelf, TEXT("Bogus"),\n        NULL, NULL, FALSE, CREATE_SUSPENDED, NULL, NULL,\n        &si, &pi)) break;\n  TerminateProcess(pi.hProcess, 0);\n  CloseHandle(pi.hThread);\n  \/\/ intentionally leak the process handle so the\n  \/\/ process object is not destroyed\n  \/\/ CloseHandle(pi.hProcess); \/\/ leak\n  if (dwPid < pi.dwProcessId) dwPid = pi.dwProcessId;\n }\n _tprintf(_TEXT("\\nCreated %d processes, ")\n          _TEXT("highest pid seen was %d\\n"), i, dwPid);\n _fgetts(szSelf, MAX_PATH, stdin);\n return 0;\n}\n<\/pre>\n
\nIn order to get the program to complete before I got bored,\nI ran it on a Windows 2000 virtual machine with 128MB of memory.\nIt finally conked out at 5245 processes with a PID high water mark\nof 21776.\nAlong the way, it managed to consume 2328KB of non-paged pool,\n36KB of paged pool, and 36,092KB of commit.\nIf you divide this by the number of processes, you’ll see that\na terminated process consumes about 450 bytes of non-paged pool,\na negligible amount of paged pool, and 6KB of commit.\n(The commit is probably left over page tables and other detritus.)\nI suspect commit is the limiting factor in the number of processes.\n<\/p>\n
\nI ran the same program on a Windows 7 machine with 1GB of RAM,\nand it managed to create all 10,000 processes with a high process ID\nof 44264.\nI cranked the loop limit up to 65535, and it still comfortably\ncreated 65535 processes with a high process Id of 266,232,\neasily exceeding the limit of\n262,144 that Igor calculated.\n<\/p>\n
\nI later learned that the Windows NT folks do try to keep\nthe numerical values of process ID from getting too big.\nEarlier this century,\nthe kernel team\nexperimented with letting the numbers get really huge,\nin order to reduce the rate at which process IDs get reused,\nbut they had to go back to small numbers, not for any technical\nreasons, but because people complained that the large process IDs\nlooked ugly in Task Manager.\n(One customer even asked if something was wrong with his computer.)\n<\/p>\n
\nThat’s not saying that the kernel folks won’t go back and try\nthe experiment again someday.\nAfter all, they managed to\n\nget rid of the dispatcher lock<\/a>.\nWho knows what other crazy things will change next?\n(And once they get process IDs to go above 65535—like they\nwere in Windows 95, by the way—or\nif they decided to make process IDs no\nlonger\n\nmultiples of 4<\/a>\nin order to keep process IDs low,\n\nthis guy’s program<\/a>\nwill stop working, and it’ll be Microsoft’s fault.)<\/p>\n","protected":false},"excerpt":{"rendered":"
Back in 2008, Igor Levicki made a boatload of incorrect assumptions in an attempt to calculate the highest a process ID can go on Windows NT. Let’s look at them one at a time. So if you can’t create more than 2,028 threads in one process (because of 2GB per process limit) and each process needs […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[26],"class_list":["post-11813","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-other"],"acf":[],"blog_post_summary":"
Back in 2008, Igor Levicki made a boatload of incorrect assumptions in an attempt to calculate the highest a process ID can go on Windows NT. Let’s look at them one at a time. So if you can’t create more than 2,028 threads in one process (because of 2GB per process limit) and each process needs […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/11813","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=11813"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/11813\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=11813"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=11813"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=11813"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}