{"id":8083,"date":"2012-03-16T07:00:00","date_gmt":"2012-03-16T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2012\/03\/16\/memory-allocation-functions-can-give-you-more-memory-than-you-ask-for-and-you-are-welcome-to-use-the-freebies-too-but-watch-out-for-the-free-lunch\/"},"modified":"2012-03-16T07:00:00","modified_gmt":"2012-03-16T07:00:00","slug":"memory-allocation-functions-can-give-you-more-memory-than-you-ask-for-and-you-are-welcome-to-use-the-freebies-too-but-watch-out-for-the-free-lunch","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20120316-00\/?p=8083","title":{"rendered":"Memory allocation functions can give you more memory than you ask for, and you are welcome to use the freebies too, but watch out for the free lunch"},"content":{"rendered":"

\nMemory allocation functions like\nHeap­Alloc<\/code>,\nGlobal­Alloc<\/code>,\nLocal­Alloc<\/code>,\nand\nCo­Task­Mem­Alloc<\/code>\nall have the property that they can return more memory\nthan you requested.\nFor example, if you ask for 13 bytes, you may very well get a\npointer to 16 bytes.\nThe corresponding Xxx­Size<\/code> functions return the actual\nsize of the memory block,\nand you are welcome to use all the memory in the block up to the\nactual size (even the bytes beyond the ones you requested).\nBut watch out for the free lunch.\n<\/p>\n

\nConsider the following code:\n<\/p>\n

\nBYTE *GetSomeZeroBytes(SIZE_T size)\n{\n BYTE *bytes = (BYTE*)HeapAlloc(GetProcessHeap(), 0, size);\n if (bytes) ZeroMemory(bytes, size);\n return bytes;\n}\n<\/pre>\n
\nSo far so good.\nWe allocate some memory, and then fill it with zeroes.\nThat gives us our zero-initialized memory.\n<\/p>\n
\nOr does it?\n<\/p>\n
\nBYTE *bytes = GetSomeZeroBytes(13);\nSIZE_T actualSize = HeapSize(GetProcessHeap(), 0, bytes);\nfor (SIZE_T i = 0; i < actualSize; i++) {\n assert(bytes[i] == 0); \/\/ assertion fires!?\n}\n<\/pre>\n
\nWhen you ask the heap manager for 13 bytes,\nit’s probably going to round that up to 16,\nand when you call\nHeap­Size<\/code>,\nit may very well say,\n“Hey, I gave you three extra bytes.\nDon’t need to thank me.”\n<\/p>\n
\nThe problem comes when you try to reallocate the memory:\n<\/p>\n
\nBYTE *ReallocAndZero(BYTE *bytes, SIZE_T newSize)\n{\n return (BYTE*)HeapReAlloc(bytes, GetProcessHeap(),\n                           HEAP_ZERO_MEMORY, newSize);\n}\n<\/pre>\n
\nHere, you said,\n“Dear heap manager,\nplease make this memory block bigger,\nand zero out the new bytes.\nKthxbai.”\nAnd, assuming the heap manager was successful,\nyou will indeed have a larger memory block,\nand the new bytes will have been zeroed out.\n<\/p>\n
\nBut the memory manager won’t zero out the three bonus\nbytes it gave you when you called\nHeap­Alloc<\/code>,\nbecause those bytes aren’t new.\nIn fact, the heap manager assumes that you knew about\nthose three extra bytes and were actively using them,\nand it would be rude to zero out those bytes behind your back.\n<\/p>\n
\nThose bytes you didn’t know about since you didn’t check.\n<\/p>\n
\nYou might think the problem is that you mixed zero-allocation modes.\nYou allocated the memory as\n“Go ahead and give me garbage,\nI’ll zero it out myself”,\nand then you reallocated it as\n“Can you zero it out for me?”\nThe problem is that you and the heap manager\ndisagree on how big it<\/i> is.\nWhile you assume that the size of it<\/i> is\n“the exact number of bytes I asked for”,\nthe heap manager assumes that the size of it<\/i> is\n“the exact number of bytes I gave you.”\nThose bytes in the middle fall through the cracks.\n<\/p>\n
\nTherefore, you might\ntry to fix it by changing your function like this:\n<\/p>\n
\nBYTE *ReallocAndZero(BYTE *bytes, SIZE_T newSize)\n{\n SIZE_T oldSize = HeapSize(GetProcessHeap(), bytes);\n BYTE *newBytes = (BYTE*)HeapReAlloc(bytes, GetProcessHeap(),\n                                     0, size);\n if (newBytes && newSize > oldSize) {\n  ZeroMemory(newBytes + oldSize, newSize - oldSize);\n }\n return newBytes;\n}\n<\/pre>\n
\nBut this doesn’t work, because of the reason\nwe gave above:\nYour call to\nHeap­Size<\/code> will return the\nactual<\/i> block size, not the requested size.\nYou will therefore forget to zero out those three\nbytes you didn’t know about.\n<\/p>\n
\nThe real problem is in the\nGet­Some­Zero­Bytes<\/code>\nfunction.\nIt decided to manually zero out the bytes it received,\nbut it zeroed out only the bytes that were requested,\nnot the actual bytes received.\n<\/p>\n
\nOne solution is to make sure to zero out everything<\/i>,\nso that if it is reallocated, the extra bytes gained in the\nreallocation will also be zero.\n<\/p>\n
\nBYTE *GetSomeZeroBytes(SIZE_T size)\n{\n BYTE *bytes = (BYTE*)HeapAlloc(GetProcessHeap(), 0, size);\n if (bytes) ZeroMemory(bytes,\n                       HeapSize(GetProcessHeap(), bytes))<\/font>;\n return bytes;\n}\n<\/pre>\n
\nAnother solution is to take advantage of the memory manager’s\nHEAP_ZERO_MEMORY<\/code> flag,\nwhich tells the memory manager to zero out the entire block<\/i>\nof memory when it is allocated:\n<\/p>\n
\nBYTE *GetSomeZeroBytes(SIZE_T size)\n{\n return (BYTE*)HeapAlloc(GetProcessHeap(),\n                         HEAP_ZERO_MEMORY, size);\n}\n<\/pre>\n
\n… and to use the same flag when reallocating:\n<\/p>\n
\nBYTE *ReallocAndZero(BYTE *bytes, SIZE_T newSize)\n{\n return (BYTE*)HeapReAlloc(bytes, GetProcessHeap(),\n                           HEAP_ZERO_MEMORY, size);\n}\n<\/pre>\n
\nMost of the heap functions let you specify that you\nwant the heap manager to zero out the memory for you,\nand that includes the bonus bytes.\nFor example,\nyou can use\nGMEM_ZERO­INIT<\/code> with the\nGlobal­Alloc<\/code> family of functions,\nand\nLMEM_ZERO­INIT<\/code> with the\nLocal­Alloc<\/code> family of functions.\nThe annoying one is\nCo­Task­Mem­Alloc<\/code>,\nsince it does not provide a flag for zero-allocation.\nYou have to zero out the memory yourself,\nand you have to do it right.\n(The inspiration for today’s article was a bug caused by\nnot zeroing out the memory correctly.)<\/p>\n
\nThere are other implications of these bonus bytes.\nFor example,\nif you use\nCreate­Stream­On­HGlobal<\/code>\nto create a stream on an existing HGLOBAL<\/code>,\nthe function uses\nGlobal­Size<\/code> to determine the size of the\nstream it should create.\nAnd that value includes the bonus bytes,\neven though you may not have realized that\nthey were there.\nResult: You create a stream of 13 bytes,\nbut somebody who tries to read from it will get 16 bytes.\nYou need to make sure that the code which reads from the stream\nwon’t get upset by those extra bytes.\n(For example, if you passed it to a function that concatenates\nstreams, you just inserted three bytes of garbage between the streams.)\nYou also need to be careful that those extra bytes don’t leak\nany sensitive information if you, say, put the memory block\non the clipboard for everyone to see.\n<\/p>\n
\nBonus chatter<\/b>:\nIt appears that at some point, the kernel folks decided that\nthese “bonus bytes” were more hassle than they were worth,\nand now they spend extra effort remembering not only the actual\nsize of the memory block but also the requested size.\nWhen you ask, “How big is this memory block?”\nthey lie and return the requested size rather than the actual size.\nIn other words, the free bonus bytes are no longer exposed\nto applications by the kernel heap functions.\nNote, however, that\nthis behavior is not contractual<\/i>;\nfuture versions of Windows may start handing out free bonus bytes again.\nNote also that not all heap managers have done the extra\nwork to remember the requested size,\nand they will continue to hand out bonus bytes.\nTherefore,\nyou must continue to code defensively and assume that\nbonus bytes may exist (even if they usually don’t).\n(And note that heap debugging tools may intentionally\ngenerate “bonus bytes” to help flush out bugs.)\n<\/p>\n
\nDouble extra bonus chatter<\/b>:\nNote that this gotcha is not specific to Windows.\n<\/p>\n
\n\/\/ resize a block of memory originally allocated by calloc\n\/\/ and zero out the new bytes\nvoid *crealloc(void *bytes, size_t new_size)\n{\n size_t old_size = malloc_size(bytes);\n void *new_bytes = realloc(bytes, new_size);\n if (new_bytes && new_size > old_size) {\n  memset((char*)new_bytes + old_size, 0, new_size - old_size);\n }\n return new_bytes;\n}\n<\/pre>\n
\nVirtually all heap libraries have bonus bytes.<\/p>\n","protected":false},"excerpt":{"rendered":"
Memory allocation functions like Heap­Alloc, Global­Alloc, Local­Alloc, and Co­Task­Mem­Alloc all have the property that they can return more memory than you requested. For example, if you ask for 13 bytes, you may very well get a pointer to 16 bytes. The corresponding Xxx­Size functions return the actual size of the memory block, and you are […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-8083","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Memory allocation functions like Heap­Alloc, Global­Alloc, Local­Alloc, and Co­Task­Mem­Alloc all have the property that they can return more memory than you requested. For example, if you ask for 13 bytes, you may very well get a pointer to 16 bytes. The corresponding Xxx­Size functions return the actual size of the memory block, and you are […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/8083","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=8083"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/8083\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=8083"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=8083"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=8083"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}