\nA customer found that a single-byte write was taking several seconds,\neven though the write was to a file on the local hard drive that was\nfully spun-up.\nHere’s the pseudocode:\n<\/p>\n
\n\/\/ Create a new file - returns quickly\nhFile = CreateFile(..., CREATE_NEW, ...);\n\/\/ make the file 1GB<\/a>\nSetFilePointer(hFile, 1024*1024*1024, NULL, FILE_BEGIN);\nSetEndOfFile(hFile);\n\/\/ Write 1 byte into the middle of the file\nSetFilePointer(hFile, 512*1024*1024, NULL, FILE_BEGIN);\nBYTE b = 42;\n\/ this write call takes several seconds!\nWriteFile(hFile, &b, &nBytesWritten, NULL);\n<\/pre>\n\nThe customer experimented with using asynchronous I\/O,\nbut it didn’t help.\nThe write still took a long time.\nEven using
FILE_FLAG_NO_BUFFERING<\/code>\n(and\n\nwriting full sectors, naturally<\/a>)\ndidn’t help.\n<\/p>\n\nThe reason is that on NTFS, extending a file reserves disk space but\ndoes not zero out the data.\nInstead, NTFS keeps track of the “last byte written”,\ntechnically known as the\nvalid data length<\/i>,\nand only zeroes out up to that point.\nThe data past the valid data length\nare logically zero but are not\nphysically zero on disk.\nWhen you write to a point past the current valid data length,\nall the bytes between the valid data length and the start of\nyour write need to be zeroed out before the new valid data length\ncan be set to the end of your write operation.\n(You can manipulate the valid data length directly with\n\nthe
SetFileValidData<\/code> function<\/a>,\nbut be very careful since it comes with serious security implications.)\n<\/p>\n\nTwo solutions were proposed to the customer.\n<\/p>\n
\nOption 1 is to force the file to be zeroed out immediately\nafter setting the end of file by writing a zero byte to the end.\nThis front-loads the cost so that it doesn’t get imposed on\nsubsequent writes at seemingly random points.\n<\/p>\n
\nOption 2 is to\n\nmake the file sparse<\/a>.\nMark the file as sparse with the\n\n
FSCTL_SET_SPARSE<\/code> control code<\/a>,\nand\nimmediately after setting the end of file,\nuse\n\ntheFSCTL_SET_ZERO_DATA<\/code> control code<\/a>\nto make the entire file sparse.\nThis logically fills the file with zeroes without committing physical\ndisk space.\nAnywhere you actually write gets converted from “sparse” to “real”.\nThis does open the possibility that a later write into the middle\nof the file will encounter a disk-full error,\nso it’s not a “just do this and you won’t have to worry about\nanything” solution,\nand depending on how randomly you convert the file from “sparse”\nto “real”, the file may end up more fragmented than it would have\nbeen if you had “kept it real” the whole time.<\/p>\n","protected":false},"excerpt":{"rendered":"A customer found that a single-byte write was taking several seconds, even though the write was to a file on the local hard drive that was fully spun-up. Here’s the pseudocode: \/\/ Create a new file – returns quickly hFile = CreateFile(…, CREATE_NEW, …); \/\/ make the file 1GB SetFilePointer(hFile, 1024*1024*1024, NULL, FILE_BEGIN); SetEndOfFile(hFile); \/\/ […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-9573","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
A customer found that a single-byte write was taking several seconds, even though the write was to a file on the local hard drive that was fully spun-up. Here’s the pseudocode: \/\/ Create a new file – returns quickly hFile = CreateFile(…, CREATE_NEW, …); \/\/ make the file 1GB SetFilePointer(hFile, 1024*1024*1024, NULL, FILE_BEGIN); SetEndOfFile(hFile); \/\/ […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/9573","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=9573"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/9573\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=9573"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=9573"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=9573"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}