August 4th, 2025
likeintriguing2 reactions

How can I read more than 4GB of data from a file in a single call to Read­File?

Raymond Chen
Raymond Chen

The nNumber­Of­Bytes­To­Read parameter to Read­File is a 32-bit unsigned integer, which limits the number of bytes that could be read at once to 4GB. What if you need to read more than 4GB?

The Read­File function cannot read more than 4GB of data at a time. At the time the function was originally written, all Win32 platforms were 32-bit, so reading more than 4GB of data into memory was impossible because the address space didn’t have room for a buffer that large.

When Windows was expanded from 32-bit to 64-bit, the byte count was not expanded. I don’t know the reason for certain, but it was probably a combination of (1) not wanting to change the ABI more than necessary, so that it would be easier to port 32-bit device drivers to 64-bit, and (2) having no practical demand for reading that much data in a single call.

You can work around the problem by writing a helper function that breaks the large read into chunks of less than 4GB each.

But reading 4GB of data into memory seems awfully unusual. Do you really need all of it in memory at once? Maybe you can just read the parts you need as you need them. Or you can use a memory-mapped file to make this on-demand reading transparent. (Though at a cost of having to deal with in-page exceptions if the read cannot be satisfied.)

Category
Old New Thing
Topics
Code

Author

Raymond Chen

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

All comments
Single comment
  • MGetz

    It strikes me that this could have been a very deliberate decision. My hunch is that Memory mapping a file is more efficient for large reads. So the goal here is to discourage people from doing something better handled using that. Obviously not having access to the NTKernel source this is wholly supposition however. It could go through the same routines.

    • ‪ ‪

      We recommend that you measure it ather than relying on intuition.

    • Igor Levicki · Edited

      I doubt it, probably there is some legacy 32-bit value path passing through the filesystem driver stack which is a limiting factor given how reading >4GB wasn’t possible on filesystems of the era when ReadFile/WriteFile were written.

      Also, I am not getting why memory mapping would be more efficient. You still have to page it in to read it and you can get exceptions to boot. If you need to read >4GB and you have enough RAM then there’s no reason not to.

      • Raymond ChenMicrosoft employee Author

        Adding to Kalle Niemitalo‘s remarks: The SMB and NFS protocols use 32-bit read/write lengths. (AFP supports 64-bit lengths, it seems.)

      • Kalle Niemitalo

        In the Windows kernel, read lengths are 32-bit ULONG in NtReadFile, IO_STACK_LOCATION, and FLT_PARAMETERS. Likewise MDL::ByteCount. However IO_STATUS_BLOCK uses ULONG_PTR and could support 64-bit lengths.

        I suppose it would be possible to define some IOCTL or FSCTL for reading larger amounts of data, without going through the normal IRP_MJ_READ. This would require a new SupportedFeatures bit so that the new interface is only used if all minifilters support it. But I doubt there is business justification for such a change.

      • Danielix Klimax

        In vast majority of cases, user code is not going to process all of it at once and it is likely to be spaced out over time. The only exception I can see would be video processing.

Read next