The nNumberÂOfÂBytesÂToÂRead parameter to ReadÂFile is a 32-bit unsigned integer, which limits the number of bytes that could be read at once to 4GB. What if you need to read more than 4GB?
The ReadÂFile function cannot read more than 4GB of data at a time. At the time the function was originally written, all Win32 platforms were 32-bit, so reading more than 4GB of data into memory was impossible because the address space didn’t have room for a buffer that large.
When Windows was expanded from 32-bit to 64-bit, the byte count was not expanded. I don’t know the reason for certain, but it was probably a combination of (1)Â not wanting to change the ABI more than necessary, so that it would be easier to port 32-bit device drivers to 64-bit, and (2)Â having no practical demand for reading that much data in a single call.
You can work around the problem by writing a helper function that breaks the large read into chunks of less than 4GB each.
But reading 4GB of data into memory seems awfully unusual. Do you really need all of it in memory at once? Maybe you can just read the parts you need as you need them. Or you can use a memory-mapped file to make this on-demand reading transparent. (Though at a cost of having to deal with in-page exceptions if the read cannot be satisfied.)
@Raymond Or Microsoft could just write a wrapper around ReadFile (if they can't be bothered or if it isn't possible to implement a full 64-bit read) and name it ReadFile64 or ReadFileEx2 or something so everyone who needs it doesn't have to reinvent the wheel?
Perhaps something like this could work?
<code>
Note that it reads large files in chunks and fails if a chunk read fails (and it also doesn't return how many bytes have been read in either case, left as an exercise for the readers).
Identify a reasonable use case where reading a large file like this actually makes sense? Bear in mind that other options like memory mapped files already solve this problem, are more efficient and don’t require the machine to have at least 4 GB of available memory to do this anyway. Outside of a very specific type of Server app I cannot think of a single case where this would make sense. It seems overly wasteful of resources.
The fact that something can be done doesn’t mean it should be done.
@Danielix Klimax Video processing does not need to have the whole file in memory. Video is processed in a frame-by-frame basis. You could argue that it would be useful to load a whole chunk between two keyframes, but even that is rarely more than a few MB.
The same can be applied to every multi-GB format I can think of (database, data set, hi-res audio...). Even mapping the whole file in memory seems a waste of address space and resources (page table entries, on-demand I/O).
This is one of those issues where if you have to ask "what is the limit?" you...
Video processing (file or pipe).
From another angle, 4 GiB of data will take a long time to transfer; the fastest SSD I can find would take 250 milliseconds to read 4 GiB of data, while reading it over 400G Ethernet is going to take you on the order of 100 ms. Even just copying it around in DRAM is going to hurt you; a high end server CPU is able to do about 500 GiB/s memory throughput, so that's still 8 ms to read 4 GiB, or 16 ms to copy it.
You get 1 million clock cycles per gigahertz of CPU clock speed in...
So you are saying that reading an 8GB ISO will take what, 0.5 sec? And that's a problem how? I don't get the argument about clock cycles? It's not like CPU will be busy doing the reads -- there's DMA and queues for that which you set and forget until its done.
Also, what's the point in doing chunked read as you suggest if you want to say do a file checksum? If you read 8 GB file in 256 MB blocks and get to 51% and next read fails you just wasted a lot of compute cycles calculating hash of...
Hanging a process for 500 milliseconds is an eternity in the multi-gigahertz era. And if you are reading off a traditional hard drive (with big data volumes, it is reasonable: a 30 TB SSD is prohibitively expensive) things go south easily. With a typical 120 MB/s SATA hard drive, loading 8 GB takes 70 seconds (or 70,000 milliseconds). More than a minute hanged is more than enough for Windows to show the “This program does not respond” and the user to force-close it. I’d say you wouldn’t want that.
The point about CPU cycles is that you have a lot of CPU cycles to use up on each read, if you're working in 4 GiB chunks; the additional CPU cost of chunking your requests to the API into 4 GiB chunks (instead of issuing a single 20 GiB read to the OS) is negligible compared to the time (measured in CPU cycles) it takes to transfer 4 GiB of data.
In addition, all you're likely to be doing is causing the OS to do chunked reads for you; nothing I've got access to (NVMe, SCSI, or SATA) can actually do...
It strikes me that this could have been a very deliberate decision. My hunch is that Memory mapping a file is more efficient for large reads. So the goal here is to discourage people from doing something better handled using that. Obviously not having access to the NTKernel source this is wholly supposition however. It could go through the same routines.
I doubt it, probably there is some legacy 32-bit value path passing through the filesystem driver stack which is a limiting factor given how reading >4GB wasn’t possible on filesystems of the era when ReadFile/WriteFile were written.
Also, I am not getting why memory mapping would be more efficient. You still have to page it in to read it and you can get exceptions to boot. If you need to read >4GB and you have enough RAM then there’s no reason not to.
In vast majority of cases, user code is not going to process all of it at once and it is likely to be spaced out over time. The only exception I can see would be video processing.