November 17th, 2021

The mental model for StartThreadpoolIo

Raymond Chen
Raymond Chen

A customer was having trouble using asynchronous I/O with the Windows thread pool. Their proof-of-concept program was crashing once they issue their second write. Here’s a sketch:

auto io = CreateThreadpoolIo(fileHandle, callback, nullptr, nullptr);
StartThreadpoolIo(io);

OVERLAPPED pending[NUMBER] = {};

for (int i = 0; i < NUMBER; i++) {
    pending[i].Offset = offset[i];
    pending[i].OffsetHigh = 0;
    bool result = WriteFile(fileHandle, data[i], size[i],
            &bytesWritten, &pending[i]);

    if (!result && GetLastError() != ERROR_IO_PENDING) {
        CancelThreadpoolIo(io);
    }
}

They found that if NUMBER is 1, then everything works great. If NUMBER is greater than 1, then the first I/O completion is successful, but the second one crashes.

The confusion is over what StartThreadpoolIo does. The customer thought that it needed to be called once for each file handle. But really, it needs to be called once for each I/O operation. If you issue ten writes against a file handle, you need to call Start­Threadpool­Io ten times, once before each call.

The point of Start­Threadpool­Io is to tell the thread pool to prepare for an incoming completion against the file handle. If you issue an I/O without first preparing the thread pool, then the completion arrives and the thread pool doesn’t know what to do with it.

The fix is to move the call to Start­Threadpool­Io to immediately before issuing the I/O operation:

auto io = CreateThreadpoolIo(fileHandle, callback, nullptr, nullptr);
// StartThreadpoolIo(io); // from here

OVERLAPPED pending[NUMBER] = {};

for (int i = 0; i < NUMBER; i++) {
    pending[i].Offset = offset[i];
    pending[i].OffsetHigh = 0;
    StartThreadpoolIo(io); // to here
    bool result = WriteFile(fileHandle, data[i], size[i],
            &bytesWritten, &pending[i]);

    if (!result && GetLastError() != ERROR_IO_PENDING) {
        CancelThreadpoolIo(io);
    }
}

If you discover that the I/O won’t generate a completion after all (because it failed synchronously, or because it succeeded synchronously on a handle that is marked as FILE_SKIP_COMPLETION_PORT_ON_SUCCESS), then you need to call Cancel­Threadpool­Io to say, “Um, it looks like there won’t be any completion at all. Sorry.” That way, the thread pool knows that it shouldn’t be expecting one.

Category
Old New Thing
Topics
Code

Author

Raymond Chen

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

3 comments

Discussion is closed. Login to edit/delete existing comments.

Newest
Newest Popular Oldest
  • switchdesktopwithfade@hotmail.com

    CreateThreadpoolIo fails for me if called multiple times on the same handle even if I call CloseThreadpoolIo in between, which makes no sense to me at all. The only way I was able to deal with it was by never allowing loose handles to participate with the thread pool, and managing the entire handle lifetime in a stream wrapper. I call CreateThreadpoolIo once and only once, and StartThreadpoolIo on every individual operation.

  • Neil Rashbrook

    So in the FILE_SKIP_COMPLETION_PORT_ON_SUCCESS case I assume you would want to use result || instead.

  • Louis Wilson · Edited

    This is not related to today’s post, but I wanted to let you know that your RSS feed is broken. It does not include any posts more recent than 11 November.

    Edit: fixed now.

Read next