The mental model for StartThreadpoolIo

Raymond

A customer was having trouble using asynchronous I/O with the Windows thread pool. Their proof-of-concept program was crashing once they issue their second write. Here’s a sketch:

auto io = CreateThreadpoolIo(fileHandle, callback, nullptr, nullptr);
StartThreadpoolIo(io);

OVERLAPPED pending[NUMBER] = {};

for (int i = 0; i < NUMBER; i++) {
    pending[i].Offset = offset[i];
    pending[i].OffsetHigh = 0;
    bool result = WriteFile(fileHandle, data[i], size[i],
            &bytesWritten, &pending[i]);

    if (!result && GetLastError() != ERROR_IO_PENDING) {
        CancelThreadpoolIo(io);
    }
}

They found that if NUMBER is 1, then everything works great. If NUMBER is greater than 1, then the first I/O completion is successful, but the second one crashes.

The confusion is over what StartThreadpoolIo does. The customer thought that it needed to be called once for each file handle. But really, it needs to be called once for each I/O operation. If you issue ten writes against a file handle, you need to call Start­Threadpool­Io ten times, once before each call.

The point of Start­Threadpool­Io is to tell the thread pool to prepare for an incoming completion against the file handle. If you issue an I/O without first preparing the thread pool, then the completion arrives and the thread pool doesn’t know what to do with it.

The fix is to move the call to Start­Threadpool­Io to immediately before issuing the I/O operation:

auto io = CreateThreadpoolIo(fileHandle, callback, nullptr, nullptr);
// StartThreadpoolIo(io); // from here

OVERLAPPED pending[NUMBER] = {};

for (int i = 0; i < NUMBER; i++) {
    pending[i].Offset = offset[i];
    pending[i].OffsetHigh = 0;
    StartThreadpoolIo(io); // to here
    bool result = WriteFile(fileHandle, data[i], size[i],
            &bytesWritten, &pending[i]);

    if (!result && GetLastError() != ERROR_IO_PENDING) {
        CancelThreadpoolIo(io);
    }
}

If you discover that the I/O won’t generate a completion after all (because it failed synchronously, or because it succeeded synchronously on a handle that is marked as FILE_SKIP_COMPLETION_PORT_ON_SUCCESS), then you need to call Cancel­Threadpool­Io to say, “Um, it looks like there won’t be any completion at all. Sorry.” That way, the thread pool knows that it shouldn’t be expecting one.

3 comments

Leave a comment

  • Louis Wilson

    This is not related to today’s post, but I wanted to let you know that your RSS feed is broken. It does not include any posts more recent than 11 November.

    Edit: fixed now.

  • switchdesktopwithfade@hotmail.com

    CreateThreadpoolIo fails for me if called multiple times on the same handle even if I call CloseThreadpoolIo in between, which makes no sense to me at all. The only way I was able to deal with it was by never allowing loose handles to participate with the thread pool, and managing the entire handle lifetime in a stream wrapper. I call CreateThreadpoolIo once and only once, and StartThreadpoolIo on every individual operation.