Renaming a file is a multi-step process, only one of which is changing the name of the file
A customer reported that the
ReadDirectoryChangesW function was reporting changes too soon. No, it wasn’t generating changes from the future, à la Minority Report. Rather, it generated rename notifications before the rename was complete.
The customer came to this conclusion because they observed their program behaving like this:
|Thread 1||Thread 2|
|Tries to read the renamed file and gets
|Tries to read the renamed file and succeeds.|
ReadDirectoryChangesW function reports the rename before the
MoveFileEx function returns, and consequently before the rename has completed.
What’s going on here?
Well, the first thing to observe is that the customer’s conclusion doesn’t match the evidence. Observe that the attempt to open the renamed file failed with
ERROR_, whereas they expected error would be
ERROR_ if the file hadn’t been renamed yet. The fact that they’re getting
ERROR_ means that the rename really did occur, but they are unable to access the renamed file due to a sharing violation.
Okay, let’s look at how renaming a file is performed internally. It’s a multi-step operation.
- Open the file with
- Close the handle.
DELETE permission grants permission to rename the file. The required permission is
DELETE because the old name is being deleted.
The call with
FileRenameInformation is what actually renames the file, and it is here that the
ReadDirectoryChangesW is signaled.
Now that the rename is complete, the handle can be closed.
It is technically correct for the
ReadDirectoryChangesW to be signaled once the
NtSetInformationFile is done, because the file is well and truly renamed.
Let’s look at that sharing violation again. The customer explained that they tried to open the file by doing this:
std::ifstream file(path, std::ios::binary, _SH_DENYNO);
_SH_ indicates that no sharing operations are denied. So why is sharing denied?
You were faked out by a flag name that makes sense in context, but has ended up being confusing due to the passage of time.
Let’s look at those sharing flags in context:
||Deny read, allow write.||Deny read.|
||Allow read, deny write.||Deny write.|
||Deny read, deny write.||Deny read and write.|
||Allow read, allow write.||Deny none.|
The mnemonic for
_SH_ is “Deny none”, but the word “none” is only with the context of read and write. You could say that it denies neither country nor western.
The important sharing mode here is neither read nor write. It’s
FILE_, which means “I’m okay with letting someone delete or rename the file while I have it open.”¹ This is a sharing flag that programs really should be using more often than they do, and the fact that the C runtime doesn’t give you an easy way to set this sharing flag may be a contributing factor.
If you call the
CreateFile function directly, then you can pass the
FILE_ sharing flag, and then you’ll be able to open the file even before
MoveFileEx cleans up its handle.
“So why not have
ReadDirectoryChangesW wait until the handle is closed before raising the rename notification?”
Well, for one thing, the file really has been renamed as soon as the
NtSetInformationFile is complete, so delaying the notification would be a little disingenuous. But seeing as it’s just a small delay, maybe that’s okay, seeing as the whole thing is a notification anyway, and notifications can be delayed for other reasons.
But the real reason is that delaying the notification until the close of the handle could delay it indefinitely. The caller is not required to close the handle immediately after the
NtSetInformationFile returns. It could leave the handle open so it can perform other operations on the file. For example, maybe it’s a log file that is being renamed while it is still being actively written to. That log file’s new name takes effect immediately, but the handle won’t be closed for a long time yet.
The customer confirmed that switching to a direct
FILE_ allowed them to open and read the file immediately after it was renamed.
Moral of the story: Don’t forget
FILE_. It lets you coexist with code that is deleting or renaming the file you are looking at.
¹ My colleague Malcolm Smith, whom I rely on for all things filesystem, notes that the name
FILE_ is rather misleading. Because the fact that you opened the file for
FILE_ prevents it from being deleted, even though you’re allowing it! In Windows, when you mark a file for deletion, the deletion doesn’t take effect until all outstanding handles are closed, and holding a file open for
FILE_ means that the last handle isn’t closed yet. What
FILE_ does is allow the file to be opened by others in
DELETE mode, which as it happens is a prerequisite for both deleting and renaming files.
> the fact that the C runtime doesn’t give you an easy way to set this sharing flag may be a contributing factor.
Yup. That’s pretty much a bug that’s so old it’s risky to fix, but not fixing it causes chronic problems. Almost nobody sets FILE_SHARE_DELETE when they should; and others spend disproportionate effort dealing with this.
I finally rebuilt the “unlocker” tool of infamy but this time I did it right; it tracks down the handles in processes; suspends the processes, closes the handles, opens handles to NUL with the same handle ids and resumes the processes. To be fair, this was deployed against something that was leaking handles and not something that didn’t use FILE_SHARE_DELETE when it should.
Another reason it is not common, Win95 did not allow FILE_SHARE_DELETE.
The documentation for this is actually really misleading, from your comment “If you call the CreateFile function directly, then you can pass the FILE_SHARE_DELETE sharing flag, and then you’ll be able to open the file even before MoveFileEx cleans up its handle” it implies that thread2 has to use FILE_SHARE_DELETE but the CreateFile docs seem to say, via “Enables subsequent open operations on a file or device to request delete access”, that thread1 has to use FILE_SHARE_DELETE. Am I misreading either your comment or the CreateFile docs? They seem to be saying the opposite thing.
You seem to have overlooked the previous paragraph, “If this flag is not specified, but the file or device has been opened for delete access, the function fails.” Thread 2 has to use
FILE_SHARE_DELETEbecause thread 1 opened the file with