The case of the oplock deadlock poppycock

A customer reported that their application would sometimes hang trying to open a file. The call to Create­File would simply hang, rather than failing with ERROR_SHARING_VIOLATION or some other error code.

One case where a Create­File can hang is when there is an oplock on the file that needs to be broken.

Explorer takes an oplock on files when generating thumbnails. This allow Explorer to be notified when another application wants access to the file in a manner that conflicts with the the thumbnail extractor, so it can abandon the thumbnail operation and close the file, thereby permitting the other application’s Create­File call to succeed. If Explorer hadn’t used an oplock, then an application that wants to access the file would get an immediate sharing violation.

The thread that owns the oplock is stuck here:

kernel32!CreateFileW+0x379
ole32!CFileStream::Init_OpenOrCreate+0x10d
ole32!CFileStream::InitWorker+0xb5
ole32!DfFromName+0x8f
ole32!DfOpenDocfile+0x1f2
ole32!DfOpenStorageEx+0x183
ole32!StgOpenStorageEx+0x9d
contoso!DllUnregisterServer+0xa9c0
SHELL32!CIconAndThumbnailOplockWrapper::Extract+0x14
SHELL32!CShellItem::_GetThumbnail+0x6f
... and so on ...

The shell took an oplock on the file and then called the thumbnail provider to extract the thumbnail. The thumbnail provider called back into Stg­Open­Storage­Ex, presumably to open the file in order to get the thumbnail. But that open call also hung.

The thumbnail extractor passed these flags to Stg­Open­Storage­Ex:

• STGM_READ­WRITE
• STGM_SHARE_DENY_WRITE
• STGM_TRANSACTED

The interesting flag here is STGM_READ­WRITE. The oplock takes read access with deny-write, and the attempt to open the file for writing conflicts with the oplock. This causes the Create­File to block waiting for the oplock to be released. But the oplock can’t be released until the thumbnail extraction is complete. Deadlock.

The customer confirmed that changing STGM_READ­WRITE to STGM_READ fixes the problem. “But they say that they need write access.”

We never did learn why the customer requires write access in order to extract a thumbnail from a file. I mean, does that mean that they cannot extract thumbnails from read-only files, or from CD-ROM drives?

We strongly encouraged them to find a way to extract thumbnails that doesn’t require write access. But if that is impossible for some reason, they can work around it by indicating that their thumbnail extractor is not compatible with oplocks.

[HKEY_CLASSES_ROOT\CLSID\{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}]
NoOplock=REG_DWORD:1

For apps packaged with the Desktop Bridge, you can make this declaration in your manifest:

<desktop2:DesktopPreviewHandler ... NoOplock="true"/>

Correction: The declaration above is for preview handlers, not thumbnail handlers.