In the flags register of the 80386 processor is a flag called the interrupt flag. If the flag is set, then the CPU will respond to hardware interrupts. If the flag is clear, then the CPU will ignore hardware interrupts. It was common for a game that used an MS-DOS extender to disable interrupts temporarily by doing this:

    pushf   ; save flags (including whether interrupts are enabled)
    cli     ; disable interrupts
    ..do something..
    popf    ; restore original flags
            ; (this re-enables interrupts if interrupts were previously enabled)

There were other variations of this pattern, but most of them boiled down to the same basic idea.

One of the quirks of the 80386 architecture is that if the application does not have I/O privilege, the popf instruction does not restore the interrupt flag. Furthermore, the instruction doesn’t trap, so the operating system doesn’t know that the application tried and failed to change the interrupt flag. It just silently fails. This means that the above code fragment behaves differently depending on whether the application has I/O privilege. If it has I/O privilege, then the popf will restore interrupts to their previous state. But if it doesn’t, then the popf instruction has no effect on the interrupt flag, and interrupts remain disabled.

MS-DOS extenders typically granted the client I/O privilege because they were designed to run in a single-tasking environment, so the client was allowed full control of the system. Windows, on the other hand, did not grant the client I/O privilege because it had to worry about multitasking. This means that in Windows, programs that used the above technique would hang because they disabled interrupts and never re-enabled them.

The DPMI specification specifically calls out this coding pattern as problematic and provides special services for managing the interrupt flag so that a client application can manipulate its interrupt flag in a way that the DPMI server understands. In practice, however, client applications were written on the assumption that they were running under the MS-DOS extender that they were packaged with, and they took shortcuts and just used the popf instruction because they knew that it would work. It never occurred to them that their preferred DPMI extender would not actually be the one in charge.

Except, of course, that it didn’t work when the DPMI server was Windows.

One of my colleagues came up with a clever solution that addressed many cases of this problem. Since the cli instruction is privileged, it will trap. The trap handler for the cli instruction inspects the code preceding the cli instruction to see if it matches the pattern above or one of the common variations. If so, and the value on the top of the stack matches the virtual machine’s current flags, then it assumes that the instruction that most recently executed was in fact the pushf. In that case, it copies the pushed interrupt flag to the pushed trap flag. In other words, if interrupts were previously enabled, then set the trap flag in the pushed flags.

In the 80386 architecture, the trap flag causes the processor to raise a trap exception after one instruction is executed. Its intended purpose is to allow debuggers to single-step through assembly code. The trick is to repurpose the flag: When the popf occurrs, the processor doesn’t pop the interrupt flag, but it does pop the trap flag. After one instruction, the trap interrupt fires, and the kernel regains control. It recognizes that this trap interrupt is being used to regain control when somebody does a popf instruction in a failed attempt to re-enable interrupts. The kernel would then re-enable interrupts.

Result: As far as the application is concerned, the popf instruction successfully restored the interrupt flag despite all the warnings that said it wouldn’t work. It was restored one instruction late, but it did eventually get restored.

That one weird trick rescued a lot of games from the “Doesn’t work” category.

Related reading: What did Windows 3.1 do when you hit Ctrl+Alt+Del?, another case where the operating system used the trap flag.