Some time ago, I mentioned that the ud2 instruction was created to be an architecturally undefined instruction. But why not just use an int 6 instruction? It’s a two-byte instruction, just like ud2, and interrupt vector 6 is the invalid opcode vector, so a program could just invoke it manually and get whatever behavior the operating system assigns to an attempt to execute an invalid opcode.
Well, not exactly.
Architecturally, the invalid opcode is a fault, which means that the address of the faulting instruction is reported to the exception handler. On the other hand, the int 6 is a trap, which means that the address of the instruction after the int 6 is reported to the handler.
This distinction between faults and traps reflects how they are used. A fault represents an error that the kernel could attempt to repair and retry. For example, a page fault could be repaired by paging the memory in. An access violation could be repaired by changing the page protections. A trap reports an operation that has already occurred and gives you a chance to do some special work in response to it before resuming execution. In the case of a software interrupt, the handler wants to process whatever the interrupt means, and then resume execution with the next instruction.
Now, some faults are unlikely to be repaired, but reporting them as faults means that you have to take extra steps if you want to execute past them. If you just ignore the fault, the instruction restarts and faults again.
If you manually raise the int 6 interrupt to simulate a #UD undefined instruction fault, the handler will think it’s dealing with a CPU-generated undefined instruction fault, so it expects the instruction pointer in the fault frame to point to the undefined instruction. But since int puts the address of the next instruction into the fault frame, the fault handler will look at the instruction and get confused as to why the invalid instruction fault was raised. “It says here that the faulting instruction is a xor eax, eax. That’s a perfectly legitimate instruction. Why did it fault?”
One case for inspecting the faulting instruction is to emulate missing instructions on older processors.
Furthermore, in protected mode, you need to have permission to issue an int 6. The interrupt descriptor table describes what to do for each interrupt, as well as what CPU privilege level is permitted to invoke it explicitly. And to avoid exactly these types of problems where a user-mode program issues an int 6 and confuses the #UD handler, all operating systems deny int 6 privileges to user mode.
So even if you try to fake a #UD via an explicit int 6, it doesn’t work. To get the #UD undefined instruction fault, you have to issue an actual undefined instruction.
@Raymond Chen
> If you manually raise the int 6 interrupt to simulate a #UD undefined instruction fault, the handler will think it’s dealing with a CPU-generated undefined instruction fault, so it expects the instruction pointer in the fault frame to point to the undefined instruction. But since int puts the address of the next instruction into the fault frame, the fault handler will look at the instruction and get confused as to why the invalid instruction fault was raised.
And what is stopping the handler from saying "Ah this is int 6, so I should look at the previous...
The handler doesn't get to know whether it was INT 6, or an undefined instruction. It just knows that you entered IDT entry 6, and the CPU pushed CS and EIP onto the IDT entry's stack for you. If you enter via INT 6, it pushes CS:EIP for the instruction after the INT 6; if you enter via an undefined opcode, it pushes the CS:EIP for the undefined opcode.
The handler could look back and decide whether or not it was plausibly an INT 6, but that is just a heuristic - what if a legitimate jump target begins just after...
Not sure what you mean by "Ah this is int 6." They handler isn't given the int 6.
Undefined instruction:
<code>
int 6:
<code>
The handler would have to peek at the previous instruction, see that it's an int 6, but now it has to guess. "Gosh, did I get called because somebody executed that int 6? Or did the CPU merely jump to 2?"
Consider this case:
<code>
What triggered the handler? Was it the int 6 or the ud2?
It’s simpler than all this. If there is a documented way of doing something, don’t go and invent undocumented ways of doing the same. You’ll only confuse yourself, others, or the OS.
The problem here is the transition from "there is no documented way to do this" to "do this the documented way". UD2 isn't defined until the Pentium; #UD exists from the 80186/80286 onwards.
UD2's problem statement was "different programmers choose different opcodes to raise #UD. We want to find an opcode we can document, that works to raise #UD on as many extant CPUs as possible (from 80186/80286 onwards), is compatible with existing #UD handlers, and that will allow us to point at the docs if someone chose the wrong undefined opcode to trigger #UD". INT 6 didn't work for that,...
So what happens if code trys to run an `int 6` and it’s denied? I’m guessing it results in a fault, just not the right one?
It results in a general protection fault if I remember correctly.