Why does the compiler sometimes use ud2 and sometimes int 3 for code that shouldn’t execute?

There are two common ways for x86 compilers to indicate that execution should not have reached a particular point: One is the single-byte int 3 breakpoint opcode. And the other is the two-byte ud2 invalid instruction opcode. How do they decide which one to use?

The two types of “bad instructions” are typically for different purposes.

The int 3 means “There is no code here. If you somehow got here, then somebody used an invalid function pointer.” It is used as padding, such as between functions. There is no way that code can reach the int 3 by normal execution. You must have generated an invalid address and called it.

The ud2 is used to mark the case when execution reached something that should be unreachable. It means “You executed a code path that the standard says is undefined behavior.” For example, falling off the end of a non-void function without returning a value, or following the call to a [[noreturn]] function in case it somehow managed to return.

Using int 3 for “there is not even code here” is important because it’s a one-byte instruction. If you had used the two-byte instruction ud2 instruction, then that stray function pointer might land on the second byte of the instruction, in which case it’s not ud2 any more. Instead of stopping immediately, it starts executing garbage code:

0b 0f            or      ecx,dword ptr [edi]
0b 0f            or      ecx,dword ptr [edi]
0b 0f            or      ecx,dword ptr [edi]

Okay, so what does this mean for you?

If you find yourself executing the ud2 instruction, then look for logic flaws in your code. If you find yourself executing the int 3 instruction, then look for an uninitialized function pointer variable, or a hard-coded breakpoint, or a debugger-inserted breakpoint.