In the product end game, every change carries significant risk, episode 2

A colleague related this story of a bug that arose at the very end of the Word 97 release cycle.

Testing had identified a ship-stopping bug in a somewhat common code path. The bad news is that the bug was sporadic. The good news is that the test team had come up with a script that could trigger the crash in the lab with fairly good reliability. The bad news is that the bug went away when you used the debugger.

The development team was very anxious, because a bug that appears only sporadically in the lab is going to occur regularly in the wild. But how do you debug a problem that resists debugging?

There was talk of using an in-circuit emulator (ICE), which is basically a separate computer that ran a CPU emulator. The ICE had a cable that plugged into the target machine’s CPU socket, and that was the mechanism by which the ICE could emulate the operation of a CPU: by physically reproducing the electrical signals that a real CPU would generate. Since the CPU was emulated, you could use the ICE to set breakpoints on things that happen inside the CPU itself, like “Break when the value 42 is written to this memory location when the CPU has interrupts disabled.” (Though in this case, it was just for setting breakpoints and inspecting memory from outside the system.) To most software developers, in-circuit emulators existed only in myth, and the possibility of acquiring one and using it was met with the excitement of a five-year-old boy who learns that he might get to ride on a fire truck.

The developers who were leading the investigation beamed with joy when they identified that most of the crashing systems came from the same manufacturer, and they were all manufactured before a specific date. At this point, it wasn’t too long before a CPU errata was found that was consistent with the manufacturing dates of the affected systems.

Now, the compiler they were using had issued an update to avoid the offending code sequence, but the team had locked their toolset before that update became available, and upgrading the compiler while in escrow¹ is not a good idea, because who knows what new bugs would be introduced by switching to a new compiler.²

The team wrote a tool to scour their binaries to look for any occurrences of a code sequence that would trigger the CPU erratum.² They found around 150 instances of the troublesome code sequence, but one of the requirements for the CPU bug was that the sequence span a page boundary, and only one of those 150 incidents crossed a page boundary.

And their testers found it.

To avoid introducing any new problems as a side effect of the fix, the team opted to do a binary patch to the binary to insert a nop into the offending code sequence. This was enough to avoid the CPU erratum without risking regression to any other code.

Bonus reading: Related story.

¹ Supplemental reading: Escrow vs. release candidate.

² There is a small risk that the compiler will have a bug that is triggered by the product source code, but there is a much bigger risk that that there are pre-existing bugs in the product source code that would be exposed by the new compiler. For example, an uninitialized variable bug could be masked by the fact that the old compiler’s choice of memory layout means that the previous use of the memory was for a pointer that was never null, but the new compiler lays out local variables differently, and now the previous use of the memory was for an integer that is sometimes zero.

Sound familiar?