The cries of "Oh no!" emerge from each office as the realization slowly dawns

Raymond Chen

Today is the (approximate) 15th anniversary of the Bedlam Incident. To commemorate that event, here’s a story of another email incident gone horribly awry. Some time ago, an email message was sent to a large mailing list. It came from somebody in the IT department and said roughly, “This is a mail sent on behalf of Person X to check if your XYZ server has migrated to the new datacenter. Please visit http://blahblah and confirm that your server name is of the form XXYYNNZZ. If not, please Reply to All.” Uh-oh. The seasoned Microsoft employees (and the new employees who paid attention during new employee orientation) recognized the monster that was about to be unleashed, and the cries of “Oh no!” could be heard emerging from each office as the realization dawned. And then it started. All the replies from people saying “I’m still on the old datacenter.” And then the replies from people saying, “Stop replying!” What’s frustrating is that you can’t do anything about the catastrophe that is unfolding. Any attempt to reply to the message telling people to stop replying only makes the problem worse. All you can do is stand back and wait for the fire to burn itself out. Ten minutes later, Person X sent a message to the mailing list. “Please DO NOT Reply all to this email thread. I am working with the IT department to see if there is another way to get this information.” It took another ten minutes for the messages to finally stop, but that seems to have shut things down. Now it’s time for blame and speculation! We were never told whose brilliant idea it was to try to gather information by sending mail to 7000 people telling them to reply all. One theory was that Person X went to the IT department saying “Hi, I’d like to collect information XYZ from this large list of people. Can you help?” And some new hire in the IT department said, “Sure, I can get that information for you. I’ll just send everybody some email!” After the dust settled, somebody made a tally of the damage. Number of people on the mailing list: around 7000.
Number of replies: 70.
Of those, number of replies saying “stop replying”: 17.

To commemorate this event, a colleague of mine who maintains a popular internal tool pushed out an upgraded version. The new version had a checkbox on the main page:

I have not been migrated to the new datacenter.

Bonus chatter: It so happens that the message was sent at the beginning of the summer, and most of the “I’m still on the old datacenter” replies came from summer interns. Maybe it was a test.

Raymond Chen

Follow