That time we had a network outage due to unexpected hunter activity

Raymond

The current Office team were very much amused by my story of the The Magical Excel 97 Far East Language Build Screwdriver™, although the details couldn’t be verified by anyone still on the team, and the timeline and other details weren’t quite accurate.¹ Nevertheless, the old-timers did admit, “Stuff like that happened all the time.”

In appreciation, they shared their own funny story from 2015:

The team were informed of emergency network maintenance between Redmond and the Quincy data center. The maintenance would require the shutdown of most of the direct circuits between the two sites, and the remaining circuits weren’t enough to cover typical daily usage. However, the network operations team were able to acquire extra circuits to make up the difference. Unfortunately, those extra circuits took a longer route between the two sites, adding latency. The network operations team requested that data migration or other large operations be postponed if possible until after the repairs are complete. The repairs were expected to take four hours, but the proximity to the power lines required coordination with the local power authority and extra safety precautions, so the local power authority requested a 10-hour maintenance window.

So far, this is typical network outage email.

The interesting part came when they explained the cause of the outage.

“The fiber was damaged by a hunter shooting at a pheasant near the power lines.”

¹ Okay, so it was Building 36, not Building 37. Not that that really affects the story.

4 comments

Comments are closed. Login to edit/delete your existing comments

  • Joshua Hudson

    I want to know what that fluid is. I’ve heard of immersing the computer in liquid nitrogen before, but this is not that.

    • MGetz

      The article says 3M but the reality is it’s almost certainly Fluorinert unless they wanted to do something super special. But it could also be “Novec Engineered Fluids”. Not sure how much people still use that as they are ‘low-toxicity’ not… non-toxic.

      • Todd C

        It would likely be Novec. It used to be Fluorinert, but since it is so stable, it does not break down in the environment, and as a result has a huge global warming potential, and that requires special handling and disposal. Novec is a new chemistry that degrades much more rapidly outdoors.

  • Andrew Brown

    in the linked article, a missed opportunity….
    “We are the first cloud provider that is running two-phase immersion cooling in a production environment,” said Husam Alissa,

    Why did he not say, “We are the first cloud provider that is actually making it rain in a production environment,”??