{"id":111383,"date":"2025-07-16T07:00:00","date_gmt":"2025-07-16T14:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=111383"},"modified":"2025-07-16T14:57:35","modified_gmt":"2025-07-16T21:57:35","slug":"20250716-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20250716-00\/?p=111383","title":{"rendered":"The Fundamental Failure-Mode Theorem: Systems lie about their proper functioning"},"content":{"rendered":"

I have on occasion referred to Le Chatelier’s Principle for complex systems<\/a>, as presented by John Gall in the book Systemantics<\/i>: “Every complex system resists its proper functioning,” meaning that whenever you make a change to a complex system, parts of the system work to counteract and possibly even neutralize that change. If you add a notification feature<\/a> so that everybody related to a pull request receives an email notification every time there is a change to that pull request, what typically results is that people create rules to auto-delete those notifications, and the resulting system is no different from where it started, except that it’s more wasteful.<\/p>\n

The Fundamental Failure-Mode Theorem says that every complex system is running in a failure mode somewhere. There is always something that is not working, but you usually don’t notice because other parts of the system are compensating for it.<\/p>\n

I ran into the Fundamental Failure-Mode Theorem many years ago when was trying to accomplish an unfamiliar task X, and the documentation suggested that should I use one particular tool. When I ran the tool, it said, “Before you can do X, you must do Y.”<\/p>\n

I found the instructions on how to do Y, and they said that doing Y takes four hours.<\/p>\n

Four hours later, Y was complete, and I went back to run the tool. This time, it gave a different message.<\/p>\n

“Sorry, this tool does not support X.”<\/p>\n

(Fortunately, most of that four hours was spent waiting around, so I was able to get other stuff done in the meantime.)<\/p>\n

Bonus chatter<\/b>: I ran into another case of this just the other day.<\/p>\n

I asked an app’s built-in AI chatbot, “Please frob the widget.”<\/p>\n

It replied, “Got it. If you need help with anything else, just let me know!”<\/p>\n

I checked on the widget. It wasn’t frobbed.<\/p>\n

“You said that you frobbed the widget, but it is still unfrobbed.”<\/p>\n

The AI chatbot replied, “Thanks for pointing that you. I don’t have the ability to frob widgets. However, I can help you frob it yourself. (instructions follow)”<\/p>\n

Bonus insult: The instructions told me to click on buttons that don’t exist. I went eight rounds with the chatbot trying to get good instructions and eventually gave up. It asked me if I wanted to submit feedback. I said yes. The instructions it gave me for submitting feedback also didn’t work.<\/p>\n

I think it’s called a chatbot because its primarily goal is to chat, not to solve problems.<\/p>\n","protected":false},"excerpt":{"rendered":"

It doesn’t say what it does on the tin.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[26],"class_list":["post-111383","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-other"],"acf":[],"blog_post_summary":"

It doesn’t say what it does on the tin.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/111383","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=111383"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/111383\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=111383"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=111383"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=111383"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}