I write a version of this letter internal to Microsoft each week. This was the one from this week - and enough people internally felt it was useful to external audiences, that I decided to repost it here. One of the most pernicious, and common, patterns that engineering teams fall prey to is the "noisy monitor". This is any kind of signal - a pager, compiler warning, or bug database - that accumulates noise relative to useful signal over time. This pattern is often the cause of outages and product failures. The problem with noise is that we are pretty good as humans at weeding it out, and we are very good at making up reasons why the warnings we are getting "aren't important". Sometimes we build tools for this, like silencing a pager, or scripts that filter out the "noisy" errors, and sometimes we just...acclimate to it.
I was hoping to see a cool story about hunting down noisy CRTs or something... alas. But this is a good letter despite that and spot on.
There are many interesting connections between tech and high end cooking. The person who explored that best is Vaughn Tan. https://uncertaintymindset.substack.com/p/52-a-year-and-change?s=r