Bonus post - Noisy Monitors

I write a version of this letter internal to Microsoft each week. This was the one from this week - and enough people internally felt it was useful to external audiences, that I decided to repost it here. One of the most pernicious, and common, patterns that engineering teams fall prey to is the "noisy monitor". This is any kind of signal - a pager, compiler warning, or bug database - that accumulates noise relative to useful signal over time. This pattern is often the cause of outages and product failures. The problem with noise is that we are pretty good as humans at weeding it out, and we are very good at making up reasons why the warnings we are getting "aren't important". Sometimes we build tools for this, like silencing a pager, or scripts that filter out the "noisy" errors, and sometimes we just...acclimate to it.

Read →