Remove the Noise to Cure Alerting Fatigue
Earlier this month at Nagios World Conference North America, Arup Chakrabarti, Operations Engineering Team Lead @ PagerDuty gave a talk on “What You Should Monitor and Alert on in a Production System” and discussed how to filter out useful metrics for actionable alerting. In case you missed it at the conference, we wanted to share a few of his best practices around IT alert management.
Why is There an Alerting Problem?
Computing is getting cheaper and automation is easier, which makes getting alerts on just about anything simple, but managing them difficult. If you subscribe to Google Alerts to monitor any topic, you’ll know what I mean. At first, it’s great to receive email alerts around “cute puppies” since they put a smile on your face. However, the content isn’t quite what you want, when you want it. Instead of depending on these alerts to bring you good information, it just becomes noise.
The same goes for IT application monitoring and alerting. With the decrease in the cost of gathering data, monitoring applications are now collecting more and more data. This is great for analytics, but the problem lies in the fact that alerts are growing at the same, exponential pace. People are becoming numb to alerts, making alerts less effective. Timing and relevance are key to alert management, so to remove the noise, turn off alerts that don’t matter.
Availability Alerting: What Alerts to Turn Off
At first, it may seem daunting to figure out which alerts to turn off because there is a fear of missing out on alerts that do signal a big problem. A good measure of what IT alerts matter is the effect on your customers, or “availability alerting”. For website monitoring, if you are an e-commerce retailer and the shopping cart checkout page is broken, that is an issue that needs to be addressed immediately. However, if there is an issue with load balancing that does not affect the customer’s browsing or buying experience, then it may not warrant an alert. For e-commerce retailers, their alerts should be around what affects the availability around customers’ desired actions on the website.
Analyzing alerting history is also helpful in determining incident severity. PagerDuty customers can figure out how many alerts they received each week, and for each alert, they can ask themselves: Was any action taken? Was a customer affected? Was this fully in my control? In the beginning, low-level alerts at 3 a.m. will require an engineer to acknowledge the incident, validate that it is not critical, go back to bed, and resolve the root cause the next day. By tagging alerts as severity 1, 2, 3, etc. in the monitoring tools or setting thresholds, he can eventually turn off non-critical alerts in the middle of the night and deal with them in the morning. This leaves room for only high severity alerts to be deliver via PagerDuty, and will help cure alert fatigue.
Wake Up When You Need To
If severity 3 and above issues arise that aren’t going to affect customers, does the engineer really need to wake up throughout the night to acknowledge them? Probably not. Those alerts should be aggregated and dealt with the next day. By analyzing incident patterns and severity, alerting can be a powerful solution to maintain a sense of urgency for big problems and to decrease mean time to resolution (MTTR). Similar to adorable Bichon Frise puppies, being able to sleep through an low severity incident while on-call can put a smile on engineers’ faces.
Nagios World Conference NA 2013: What You Should Monitor and Alert on in a Production System (Video)