Outage Post-Mortem – Dec 11, 2013
On Dec 11th, PagerDuty suffered an outage which affected a subset of customers and blocked access to all pagerduty.com addresses. First off, we deeply apologize for this. Any outage, no matter how many customers were affected, is unacceptable. The root cause of the outage can be traced to a problem with our DNS infrastructure, specifically, DNSSEC. This post details what happened and improvements we will make to prevent this from happening in the future.
Background on DNSSEC at PagerDuty
Back in June, we enabled Domain Name System Security Extensions (DNSSEC) for all PagerDuty domains. We did this in order to give customers the ability to validate DNS records received from PagerDuty in a secure manner. Part of the signing process for DNSSEC involves using Zone Signing Keys (ZSK) to sign Resource Records (RR), and the resulting signature (RRSIG) is then automatically deployed by our DNS provider. New RRSIG’s are generated every three months and are regularly rotated and deployed. In order to make sure that there is always a valid RRSIG, both the new and the existing RRSIG’s are deployed along side each other to make sure there is some overlap. DNSSEC has been in use since 2005, with most major DNS providers have implemented in the past five years. Earlier this year, both Comcast and Google DNS started enforcing DNSSEC. This means that if a DNS record has DNSSEC enabled for it, but the RRSIG cannot be validated, the DNS request will return as a failure in order to protect the requestor from potentially manipulated DNS data. That is what occurred on the evening of Dec 11, 2013.
Timeline of Events (all times are in PST):
- Nov 27 – A new ZSK for our pagerduty.com zone is created by our DNS provider, however the new RRSIG is not deployed due to a fault in the automation infrastructure from our DNS provider.
- Dec 11 10:51pm – The existing RRSIG deployed to the pagerduty.com zone expires. DNS providers enforcing DNSSEC start returning no results for the pagerduty.com zone.
- Dec 11 11:51pm – Our secondary external monitoring provider alerts us that pagerduty.com is not accessible.
- Dec 12 12:03am – A PagerDuty Engineer removes the Delegation Signer (DS) record to disable DNSSEC on the pagerduty.com zone
- Dec 12 12:37am – DNS providers with DNSSEC enforcement enabled are now returning pagerduty.com DNS records.
The Impact
During the outage, any customer that was using a DNS provider which enforces DNSSEC could not access any pagerduty.com address. This means a small subset of our customers were not able to use the website, our API or our mobile apps. We advised affected customers to use another DNS provider to get around this problem. Since we were not receiving traffic from affected customers, it is difficult to calculate what the impact was. Our estimate, based on looking at our median load balancer traffic levels, is that less than 2% of our total request volume was affected.
What We Learned
When we originally enabled DNSSEC, we set up monitoring around our DNS records. However, the monitoring that was set up had a networking bias instead of a security bias. So even though our primary external monitoring provider detected that Google DNS was returning invalid results, we didn’t receive appropriate alarms for it. If we had been looking at the expiration times on the RRSIG’s, we could have caught this problem much earlier. Even though a secondary DNS provider would not have solved this specific problem, we need to have additional redundancy to protect ourselves from DNS outages. Lastly, the reason that this outage was not more visible was because not all DNS providers are enforcing DNSSEC. Both Google and Comcast have taken steps to help improve the security around DNS and we hope to see more DNS providers do the same.
What We Are Doing About This
While the root cause of this outage was not something that we had direct control over, there are improvements that we will be making in order to prevent this from happening again and catch the problem sooner.
We will:
- Update our primary external monitoring provider to treat DNS changes as catastrophic level alerts
- Set up continuous monitoring and alerting around our RRSIG expiration times
- Tune the TTL on our DS record down to 1 minute to reduce propagation delays around changes
- Setting up a secondary DNS provider
We are continuously improving our infrastructure to provide our customers with the most reliable service. If you have any questions or comments, please let us know.