The new on-call scheduling tool is here. We migrated all accounts to it prior to the holiday break. Some of you have played with it. Some of you love it. Some of you don’t love it as much, but still like it a lot. I’m happy with both of those result types.

At a high level, the new PagerDuty on-call scheduler is a toolkit that allows you to create pretty much any (recurring) on-call rotation you want. Say you’re the owner of a web-based startup and you want to be on-call all the time to get notified when your site goes down. You can do that. (You could do it before, but you know… I’m being explicit.) Say you want to create a DB Secondary Rotation where you have in-house DBAs on-call weekly from 8:00am to 8:00pm, and then a daily rotating night support staff in an entirely different continent on-call overnight. You can do that. (We call that a “Follow the Sun” schedule.) Say you want to create a Frontend Dev Primary Rotation that has three developers who rotate daily, but not on weekends. You can do that. (We don’t recommend having a dead-zone where nobody is on-call, but it is technically possible.) Say you want to put your Grandma on-call for 30 minutes once a week at 7:30am on Wednesday. You can do that. (Though I don’t know why you would.)

How do you achieve all these wacky calendars designed to torture your loved ones at random intervals? Think Photoshop. Think digital image layers. What you do is basically superimpose different rotations on top of each other, leaving gaps in the higher priority layers to allow lower priority ones to “show through.” The layer gaps are like the transparent parts of an image layer that allow you to compose something more interesting.

So that’s all well and good, and actually the important part. What is less important, but more exciting to me (because I’m a dork) is all the behind the scenes magic and the actual design and usability that plays out on the front stage. I’m not going to go into that so much though. I encourage you to play with the new tool and see how it feels. Do you like the feel of smooth responsive Javascript under your mouse clicks? How do you like the non-blocking page loads and the on-the-fly calendar previewing which responds to all of your actions? These are the sorts of things we are going to push for more and more as we continue to develop the PagerDuty product.

With this project (and another one which will be announced very soon), we are starting a rollout of a site restyling and modernization of our app stack. For those of you who are up to date on your design blogs, yes, Twitter’s Bootstrap Framework does lay some basic foundation for our new layout and basic page elements. We feel that it’s best to stick with well known visual paradigms so you can intuitively jump into our (somewhat complicated) application quickly, get productive, and get your systems monitored properly without breaking too much of a sweat.

I better cut this off now as it’s getting long-winded. But, rest assured, this is just the start of even more greatness. We have been getting some great feedback during our Beta period for this feature, and we listen to your suggestions very intently. There is plenty more where this came from!

Oh, and we’ve also added iCal and Webcal integration for the calendar; stay tuned for another blog post with the full details.

This is very much a work in progress, and we’ll continue adding documentation — I’ll be watching which pages people ask for help with and prioritize them, and you can continue to email us at support@pagerduty.com

Just in case you haven’t heard of Loggly, they do logging as a service. Alert Birds is a new application built on top of Loggly which allows you to set up automatic real-time searches over your log files. It allows you to detect exceptions, security problems, performance issues, or any type of custom search you can think of.

By default, if you leave the Alert Birds webapp open, it plays ambient bird sounds when various things happen in your logs (hence the name). Alert Birds can also integrate with PagerDuty: you can set it up so that if something bad happens in your log files, Alert Birds tells PagerDuty and we alert the on-call person via phone call, SMS or email.

If you want to learn more about setting all of this up, please sign up for the webinar here (but you have to hurry as time is running out).

The best part is that when you get paged via phone or SMS for an issue detected by Keynote, the page will contain the full error details: the name of the affected website/webapp, the severity of the issue (Warning or Critical), and whether the issue is performance or availability related.

About Keynote

Keynote is a global leader in Internet and mobile cloud monitoring. With Keynote, you can quickly diagnose website performance problems at the application, transaction, and infrastructure levels in real-time. Their measurement infrastructure consists of over 3000 measurement computers and mobile devices in over 275 locations around the world — wow!

Integration Details

The integration between Keynote and PagerDuty is very simple to set up: it shouldn’t take more than a couple of minutes. On the PagerDuty side, we’ve created a new service type specifically designed for Keynote. It accepts and intelligently parses Keynote email alerts and triggers new incidents when Keynote detects website performance or availability issues. It can also automatically resolve open incidents when Keynote detects that the underlying problems have been fixed. Finally, the new service type allows you to set up alert filters based on the severity of the problem. For instance, you can configure it to only alert on Critical issues and ignore Warnings.

On the Keynote side, you simply need to log into your MyKeynote account and configure your alarms to send emails for both Warning and Critical issues to your PagerDuty-generated email address.

Here is the full Keynote / PagerDuty integration guide, which includes step-by-step instructions on setting up the new integration.

Please understand that we aren’t trying to shift the blame to any other parties, but part of our processes involves understanding any serious downtime and coping with it openly.

What Happened: The Outage

PagerDuty is presently hosted on Amazon Web Services (AWS) Elastic Computing Cluster (EC2).  One of AWS’s most attractive features are “Availability Zones”.  These are “distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region”.

Like many high availability applications, PagerDuty uses multiple Availability Zones to protect our application from data center level failures.  AWS’s high speed inter-AZ networks allow us to synchronously replicate every event, notification, and incident we process to at least two physically separate locations.  Under normal circumstances, in the event of an AZ (i.e. data center) wide failure, we are able to redirect all traffic to one of the surviving AZs within 60 seconds with absolutely no loss of incoming events.

Unfortunately, yesterday the system did not perform as designed. While we’re looking forward to reading AWS’s official post mortem, our own investigation indicates that at least three nominally independent AZs in US-East-1 all simultaneously dropped from the Internet for 30 minutes.  This left us with no hardware to accept incoming events, nor to dispatch notifications for events we’d already received.

What Happened: The Aftermath

The region wide failure of EC2 impacted a large fraction of our customers. Once connectivity was restored, we received an extremely high load of incoming events and emails, and our (only semi-recovered) infrastructure was not able to process the backlog quickly enough.  The load also exposed some performance-related issues within our notification dispatch system.  In the future, our load testing framework will test a scenario where we are hit with a similar level and distribution of traffic.

Prevention: Immediate Plans

We strive to ensure that PagerDuty delivers every alert within 3 minutes of its scheduled delivery date.  A system-wide outage of 30 minutes is obviously completely unacceptable. We’ve already taken the following steps to ensure that a similar region-wide event won’t cause an extended outage:

1. We’ve deployed a replica of our entire stack on an additional hosting provider.  In the event of a similar AWS failure, we will flip our DNS entries to the alternate stack and continue processing from where we left off.  While the flip process won’t be as fast or transparent as is possible with AWS’s elastic IP functionality, this alternate stack provides us with a level of redundancy we cannot achieve using AWS alone.
2. We’ve doubled our front-end capacity.  Unfortunately, PagerDuty has an inherently high-variability load.  When a major hosting provider experiences an outage, a large fraction of our customer base will need to be simultaneously alerted.  We can therefore go from a system that is well under capacity to one that is severely under-provisioned in a matter of moments.  We have some thoughts on how to better address this issue in the future, but for now we have added more idle capacity to our system to handle the extra potential load.

Prevention: Future Plans

Over the coming months, we are planning on making a number of additional improvements to our infrastructure.  These changes will further decrease the chance of a system-wide outage.

1. We intend to switch off of AWS EC2 entirely.  A very high fraction of our customer base host their services on AWS.  This creates a dangerous correlated failure condition where the periods during which our load is likely to be abnormally high are also likely to be times when we are experiencing operational issues ourselves.  Even when a failure is limited to just a single AZ, this creates issues, as it causes us to lose redundant capacity when we most need it.
2. We will host PagerDuty across multiple providers.  Using a single hosting provider with multiple data centers is tempting — it’s much easier to write a distributed app with tools like virtual IPs that can be migrated from one data center to another.  But, in our opinion, the failure coupling that such features introduce to the environment are not worth the risk.  When using multiple hosting providers, the potential for single points of failure are much lower.
3. We will provision and test with the assumption that, at any moment, we could need to alert 33% of our customers within 5 minutes.  This will allow us to cope with “perfect storm” scenarios where a provider level outage triggers failures across a very large proportion of our customer base.  In the past, our load testing has focused on large bursts of traffic from a smaller number of customers.  We will also take steps to ensure that our event queuing and notification dispatch systems degrade gracefully in overload scenarios.

Emergency Response

Another problem uncovered by yesterday’s outage is that we had no effective way to alert our customers that there was a gap in PagerDuty’s coverage.  While we obviously intend to prevent such a gap from ever occurring again, we believe it’s important to plan for all eventualities.

To that end, we’ve created a Twitter account where we will only announce PagerDuty downtime.  By subscribing your cell phone to this Twitter feed, you’ll be alerted any time there is a gap in your PagerDuty coverage.  To learn more about how to set this up, please see our previous blog post.

In addition, we intend to create a custom facility where users can subscribe to receive phone alerts if PagerDuty experiences another system-wide outage.  Naturally, it is our intention to never need to use this system.  However, we want to ensure that we have a way to rapidly notify interested customers of any gap in their PagerDuty coverage.  Obviously, we’ll ensure that this emergency system shares no dependencies with our main notification dispatch service.

Conclusions

Needless to say, we’re sorry for letting you all down. We’ve already taken several steps to ensure this won’t happen again, and we will be taking several more in the upcoming weeks.

Please don’t hesitate to contact us if you have any questions or concerns. We look forward to earning back your trust.

Such an outage happened recently due to the major AWS cross-AZ failure in US-EAST.  We are moving cross-cloud very soon and will be following up this posting with a post-mortem of the event.

We are implementing a new (completely separate and external) opt-in system for those interested in getting notified when PagerDuty is experiencing large-scale issues.  While this is being constructed, we have a temporary (but equally functional) mechanism to alert you of such catastrophic events.  We are using Twitter for this: the @pagerdutyops account has been setup and we ask that you follow it.

Again, it is an opt-in program, but we’d recommend that you participate in it.  We would tweet from this account very rarely, and only during such times that PagerDuty is having major problems delivering notifications (and hopefully never again). These tweets will never be automated; it would come directly from the PagerDuty Ops team. We also ask that you enable mobile notifications for your Twitter account so that tweets from @pagerdutyops reach you instantly — we’re including the simple setup instructions below.

What you’ll need to get started

You’ll need a Twitter account — please create one from here, if you don’t have one already. Follow @pagerdutyops from your Twitter account.

Setting up Twitter to send mobile notifications

Here is the short instructions on how to setup Twitter to send you a text message, when we tweet from @pagerdutyops.

1. In Twitter, click on Settings under your profile.
2. Click on the Mobile tab.
3. Select your Country, enter your Phone Number and select your Mobile Carrier.
4. Click the Start button.
5. Twitter will now ask you to verify your mobile number. Follow the instructions on this page — this may require you to send a text message to Twitter with a special code.
6. Once verified, you’ll be taken to the Text messaging page. Select Tweets from people you’ve enabled for mobile notifications, if not selected already.
7. Click on the Save button to save your changes.
8. Navigate to @pagerdutyops. Click on the Follow button, if not following already.
9. Click on the small mobile icon besides the Following button to enable mobile notifications from @pagerdutyops.

And you should be all set to (hopefully, again, never) receive external notifications to learn about when and if PagerDuty is having failures in delivering notifications.

Our team will be at HostingCon to meet some of our current customers and potential new customers in the hosting industry. We’re also launching a new partnership program at the conference for hosting providers. Make sure you come by our booth (#450) to find out all about it.

As you know, PagerDuty dispatches SMS and phone call alerts to let you know when your servers are down. We’ve also discovered that our service is perfect for alerting people so that they can win awesome prizes. Swing by our booth any time on Aug 9 or 10 to enter the PagerDuty giveaway and put yourself on “prize duty”. Here’s how it works: you enter the contest with your mobile number, and we’ll SMS page you later that day. The people who acknowledge the page the fastest win! Prizes include iPad 2s, Amazon gift certificates and other cool stuff. Plus, we’ll have PagerDuty t-shirts and other cool swag to give out.

Be on the lookout for our booth (#450) and keep your phone at the ready.  We’ll see you next week in San Diego!

Pingdom users can now take advantage of PagerDuty’s comprehensive alerting and scheduling functionality. Users can receive Pingdom alerts by phone, SMS, and email, and can acknowledge, escalate or resolve incidents on the go. PagerDuty also adds incident tracking to Pingdom and helps support teams set up fair on-call duty schedules.

About Pingdom

Want to be the first to know when your website goes down? Pingdom tracks uptime, verifying the availability of your website from multiple locations across the world. Should something go wrong, Pingdom also offers a variety of tools to help you get to the root of the problem.

Integration Details

On the Pingdom side, simply configure downtime notifications to be sent by email to a PagerDuty-provided email address.

On the PagerDuty side, we’ve created a new service type specifically for Pingdom. It accepts and parses the alert emails to create incidents, and, just as importantly, automatically resolves them should your website come back on line.

To read more about how Pingdom can help you keep tabs on your website’s uptime, please visit their website. To learn just how easy it is to add the benefits of Pingdom monitoring to your PagerDuty account, check out the Pingdom Integration Guide.

As for the contest, that was a lot of fun too.  This is how it worked:

• Conference attendees signed up for the contest at our booth (157 pager jockeys and/or SMS ninjas signed up in total).
• We “paged” all of the contestants once over the course of the conference via SMS.
• The fastest person to acknowledge the page (as measured by our servers) won an iPad2.
• Runner-up prizes included 6 months of free PagerDuty service, as well as several BluAlert Bluetooth Bracelets.
• Much fun was had by all.

Here is some of the PagerDuty crew posing with the grand prize winner, Allyson White from Cotendo.

And here is one of the runner up prizewinners, Greg Schechter from Google, stoked about his winning of a BluAlert bracelet.

Thanks to everyone who played, and we hope to see you again next year!