Have you ever gotten paged, and known right away that this problem isn’t like the last 15 operations issues you’ve dealt with this week? That this problem is special, and is really, really bad? You know, that kind of problem that you’ve been worrying about deep in your subconscious for weeks now, and that you’ve been hoping would never happen?

Well, what do you do when it happens? Often in these high-pressure situations, you’ll have a very brief period of time (say, minutes) before a problem goes from ‘pretty-bad-but-our-customers-will-forgive-us-and-some-might-not-even-notice’ to simply catastrophic. If you’re a Boy or Girl Scout, you’d just open up the Pressure Release Valve you’ve prepared beforehand and prevent the problem from escalating out of control.

Build Pressure Release Valves

When building or maintaining one of the systems or services that you own, have you ever said to yourself:  ”You know, if situation X ever happened, as improbable as it is, we’d be really boned”? Situation X could be any hypothetical catastrophic disaster scenario for your given system:  both master and slave datastores go down simultaneously;  all your customers or clients decide to flood you with their theoretical peak loads of traffic at once;  your cloud provider of choice suffers a multiple-availability-zone outage;  your multicast-based messaging system suffers from a feedback loop;  etc.

The problem is, if you work with a given system long enough, there’s a higher-than-you’d-like chance that “Situation X” will actually crop up.

So what can you do?  Yes, you could try to engineer a system to try to prevent these catastrophic failures altogether.  But building something like this could be time and cost prohibitive and can easily lead to over-engineered systems if you go too far.  Spending a lot of development time targeting failure scenarios that perhaps have a 5% chance of happening over the course of your lifetime isn’t the best use of your resources^[1].

Instead, create pressure release valves.  You can think of these as a sort of lever or knob that you can adjust during failures in order to reduce the severity of your problem while it is being worked on.  They can often take the form of a configuration-based boolean or constant that can be easily changed in case of an emergency, but can come in other forms too.

You can use these pressure release values to easily flip off (or on) some piece of critical functionality or to dial up or down some important value used in your application.  I’ll go into some examples below.

Brainstorm

To come up with these pressure release valves, get together with your team and brainstorm some (perhaps even semi-outlandish) ways in your system or service can fail catastrophically.  

For each of these failure modes, figure out a way in which the system could be temporarily patched, re-routed, short-circuited, or generally hacked to temporarily reduce the magnitude of the problem.  The goal would be to bring the system back to a functioning state: you will probably be forced to sacrifice functionality in order to do so.  Usually, the 1 – 2 people who are most intimately familiar with a given system must design these hacks. Since these people are not always available in an emergency, it’s good to explore these ideas ahead of time.

After you create a list of all the catastrophic failure modes and the corresponding hacks that would be needed to get the system back in a (semi) working state, you can also start figuring out common patterns in the hacks:

• Would adding a throttle on the incoming requests help in a large number of these failure situations?
• Would disabling the computationally-expensive widget X or Y on your website reduce load?
• Would the ability to re-route all incoming requests from datacenter A to B turn a partial outage into just some latency issues?
• Would relaxing your consistency requirements result in a bit of corrupt data but would make your system available again?
• What other functionality can you sacrifice on-demand from your datastore to get it partially functioning again? Durability? Historical data? The ability to do writes (by using a read-only slave)?
• Would flipping off some of your non-critical background workflows free up capacity for your more important ones?
• Would the ritual sacrifice of an intern appease the operations gods?^[2]

Limping along at only partial functionality is much better than a complete outage, and also takes pressure off the on-call staff while they get started on their methodical S.O.P for fixing the root cause of the problem.

As I said earlier, you could try over-engineering a system to prevent these rare exotic catastrophes before they happen, but it often just isn’t worth it.  Plus, even then, there would probably still be other even-more-improbable-but-still-possible failure modes that could benefit from these brainstorming discussions.  So don’t necessarily waste large amounts of time engineering ways to prevent these obscure problems, but don’t ignore their possibility either. Talk about them!

If anyone has more examples of pressure release valves you keep in your own operations toolkit, I’d be very interested in hearing about them in the comments.

[1] Ignore this advice if you’re building something like a nuclear reactor. Make that shit work.
[2] Just kidding. Operations gods don’t get out of bed for anything less than a fulltime newhire college grad.

For a while now, we’ve had a feature to send out alerts as short emails for our customers who send emails to pagers but we’ve selectively enabled it for some accounts. Based on customer demand, we’re going to turn that on for all accounts now so if you’re sending emails to a pager (or an SMS gateway*) you can enable short emails and we’ll send the same text we’d send via an SMS:

Sends:

* Just a gentle reminder: although we don’t stop you from sending emails to an SMS gateway, we can’t guarantee delivery. SMSes sent this way are not guaranteed to be delivered quickly or at all.

The Audience for the “Monitoring Sucks!” panel, from Zenoss

It was a great crowd with a lot of great open source projects on the go, even if I feel I may have to hand in my nerd card for my lack of github followers. In my defense, I managed to weigh our helium balloons to neutral buoyancy quicker than the Linux Chix (we were studying air currents in the conference hall, as one does [I loved their penguins]). There are already some good recaps out there, but I missed the talks and can’t comment on those (though I only heard good things). Apart from the great customer contact, we picked up a few exciting resumes from applicants. Ilan and the all volunteer team did a great job, and I expect we’ll be at SCALE11x next year.

We gave away an iPad as part of our "Be a PagerDuty Hero contest"

We gave away a lot of t-shirts (we’re down to a few smalls and XXXL shirts left) and all of our pens, so next year we’ll bring more stuff — I don’t think I really expected over 2000 attendees at the event, all of whom needed to see James from Crash Space‘s Wiimote light-saber and OE Cake demo.

Great time, see you all in February 2012.

The new on-call scheduling tool is here. We migrated all accounts to it prior to the holiday break. Some of you have played with it. Some of you love it. Some of you don’t love it as much, but still like it a lot. I’m happy with both of those result types.

At a high level, the new PagerDuty on-call scheduler is a toolkit that allows you to create pretty much any (recurring) on-call rotation you want. Say you’re the owner of a web-based startup and you want to be on-call all the time to get notified when your site goes down. You can do that. (You could do it before, but you know… I’m being explicit.) Say you want to create a DB Secondary Rotation where you have in-house DBAs on-call weekly from 8:00am to 8:00pm, and then a daily rotating night support staff in an entirely different continent on-call overnight. You can do that. (We call that a “Follow the Sun” schedule.) Say you want to create a Frontend Dev Primary Rotation that has three developers who rotate daily, but not on weekends. You can do that. (We don’t recommend having a dead-zone where nobody is on-call, but it is technically possible.) Say you want to put your Grandma on-call for 30 minutes once a week at 7:30am on Wednesday. You can do that. (Though I don’t know why you would.)

How do you achieve all these wacky calendars designed to torture your loved ones at random intervals? Think Photoshop. Think digital image layers. What you do is basically superimpose different rotations on top of each other, leaving gaps in the higher priority layers to allow lower priority ones to “show through.” The layer gaps are like the transparent parts of an image layer that allow you to compose something more interesting.

So that’s all well and good, and actually the important part. What is less important, but more exciting to me (because I’m a dork) is all the behind the scenes magic and the actual design and usability that plays out on the front stage. I’m not going to go into that so much though. I encourage you to play with the new tool and see how it feels. Do you like the feel of smooth responsive Javascript under your mouse clicks? How do you like the non-blocking page loads and the on-the-fly calendar previewing which responds to all of your actions? These are the sorts of things we are going to push for more and more as we continue to develop the PagerDuty product.

With this project (and another one which will be announced very soon), we are starting a rollout of a site restyling and modernization of our app stack. For those of you who are up to date on your design blogs, yes, Twitter’s Bootstrap Framework does lay some basic foundation for our new layout and basic page elements. We feel that it’s best to stick with well known visual paradigms so you can intuitively jump into our (somewhat complicated) application quickly, get productive, and get your systems monitored properly without breaking too much of a sweat.

I better cut this off now as it’s getting long-winded. But, rest assured, this is just the start of even more greatness. We have been getting some great feedback during our Beta period for this feature, and we listen to your suggestions very intently. There is plenty more where this came from!

Oh, and we’ve also added iCal and Webcal integration for the calendar; stay tuned for another blog post with the full details.

To that end, I’m hoping you could spend 2 minutes filling out this survey: https://www.surveymonkey.com/s/PDMONB

Thanks!

You can donate to our team using any of them links above.

This is very much a work in progress, and we’ll continue adding documentation — I’ll be watching which pages people ask for help with and prioritize them, and you can continue to email us at support@pagerduty.com

In the first post of this series, we defined and introduced some concepts of system availability, including mean time between failure – MTBF – and mean time to recovery – MTTR. In our second post we went on to discuss easy ways in which you can effectively reduce MTTR starting now. This post continues on that theme with more tips on reducing MTTR and directly increasing your availability.

Have an S.O.P

For the really bad problems, have a Standard Operating Procedure that everyone knows how to follow.  The SOP should be a set of steps that can be taken to make it easier to work on the problem by increasing communication and organization. This is different than the documented failure modes in the “know your enemy” section in my last post, in that an SOP is a generic procedure that can be used in pretty much any major failure scenario.

The SOP would be something your on-call primary engineer can bust out if some major catastrophe happens that he or she can’t immediately resolve within a designated timeframe – like say 10 minutes.

An example SOP might be for your on-call to:

• Start a conference call and invite others on the team.  This ensures that everyone working on the problem has a quick and easy communications link, and that you don’t step on each others’ toes.  You could use a dedicated conference bridge phone line, or something as simple as Skype.  Just have this voice channel organized beforehand, so you don’t have to figure out everyone’s skypeID, etc, when really pressed for time.
• Make sure there is a designated call leader or “Incident Commander” – a seasoned on-call veteran who doesn’t necessarily know the broken system in question but knows how to direct others in debugging and resolution tasks.  This call leader should keep everyone on track, make sure balls don’t get dropped, and resolve disputes if they come up.
• Have a set of diagnostics that the on-call primary can start on ASAP while the call is being setup and people are joining.  These diagnostics can be things like monitoring data, relevant graphs, related problems in other systems, etc, and will be immediately useful when the conference call begins.
• Have a designated chat system prepared for sharing data, links, code snippets, or whatever needs to be shared non-verbally.  Some ticketing systems are good for this as well.  Here at PagerDuty we use HipChat.
• If you’re part of a large organization, have a designated persion who communicates with business stakeholders, such as upper management.  These stakeholders will be understandably (very) interested in your fixing the problem as fast as possible, but will likely be disruptive if they join the conference call directly.  Your VP probably doesn’t know the bowels and intricacies of your messaging layer or caching system, but can definitely intimidate the already-stressed engineers in a conference call.  These people can, however, be very useful if major decisions need to be made that might disrupt other facets of the business (say by causing you to temporarily lose orders/requests/money/face/whatever) in order to help fix the problem at hand, so communication with them is important.

Having an SOP for really high-severity issues reduces the variability of response and resolution times, and gets everyone up to speed quickly, including business decision makers.  It also reduces stress, uncertainty, and confusion when there is a pretty clear procedure to follow when starting to deal with large issues.

“..in the example above, if my web servers are spewing 500 exceptions, I want my ops folks to get notified, provided there are more than 10 – I don’t want to wake anyone up over a little blip!”

As someone who gets those 2am phone calls, anything to eliminate false positives is exciting.

Now, you can either download your calendar for the next month, or use a secret URL that’s unique to your user profile to integrate your on-call schedule with whatever calendar app you use.  It’s read-only right now, but there are two things that I’m really excited about:

• If you share your secret URL with your team, they can add your schedule to their calendar, so managers can see the whole team at once in their calendar.
• If you use a program that gives you reminders, you can be notified before you go on call, which is a popular ask on our feedback forum