We’d like to announce we are making changes to our pricing for PagerDuty: we’re simplifying the pricing scheme, reducing overage alert charges, and increasing the price per user. Traditionally, pricing changes have been a very touchy subject for SaaS companies. We would like to do the right thing for all of our customers, so we are automatically grandfathering all existing accounts. This means if you’re a PagerDuty customer, you won’t be subject to any potential price increases.

Why are you changing your prices?

When we first launched PagerDuty, we based our pricing model on the 37signals classic tiered model where prices double as you go up the tiers but you also get more than double the stuff. This pricing model has served us well in the early days. More recently, in the last year, we’ve run into several limitations:

• Smaller IT ops teams are paying for a larger plan than what they need on a user basis, in order to get more alerts.
• We have lots of companies with well over 25 users. Since our max plan, the X-Large, only supports 25 users, we find ourselves spending a lot of time doing custom billing for customers.
• Our current pricing has expensive overage alerts on most of the plans ($0.60 each). If one of our customers had a bad month and used a lot of alerts, they’d end up with a large bill from us, which sucks.

So, we’re making the switch to a per user per month pricing model. The main goal for the new pricing is simplicity: no more pricing tiers, user limits or alert limits; you can add as many users as you need. We’ve also eliminated all overage alert charges for US/Canada and reduced the price for overages in other countries.

What is the new pricing?

We’re scrapping the tiered plans and switching to per user per month simple pricing:

• $18 per user per month

Every user gets:

• UNLIMITED phone & SMS alerts to US and Canada
• 20 phone & SMS alerts per user per month to all other supported countries
• UNLIMITED email alerts

(Additional international phone & SMS alerts are $0.35 each)

The good news: Unlimited alerting for US/Canada, which means no more overage charges ever. For international accounts, we’re including 20 phone & SMS alerts per user per month. These alerts pool together and can be used by any user in the account. We’ve also reduced the price for international overages, should you need more.

The not-so-good news: A higher per user per month price than the old pricing.

Startups and small teams: We’ll also offer a Starter plan at $9 / user / month for up to 3 users. This plan comes with 50 US/Canada phone & SMS alerts per user per month and 10 International phone & SMS alerts per user per month.

When will the changes happen?

We’ll switch to the new pricing on May 7, 2012 (3 weeks from today).

How does this impact me?

If you’re already a paying customer: Nothing will change. All paying customers are automatically grandfathered into their current pricing plan indefinitely. That being said, after the new pricing launch date, you’ll no longer be able to switch to another one of the old plans (so, if you’re on the Large, you wouldn’t be able to upgrade to the X-Large or downgrade to the Medium after May 7). Of course, after May 7th, you’ll have the option to upgrade to the new pricing or stay on your existing plan.

If you’re currently on your 30-day trial: You should check and see if our existing plans are cheaper for your team than $18/user/month — if you use a lot of overage alerts each month, they might not be. If they are, sign up for a paid plan before May 7th and you’ll be able to keep that price forever. If you don’t, you’ll pay $18/user/month (or $9/user/month for the small team Starter plan).

If you have any questions or feedback, please contact us.

In the past, our Reporting tab on your PagerDuty account read more like a phone bill than a report. You could see the total number of alerts sent for a given month, and get a specific list of phone calls made or emails sent for that month. That was about it.

Sure, you can still do that. But you can now do a lot more.

You can now get a high level view of how your operations are trending over time. Are you getting better at it? Worse? What weeks last year were totally abysmal? Do your employees get a case of the Mondays after being hounded by operational issues? Are your weekends and holidays generally light on headaches and nice to your Ops Engineers? What was the all-time worst day for operations last year? You can now answer these questions quickly and relatively easily by visualizing the data in PagerDuty. Just tune your date range with some of our default date-ranges, or use your own. Change your data granularity by setting the data rollup and you’re off to the races.

Once you’ve found the data you care about, you can drilldown to get specific details about what alerts were firing or what incidents were causing them for a particular time range. Once you’ve drilled down, you can further refine your time range query to locate operational problems. Maybe you’ll be able to correlate events you never realized were related.

If the new reporting interface doesn’t provide you all of the data slicing you need, you can always export the data to CSV, open it up in Excel and perform further ninja operations on your numbers.

Regardless, we’re just getting started and we’ve barely scratched the surface on things we can do. We have lots more ideas on where we can go from here. What do YOU want to see?

We are very excited to announce a major change in the business model of PagerDuty: free + ad-supported. As you all know, our old business model involved charging money for the PagerDuty product. In fact, we didn’t even have a free plan; only paid plans with a 30-day free trial.

This is a perfectly cromulent business model which works for many companies, but it just doesn’t scale as we set our sights on becoming the next Facebook or Google. Both Facebook and Google have free products and both companies monetize these products by showing lots of very relevant ads to their users. These companies are definitely web-scale and so is their very successful business model of “free + ad-supported”.

Visual display ads (aka “ignorable” ads)

After making the decision internally to switch to free + ad-supported, we were faced with an essential question: where and how to advertise? Everyone’s already doing visual display ads. In fact, display ads became popular back in the 90s during the first dot-com bubble.

The main issue we have with display ads is that they’re very easy to ignore: you can do so by just looking away. We’ve dubbed these type of ads “ignorable” and we’ve decided not to adopt them for our web application. If we sold these types of ads to advertisers, we’d feel like we’re ripping them off because 99.9% of people just ignore them (assuming they haven’t installed ad-block to do so automatically). As such, we are perfectly content to let Facebook and Google fight over the ignorable display ad scraps.

Ads you can’t ignore

We knew we had to come up with a revolutionary new type of ad, one that you can’t just ignore. We tried really hard to do so and even had multiple company-wide brainstorming meetings in the process, yet we were completely stumped. Then, one day, one of our interns came up with an idea: advertise inside the PagerDuty alerts. Brilliant!

As you know, PagerDuty plugs into a variety of monitoring systems and manages alerting, escalation and on-call scheduling. We dispatch phone call alerts, SMS alerts and email alerts to our users. Instead of messing about with display ads, we thought let’s leverage the alerts we already send to our users in order to advertise to them. Let’s go through an example to see how it works:

• Let’s say you are on-call this week and your entire data center goes down at 2am.
• Your monitoring system notices and tells PagerDuty.
• We ring your cellphone and wake you up.
• When you pick up, you hear “PagerDuty alert” followed by a quick audio advertisement from one of our many ad sponsors, followed by the details of the particular issue (in this case “Your data center is down.”)

The best part is that you can’t ignore these ads. In order to hear the critical details of your outage, you have to sit patiently and listen to a quick 30-second spot about delicious Pepsi Cola or durable Goodyear Tires. For those of you who are responsible for on-call duty, it’s your job to receive alerts from our service and listen to them, including the ad within. Brands can now, for the first time in history, advertise to an extremely captive audience, one whose job it is to listen to our alerts and the brands’ ads.

The new-and-revolutionary Alert-Ad™ ad network

We knew we were onto something really big here. The world has already experienced ads on the web, TV, radio, billboards, magazines and newspapers. However, nobody has ever advertised inside IT alerts, until now.

PagerDuty is the first company in history to bring advertising to phone call, SMS and email alerts. This is absolutely groundbreaking and revolutionary; alerts are the next frontier of advertising. We are doing this via our new Alert-Ad™ advertising network.

Alert-Ad™ allows advertisers and brands to reach a very captive audience of sysadmins, devops and developers, a market segment that has been notoriously difficult to advertise to. This demographic has typically been anti-advertising: they tend to use ad blockers, prefer browsing the web using text-only browsers like Lynx and tend to avoid paying for cable service, instead preferring to watch Netflix. They also tend to be middle-class professionals with large disposable incomes.

Alert-Ad™ allows the best brands and advertisers in the world to target this fertile, lucrative demographic very precisely. Advertisers can target ads to specific times of the day, for specific types of errors, to various regions in the world. For example, Microsoft could advertise the uptime of Windows Server 2008 to sysadmins receiving alerts about their Ubuntu servers going down. Another example: 5-Hour Energy could advertise their energy drink to sysadmins receiving alerts between 1am and 5am. The possibilities are endless.

Advertisers are already flocking to Alert-Ad™

We’re also very excited to announce that we’ve already signed up some really great brands onto the Alert-Ad™ network. We’re already working with Match.com, the leaders in online dating, Pfizer, makers of Viagra, and 5-hour Energy, makers of the 5-Hour Energy energy drink.

Here’s an actual Alert-Ad™ that was recently sent to one of our users (who was asleep at the time):

We can also include ads inside the PagerDuty SMS alerts. Here’s an SMS Alert-Ad™:

In conclusion, we’re very excited about the possibilities of our new Alert-Ad™ system as well as the now-free PagerDuty product. We have lots of other new and revolutionary ideas in the pipeline for other places we can place ads that cannot be ignored. Stay tuned!

Update: We’ve already gotten early feedback from PagerDuty users who are receiving the first Alert-Ad™ ads: they’d like an option to skip the ad and get to their critical alerts faster.

We are an agile company, so we’ve responded. For the low fee of $0.15 (a nickel and a dime), you can press star (*) on your phone’s keypad during a phone-call alert to listen to the ad at twice the speed.

We also want to let you know that we are very sorry this outage happened. We have been working hard over the past 6 months on re-engineering our systems to be fully fault tolerant. We are tantalizingly close, but not quite there yet. Read on for the full details and steps we are taking to make sure this never happens again.

Background

PagerDuty’s main systems are hosted on Amazon Web Services’ EC2.  AWS has the concept of “Availability Zones” (AZ’s), in which hosts are intended to fail independently of hosts in other availability zones within the same EC2 region.

PagerDuty takes advantage of these availability zones and makes sure to spread its hosts and datastores across multiple AZ’s.  In the event of a failure of a single AZ, PagerDuty can recover quickly by redirecting traffic to a surviving AZ very quickly.

However, it’s quite obvious that there are many situations in which all Availability Zones in a given EC2 region fail at once.  From experience, these situations happen roughly every 6 months.  One such region-wide failure occurred early this morning, in which AWS suffered internet connectivity issues across all of its US-East-1 region at once.

The Outage

PagerDuty became inaccessible at 2:27am this morning.

Knowing that fallbacks within other AZ’s aren’t enough, PagerDuty has another fully-functional replica of its entire stack running in another (completely separately owned and operated) datacenter.  We began the procedure to flip to this replica after we were notified of the problem with EC2 and when it became obvious that EC2 was having a region-wide outage.

At 2:42am (15 minutes after the start of the outage), EC2′s US-East-1 region re-appeared, and our systems started to quickly process the backlog of incoming API and email-based events, creating a large number of outgoing notifications to our customers.  At this point we aborted the flip to our fallback external notifications stack.

What we did wrong

Fifteen minutes seems like a long time between when our outage began and when we perform our flip.  And it is.

We use multiple external monitoring systems to monitor PagerDuty and alert all of us when there are issues (we can’t use PagerDuty ourselves, alas!).  After careful examination, the alerts from these systems were delayed by a few minutes.  As a result, we responded to the outage a few minutes late.

This is obviously an action item on us to remedy as soon as possible.  These minutes count.  We know they are very important to you.  We will look at switching or augmenting our monitoring systems as soon as possible.

Another miss on our part was not notifying all of you immediately of our outage via our emergency mass-broadcast system (see http://support.pagerduty.com/entries/21059657-what-if-pagerduty-goes-down).  This was due to an internal miscommunication on when it is appropriate to use this system.  We will come out with another blog post shortly that details exactly how we use this system going forward, and a reminder on how you can register yourself for it.

What we did right

We’ve previously taken steps to be able to mitigate these large-scale EC2 events when they happen.

One such step is the very existence of our externally-hosted fallback PagerDuty environment.  This is an (expensive) solution to this rare problem.  We regularly run internal fire drills where we test and practice the procedure to flip to this environment.  We will continue these drills.

Another step that we’ve taken to mitigate these large-scale EC2 events is to make sure our systems can handle the very high amounts of traffic we see when a third of our customers (all the ones hosted on EC2) all go down at the same time. We’ve made many improvements to our systems over the past 6 months: our system now queues events quickly, intelligently sheds load under high-traffic scenarios in order to continue operating, and makes absolutely sure not to fail to page any of our customers.  These systems performed very well this morning, preventing further alerting delays.

What we’re going to do

A flip, no matter how quick, involves some downtime. This leaves a sour taste in our mouths. We are working (hard!) on our internal re-architecture to fully move to a notification processing system that involves NO temporary single points of failure, even when that SPOF is “all of EC2 east”.

Our new system will use a clustered multi-node datastore deployed on multiple hosts located in multiple independent data centers with different hosting providers. The new system will be able to survive a data center outage without any flips whatsoever. That’s right, we’re going flip-less (because the word “flip” is synonymous with “outage”). We are working full-steam on building this new system and deploying it as soon as possible, while making sure we stay stable during the changeover. This re-engineering effort is fairly substantial, so stand by for a few shorter term solutions.

During our internal post-mortem this morning, we have identified a few places where we can immediately improve the availability of our external event endpoints. These include building better redundancy into our email endpoint as well as our API endpoint. We are prioritizing these changes to the top of the heap.

We are also taking a closer look at moving our primary systems off of AWS US-East. In the short-term, we will continue to use US-East in some capacity (perhaps as a secondary provider). Longer term, we will switch all of our critical systems off of AWS altogether.

Finally, as mentioned above, we will improve our own monitoring systems. We’ve had alerts delivered too slowly by our own external web monitoring, and we will fix this asap. We will also improve our Twitter-based emergency broadcast procedure, which helps us announce to you when we are experiencing internal problems. Keep turned for another blog post about this in the next few days.

You need a backup! One or more people, waiting in the wings, ready to spring into action if your primary on-call is criminally negligent unable to perform his or her duties to the best of their abilities at any given time.

These backups don’t need to be AS “on-call” as your primary engineer^[1], but what they lose in readiness they make up in numbers. It often makes sense to have multiple backups.

In PagerDuty we group the currently-on-call engineer and all of his or her backups into an “escalation policy”, which sets the order in which we alert (email/phone/SMS) people, and the delays in between alerts. There are a lot of ways to organize these escalation policies, but I’ll go over some patterns used by many of our customers (both big and small).

Primary and Secondary

You of course already have some sort of “primary” on-call engineer, and this glorious position is hopefully determined by a calendar that rotates between the people in your ops team in a fair way^[2].

Many of our customers will supplement this rotation with an (unimaginatively-named) “secondary” rotation. This secondary rotation is usually setup to shadow the primary rotation by being configured identically to the primary rotation, but offset by a certain amount of time so it’s impossible to be both primary on-call and secondary on-call at the same time. For example, if your primary rotation contains “Alex, Bob, and Charlie”, then your secondary rotation can contain “Bob, Charlie, and Alex”, in that order.

Over 25% of the (many) on-call calendars we manage here at PagerDuty have either the word “primary” or “secondary” in them, so this is a very commonly-used pattern throughout our customer base.

First-tier and Last-tier Support

If your company is big enough or your operations tasks plentiful enough, you may also have a separate first-tier support team that handles all the basic operations tasks before even your “primary” on-call engineer. This front-line team is usually trained in handling all of the repetitive and annoying small problems that crop up often and have clear resolution procedures. (You know, that stuff that you should seriously just fix already, but hey, you’re busy.) These first-tier support teams are often shared amongst multiple engineering teams, and are placed first in an escalation policy.

So who is placed last in the escalation policy? Who is the last-tier support team? Management, of course!

Your escalation policy shouldn’t just end with your primary or secondary ops teams, but should escalate up to their manager’s phone, and then maybe even their manager’s manager’s phone, assuming nobody acknowledges the alert in time.

This has two purposes: (1) management is ultimately responsible for these important systems and is logically who should be informed if any major problems fall through the cracks, and (2) your ops team will be less likely to “miss” an automated PagerDuty phone call if they know it just means that their boss will phoning them up personally in a few minutes and asking some very pointed questions.

Example

So, putting this all together, a (very complete) Escalation Policy example for a hypothetical “Database Ops” team might look something like this:

1. Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
2. Assign the incident to the user who is on-call in the Primary DBA schedule
3. Assign the incident to the user who is on-call in the Secondary DBA schedule
4. Assign the incident to Dilbert Adams (team lead)
5. Assign the incident to Pointi Haredboss (dev manager)

With timeouts of maybe 10-30 minutes in between escalations, depending on your needs. Note that any individuals (managers) put directly in the escalation policy are essentially on-call all the time, as last lines of defense, so should try to keep their cell phones handy and charged at all times.

[1] They don’t need to carry around a mobile broadband device, for instance, and can feel less guilty about doing things like binge drinking while on duty occasionally partaking in activities that might slightly decrease their on-call abilities.

[2] The primary engineer doesn’t have to be determined by a rotation, and you can instead have some poor soul be primary on-call every minute of every day for his whole tenure at your company, but this tenureship may be short due to burnout.

Some of you may have already been using PagerDuty for New Relic alerts by forwarding alert emails from NR into PD. Our new API-based integration is much better: New Relic can both create and resolve incidents in PagerDuty. That means once an outage is resolved, there’s no need to go into PagerDuty to manually resolve the associated incidents; they’re automatically resolved when New Relic detects the problem is fixed.

Just in case you haven’t heard of New Relic (really?), they’ve built an awesome web application performance tool that works with most of the popular web stacks, including Ruby on Rails, Java and .NET. In addition to webapp performance, New Relic also monitors your servers and end-user performance. We use them at PagerDuty to monitor the performance of our front-end web stack and we’re definitely big fans of the product.

Integration Details

The integration is super-simple to set up. In New Relic, click on “Account Settings” and then “API + web integrations” and you’ll notice a PagerDuty tab and a large green “Alert with PagerDuty” button. Just click the button and you’ll be guided through the integration process. You can also refer to our integration guide for the step-by-step rundown.

We have regular hackdays at PagerDuty, where we build things outside the core product without management (another reason you should work here). A few weeks ago, I rolled out a proof of concept Google App Engine script to use Twilio to record a voicemail and then to pass it around like a regular alert. Triggering alerts from phone calls hasn’t made it’s way on to the development roadmap, so I’m sharing this code sample as a work around for our more technically inclined users — so all the usual caveats and disclaimers apply, namely that our SLAs don’t apply.

Twilio will happily turn a phone call into an MP3 and give us a link to it (which means to get this to work you’re going to need to sign up for a Twilio account as well as a Google App Engine account). We then use Google’s URL shortener to shrink the URL into something that will fit in an SMS — all modern smart phones can figure out what to do with that.

End result, assuming you have SMS contact methods set up, they’ll receive an SMS like:

ALRT #145 on Phone in: http://goo.gl/UMmDx +14153490382 Reply 4:Ack, 6:Resolv.

If you’re comfortable deploying code, it’s up on https://github.com/eurica/PagerDutyCallDesk

We use Google Analytics in our webapp to get a feel for how users use the product, most recently to determine which functionality was prioritized for the mobile site. So generally I look at our analytics every week or two to help developers out, and when Simon asked me to see how popular the mobile site was, I was pretty sure the answer was not “It decreased the use of our webapp by 98%.”:

I won’t name names, but the culprit rhymes with “itwasian”.

What broke

Our UI consists almost entirely of HAML powered by backbone.js, often at the same time. Which meant that we refactored the default Google Analytics code:

  :javascript
    var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
    document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
  :javascript // <-- This line was removed
    try {
      var pageTracker = _gat._getTracker("UA-8759953-1");
      pageTracker._setDomainName(".pagerduty.com");
      pageTracker._setAllowHash(false);
      pageTracker._trackPageview();
      #{yield :google_analytics}
    } catch(err) {}

You'll notice that this generates two JavaScript blocks, which we helpfully merged into one.

That's what broke everything - and by everything, I'm excluding some mobile and rare browsers that still executed the code as intended. For 98% of our visitors, the fact that we merged those two script blocks means that the DOM does not get control after the document.write and the loading of ga.js doesn't happen before _gat is referenced. _gat doesn't exist and that's the end of our analytics on this page.

The simple fix is, of course, to put the second script block back in. But instead we moved to the newest asynchronous Google Analytics code, which doesn't need 2 blocks, since it only requires _gaq to be a JavaScript object, with the rest of the functionality coming later, whenever the browser gets around to it.

:javascript
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-8759953-1']);
  _gaq.push(['_setDomainName', 'pagerduty.com']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

Testing that Google Analytics is working

To ensure I catch this sooner in the future, I've set up some intelligence events on our application and the website inside of Google Analytics to detect if we have abnormally low or high amounts of visitors.

There's at least a day's delay before the email gets sent out, which is a shame (I received the last one while writing this) but it's another layer in our web of alerting. Log into your Google Analytics account, and you'll see "Intelligence Events":

I intend to set up some more advanced heuristics later, but for now let's just test that analytics is working:

That day of lag had the advantage for testing that it sent me yesterday's alert, when our analytics were still broken (but I'm really stretching to call that an advantage).

Tying it all in to PagerDuty

This isn't the kind of alert I want to be woken up in the middle of the night for, but I still use PagerDuty as an incident management system for my analytics alerts, partially for the dogfooding, but also to track our media mentions, twitter mentions etc.

For this I've set up a service that doesn't auto-resolve or expire acknowledgements, to track everything that emails "analyze-me@pdt-dave.pagerduty.com"

Test all the things

So now I'm filling up our Fogbugz with new things to test:

• Our t-shirt mailings
• whether we respond to customer inquires quickly enough
• testing our load times across the website, app stack, blog and the support site (again with automated alerts from Google). We already test this with New Relic.

I don't have a good procedure for determining what we're forgetting to test, but I do have a couple of principles:

• If it fails once, it gets tested forever. I'm kind of expecting this to get lost in the shuffle, but apparently we never sent t-shirts to one or two people that I promised them to, so that's been automated and I now have an adhoc report of who has and hasn't received their shirts. When push comes to crazy, I may integrate it with USPS tracking.
• It needs to be automatic, ideally sending you an alert when some measurement is out of band. We track our time to resolution with Zendesk, so one of my projects is to automate our metrics with automations
• Be a jerk. I'm testing other people's work and trying to get it to fail. I don't care what the reliability team's metrics say, if my page load time spikes, I'm going to demand some answers.

We're still a young company, but we're fiercely dedicated to uptime and when you're dealing with bugs, there are known unknowns and unknown unknowns - and when I started here I never would've known how much I'd enjoy shooting Nerf guns across the office whenever the average page load time increase.

(With any luck, this will be the first post in a series on what happens when you make a mathematician do your marketing.)

Now you’ll have all the major PagerDuty functionality in an easier form factor, and the full webpage is just a click away if you need it.

On the full web-app, Ian has dragged us (kicking and screaming) into the second decade of the 21^st century with a more consistent modern feel. There havn’t been any functionality changes in either of these updates, but hopefully we can make your experience a little less awful when we wake you up at 4am.

Have you ever gotten paged, and known right away that this problem isn’t like the last 15 operations issues you’ve dealt with this week? That this problem is special, and is really, really bad? You know, that kind of problem that you’ve been worrying about deep in your subconscious for weeks now, and that you’ve been hoping would never happen?

Well, what do you do when it happens? Often in these high-pressure situations, you’ll have a very brief period of time (say, minutes) before a problem goes from ‘pretty-bad-but-our-customers-will-forgive-us-and-some-might-not-even-notice’ to simply catastrophic. If you’re a Boy or Girl Scout, you’d just open up the Pressure Release Valve you’ve prepared beforehand and prevent the problem from escalating out of control.

Build Pressure Release Valves

When building or maintaining one of the systems or services that you own, have you ever said to yourself:  ”You know, if situation X ever happened, as improbable as it is, we’d be really boned”? Situation X could be any hypothetical catastrophic disaster scenario for your given system:  both master and slave datastores go down simultaneously;  all your customers or clients decide to flood you with their theoretical peak loads of traffic at once;  your cloud provider of choice suffers a multiple-availability-zone outage;  your multicast-based messaging system suffers from a feedback loop;  etc.

The problem is, if you work with a given system long enough, there’s a higher-than-you’d-like chance that “Situation X” will actually crop up.

So what can you do?  Yes, you could try to engineer a system to try to prevent these catastrophic failures altogether.  But building something like this could be time and cost prohibitive and can easily lead to over-engineered systems if you go too far.  Spending a lot of development time targeting failure scenarios that perhaps have a 5% chance of happening over the course of your lifetime isn’t the best use of your resources^[1].

Instead, create pressure release valves.  You can think of these as a sort of lever or knob that you can adjust during failures in order to reduce the severity of your problem while it is being worked on.  They can often take the form of a configuration-based boolean or constant that can be easily changed in case of an emergency, but can come in other forms too.

You can use these pressure release values to easily flip off (or on) some piece of critical functionality or to dial up or down some important value used in your application.  I’ll go into some examples below.

Brainstorm

To come up with these pressure release valves, get together with your team and brainstorm some (perhaps even semi-outlandish) ways in your system or service can fail catastrophically.  

For each of these failure modes, figure out a way in which the system could be temporarily patched, re-routed, short-circuited, or generally hacked to temporarily reduce the magnitude of the problem.  The goal would be to bring the system back to a functioning state: you will probably be forced to sacrifice functionality in order to do so.  Usually, the 1 – 2 people who are most intimately familiar with a given system must design these hacks. Since these people are not always available in an emergency, it’s good to explore these ideas ahead of time.

After you create a list of all the catastrophic failure modes and the corresponding hacks that would be needed to get the system back in a (semi) working state, you can also start figuring out common patterns in the hacks:

• Would adding a throttle on the incoming requests help in a large number of these failure situations?
• Would disabling the computationally-expensive widget X or Y on your website reduce load?
• Would the ability to re-route all incoming requests from datacenter A to B turn a partial outage into just some latency issues?
• Would relaxing your consistency requirements result in a bit of corrupt data but would make your system available again?
• What other functionality can you sacrifice on-demand from your datastore to get it partially functioning again? Durability? Historical data? The ability to do writes (by using a read-only slave)?
• Would flipping off some of your non-critical background workflows free up capacity for your more important ones?
• Would the ritual sacrifice of an intern appease the operations gods?^[2]

Limping along at only partial functionality is much better than a complete outage, and also takes pressure off the on-call staff while they get started on their methodical S.O.P for fixing the root cause of the problem.

As I said earlier, you could try over-engineering a system to prevent these rare exotic catastrophes before they happen, but it often just isn’t worth it.  Plus, even then, there would probably still be other even-more-improbable-but-still-possible failure modes that could benefit from these brainstorming discussions.  So don’t necessarily waste large amounts of time engineering ways to prevent these obscure problems, but don’t ignore their possibility either. Talk about them!

If anyone has more examples of pressure release valves you keep in your own operations toolkit, I’d be very interested in hearing about them in the comments.

[1] Ignore this advice if you’re building something like a nuclear reactor. Make that shit work.
[2] Just kidding. Operations gods don’t get out of bed for anything less than a fulltime newhire college grad.