As for the contest, that was a lot of fun too.  This is how it worked:

• Conference attendees signed up for the contest at our booth (157 pager jockeys and/or SMS ninjas signed up in total).
• We “paged” all of the contestants once over the course of the conference via SMS.
• The fastest person to acknowledge the page (as measured by our servers) won an iPad2.
• Runner-up prizes included 6 months of free PagerDuty service, as well as several BluAlert Bluetooth Bracelets.
• Much fun was had by all.

Here is some of the PagerDuty crew posing with the grand prize winner, Allyson White from Cotendo.

And here is one of the runner up prizewinners, Greg Schechter from Google, stoked about his winning of a BluAlert bracelet.

Thanks to everyone who played, and we hope to see you again next year!

The Presentation

The Q&A

The Slides

This kind of outage is one of PagerDuty’s big “moments”;  when a big chunk of the services on the internet say:  ”Hey PagerDuty, I’m down, so wake someone up to fix me!”

There’s lots of coverage on this issue already out there, so we won’t go into much detail on the AWS situation itself.  But we’d like to share some statistics on the alerts we sent out – via phone or SMS – during the outage.  We think these numbers could shed some light on what proportion of the internet was affected by the issues. We don’t presume that we’re being used (yet!) by a “huge”, “moderate”, or even “realistically statistically significant” proportion of internet websites or SaaS providers, but we think these numbers are definitely interesting and can be taken on the whole as a sort of pain metric for this AWS outage.

Since the outage began, we have routed notifications to around 36% of our customer base.  In other words, 36% of PagerDuty customers have been having issues – ones big enough to actually page one of their sysadmins or engineers to work on the problem – since the AWS problems began.

Most PagerDuty customer accounts have more than one user – sysadmin, engineer, “ops guy”, etc – involved in their on-call rotations.  We have paged more than 10% of our entire user base.  In other words, more than 10% of all of our customers’ operations staff have been woken up and/or called in by our systems to work on their problems.  This is probably just the tip of the iceberg as well, since we usually just handle the first alert;  these AWS problems are probably causing a lot of “all-hands-on-deck”-type situations where the entire ops teams (and more) will be called in to fight fires after the on-call has been woken up by PagerDuty.

Below is a graph of the number of alerts – phone, SMS, and email – that we have sent out over the last 48 hours.  There was a big spike in outgoing alerts around the time of the AWS outage, and alert levels have remained high since.

PagerDuty outgoing phone/SMS/email alerts during AWS outage

Below is a graph of the number of “events” being sent to PagerDuty by our customers’ monitoring systems, through our API or through email.  We don’t send out phone/SMS/etc alerts for every “event” that gets sent to us by monitoring systems, but we de-duplicate them so as not to overwhelm our already-harassed and bleary-eyed users.  As you can see, we were flooded by a huge number of events when the outage first began, and incoming event levels are still high.

PagerDuty incoming events during AWS outage

For every entrepreneur, there comes a time to pivot. For some its because their ideas are not producing the results they desire, and the users are not responding as they expect. For us, its the opposite; our product has been very well received and we feel like we’ve solved this problem, and now we need a new challenge. At PagerDuty, we’ve enjoyed waking you up in the middle of the night to tell you that your servers were broken, and if you didn’t acknowledge the problem in time, then we were even more excited to wake your manager up and tell them that you were too busy dreaming or that you didn’t even care. But now we’re even more excited about the new challenges ahead of us.

The public transportation system has been broken for too long. In the valley it’s so bad that to get from point A to point B, it will take you three times longer by a bus than it would for you to walk. The US public transportation ranks 104th in the world, behind Uzbekistan where a massive fleet of private vehicles will taxi you to your location for similar costs as their transit system.

To provide a solution we must first understand the problem. Why is the US public transportation system so broken? Some critics say:

“Everyone has cars already, so they don’t need public transportation” – It is true that every household in the US has multiple cars, but that is merely because there are no alternatives to driving. To get from point A to point B you must have a car. Carpooling is not always an option in the family if you want the family to live together peacefully, so every person ends up needing their own cars. Carpooling with coworkers is not possible since they often live on the opposite side of town, and besides there are too many hybrids in the carpool lane.

“Population density is too low to deploy a cost effective public transportation system” – This is also very true when considering traditional forms of public transportation such as buses and trains. With that said, while traveling through the Middle east and Asia, we’ve noticed that even in the low density cities and towns, you are able to easily grab a taxi at costs extremely affordable to the locals. This is mostly possible due to an unregulated taxi cab business, opening up doors for the unemployed and unemployable to buy a cheap car and start transporting people. In the US, the taxi cab business is so regulated that you can’t even have the word cab in your business’s name.

The Curated Arial Non-Orbital Navigation System

To solve these problems we would like to introduce the Curated Arial Non-Orbital Navigation System or CANON for short. CANON is the first non-traditional transportation system able to cover even the least densely populated cities cost effectively. CANON stations have a three to five days setup time and have a low footprint of around 5000 square feet each, the size of an average single family home lot. Many nodes will be distributed throughout the city and each node can transports a person to any other node in the city through the air avoiding traffic and the need to pool people into groups who are traveling the same path or direction.

The CANON’s non-orbiting transporters have dynamic weight distributors designed to avoid spinning and will travel through the air smoothly at high velocity, staying upright the entire time. Each CANON can rotate and launch a transporter quickly and percisely. The empty transporters can be launched to different areas in the city as the population shifts throughout the day.

The initial test phase has been completed successfully and we have just closed a round of financing partially funded by the government’s Alternative Public Transportation Fund. We are now piloting the project in the valley, and have a planned rollout to major US cities by 2018. The only small hurdles remaining are the semi-lethal issues dealing with the effects of rapid acceleration and deceleration, but we feel that innovation can’t happen without a modicum of risk. PagerDuty will continue to be our baby and we will still enjoy waking you up when sh*t breaks. But we hope that now we can also transport you quickly to your servers.

The picture below was taken as part of our initial testing, but was conducted before the idea for the non-orbiting transporters came into existence for softer landings.

Well, that’s the theory, anyway. Unfortunately, every time I’ve evaluated any AWS service beyond their simple EC2 hosting, AWS has come up short. Perhaps most frustrating, their services cover 95% of what we need. But without fail, they are lacking some small but critical piece of functionality.

Consider AWS’s elastic load balancer (ELB), for example. It provides an easy way to distribute traffic fairly over all of your front-end instances.  It can automatically stop routing requests to failed instances, completely hiding network and instance failures from the user. The ELB can even automatically spin up new instances in response to traffic spikes. All of this would take some serious engineering effort to replicate on your own.

Unfortunately, it’s totally unusable in many real-world deployments. The problem is that Amazon doesn’t assign static IPs to their load balancers. Instead, you get a hostname and are told to setup CNAME records aliasing www.yourdomain.com to the ELB’s name. This has three serious problems.

First, you can’t use a CNAME for the root of a domain.  This is because a CNAME record can’t coexist with a SOA record at the same point in the DNS hierarchy.  As a result, if your site is hosted at yourdomain.com, you’ll need to move it to www.yourdomain.com. Of course, even with redirects in place at the original domain, this sort of branding change is going to be unacceptable to many businesses.

Second, you can’t properly accept email to a domain hosted by an ELB.  This too is due to a DNS limitation — you can’t have a MX and CNAME record at the same point in the DNS hierarchy.  While you might be able to accept mail if you run a SMTP server on the machines behind the ELB, this is far from a typical configuration.  At PagerDuty, this is a showstopper, since we need to be able to both host a site and accept mail at yoursubdomain.pagerduty.com.

Finally, you have no “out” if the ELB blows up, short of adjusting your DNS records and waiting for cached records to expire. This is a big problem for us, since we’re very hesitant to introduce components into PagerDuty’s infrastructure that we can’t quickly swap out in the event of a problem.

The solution to this problem is simple — it should be possible to map an Amazon Elastic IP to an ELB. Since the ELB would now have a static IP, the DNS issues would be solved. And if the ELB blew up, you could simply provision another and remap the IP — no DNS changes required. I realize that ELB’s “no static IP” architecture is probably a deeply baked in design decision — but unfortunately, a LB without a static IP isn’t really usable.

Anyway, we plan to use this blog to keep our customers (and curious onlookers) up to date regarding the development of PagerDuty, and our experiences running a startup out of Toronto.

Just in case you’ve randomly stumbled across our blog, PagerDuty is an alert management product we’ve been working on since February. Most web developers are pretty familiar with monitoring systems like Pingdom and Nagios. These tools are indispensable for any serious online business: they rapidly detect problems with your systems. What they don’t do as well, though, is ensure that someone knows about a detected problem.

With PagerDuty, we’re trying to bring top-notch alerting functionality to the monitoring tools IT pros have already come to trust. We help you establish an on-call rotation for your IT/ops team, and then dispatch alerts to the on-call engineer using your choice of phone calls, SMSes, or emails. PagerDuty’s email-based triggering makes integration with all of your existing monitoring software a snap.

If you haven’t already, take a look at PagerDuty. The service is completely free for the next little while (during our Beta period), so sign up for an account and let us know what you think.