Tag Archives: john

Approaching the Hiring of Engineers as a Machine Learning Problem

Hiring software engineers is hard.  We all know this.  If you get past the problem of sourcing and landing good candidates (which is hard in itself), the whole issue of “is this person I’m talking to ‘good enough’ to actually … Continue reading

FacebookTwitterGoogle+
Posted in Best Practices, Blog | Tagged , , , , , , | 1 Comment

On-call best practices: Page your manager

Having one person on-call isn’t enough. What happens if your on-call engineer sleeps through their alert? What happens if their phone’s battery dies without them knowing, or if they get an alert at a really inconvenient time, like when stuck … Continue reading

FacebookTwitterGoogle+
Posted in Best Practices | Tagged , | 2 Comments

Pressure Release Valves

This is the fourth in a series of posts on increasing overall availability of your service or system. Have you ever gotten paged, and known right away that this problem isn’t like the last 15 operations issues you’ve dealt with … Continue reading

FacebookTwitterGoogle+
Posted in Availability | Tagged , , | Leave a comment

A Standard Operating Procedure for when s*IT hits the fan

This is the third in a series of posts on increasing overall availability of your service or system. In the first post of this series, we defined and introduced some concepts of system availability, including mean time between failure – MTBF … Continue reading

FacebookTwitterGoogle+
Posted in Availability | Tagged , , | Leave a comment

More control over Optimistic Locking in Rails

Like pretty much everything else in Rails, optimistic locking is nice and easy to setup:  you simply add a “lock_version” column to your ActiveRecord model and you’re all set.  If a given Rails process is trying to update some record, … Continue reading

FacebookTwitterGoogle+
Posted in Code | Tagged , , | Leave a comment

Availability lessons from shoe companies and ancient warlords

This is the second in a series of posts on increasing overall availability of your service or system. In the first post of this series, we defined and introduced some concepts of system availability, including mean time between failure – … Continue reading

FacebookTwitterGoogle+
Posted in Availability | Tagged , , | 1 Comment

Getting the most out of PagerDuty: Incident De-Duping

Tired of getting a flood of PagerDuty incidents whenever a problem occurs with one of your systems?  Do many of the incidents seem identical?  Do you spend valuable time trying to fend off the seemingly never-ending PagerDuty phone calls and … Continue reading

FacebookTwitterGoogle+
Posted in Best Practices, Features | Tagged , | 1 Comment

New APIs Available Now

Have you ever said to yourself: “PagerDuty is great, but I wish I could better integrate it into the custom tools I already use.” Or maybe: “Why can’t I see more reports on the number of incidents each of my … Continue reading

FacebookTwitterGoogle+
Posted in Announcements, Features | Tagged , | 7 Comments

Standing on the shoulders of giants and stumbling with them – the Amazon AWS outage’s “pain” statistics

Today, at around 1am Pacific Time, Amazon began having major problems with some of their cloud infrastructure: specifically with their EC2, EBS, and RDS offerings. We’d like to share some statistics on the alerts we sent out – via phone or SMS – during the outage. Continue reading

FacebookTwitterGoogle+
Posted in Announcements, Blog | Tagged | 14 Comments

The ups and downs of Availability

This post is meant as a quick introduction to some concepts of system availability, so that subsequent posts in this series make sense. I’ll go over concepts like availability, SLA, mean time between failure, mean time to recovery, etc. Continue reading

FacebookTwitterGoogle+
Posted in Availability | Tagged , , , , | 5 Comments