ChefConf 2014: How to Mock a Mockingbird
Chef is a powerful operations tool – its flexibility and automation capability make it extremely popular in organizations with extensive service deployments. Our own Ranjib Dey was invited to speak at Chef’s annual confab, ChefConf, which took place in April. Here are some of the highlights of Ranjib’s talk, “How To Mock a Mockingbird” (named after a book on mathematical philosophy).
Assume that things will break
“Design for failure” may be a more concise way of expressing this point. It’s a fundamental piece of the PagerDuty story – PagerDuty was created around the principle that crises are inevitable for all operations teams. Our role is to make handling those crises a little more pleasant.
Ranjib suggests that developers accept failure as a given. What DevOps teams can do to mitigate failure, Ranjib goes on to say, is strive to make failures as “inexpensive and isolated” as possible.
Design matters
If planning for failure is the “why” in Ranjib’s discussion (as in, “Why should I take Ranjib seriously?”), code design is the “how”. In an environment where things break, mitigation-by-design is the best defense against downtime.
What this means in practice, Ranjib says, is to minimize the code you deploy. Not only does greater code complexity make testing more difficult, but leaner code can be replaced more quickly should bugs present themselves (which, again, they inevitably will).
“Anything that is complex in theory will be difficult to test.”
Test intelligently
How you validate your codebase is linked closely to the way the code actually looks. That is, your testing processes will depend on the strength of the code’s design.
Your code should work well in happy-path scenario tests, for example. Happy-path scenarios, a.k.a. acceptance testing, measure code performance in ordinary use cases.
Code design is truly validated when your services are placed under stress – and the best way to do this is unit testing. Not only is unit testing very fast, Ranjib notes, it can capture 80 percent to 90 percent of errors (in an ideal scenario). That makes it a particularly smart validation strategy against acceptance testing, which, while comprehensive, is slow (and comes with a steep learning curve).
Further supporting the value of unit testing, Ranjib goes on to say, is the role it plays in enforcing codebase consistency. Rapid-fire unit tests help maintain conventionality in code design; for this reason, unit testing should be viewed as a foundational testing methodology.
“Unit testing is invaluable for long-term maintainability.”
Communicate for best results
We are evangelists for unit testing because it empowers devs to manage their code from end to end (with support from ops). In an environment where devs and ops work towards shared goals, collaboration is hugely important – and in order for collaboration to thrive, everyone has to communicate.
With communication a critical piece of DevOps culture, Ranjib says, breaking down knowledge silos should be made a priority. Good design is the principal element in healthy codebases (as stated in Ranjib’s first point), and design cannot flourish if teams aren’t sharing with each other what they know.
“Communication will influence design.”
Get comfortable with complexity
Illustrating the importance of communication were two of Ranjib’s slides – one depicting a typical reference code topology and the next showing a picture of a plate of spaghetti. Reference topologies, Ranjib argues, fail to capture all of the dependencies and ops services that deployments contain. Real topologies, he goes on to say, look a lot more like that mass of spaghetti.
Quite simply, modern codebases are going to be too complex to communicate simplistically. And the most fault-tolerant topologies will be scale-free – that is, contain a plurality of nodes and spokes such that no single element is “linked” dependently to another.
The idea of not knowing how everything in your codebase is connected (and being OK with it) was put forward by Netflix site reliability engineer Corey Bertram when he stopped by PagerDuty HQ in March. Corey’s talk illustrated the nature of deployments, suggesting that embracing a Zen attitude is the only logical response to their wild complexity.
“Nobody knows exactly what a fault-tolerant network architecture should look like.”