Marc Brooker's Blog

Quorum Availability It's counterintuitive, but is it right?

Getting Big Things Done In one particular context.

Consensus is Harder Than It Looks And it looks pretty hard.

Focus on the Good Parts Skepticism and cynicism can get in your way.

Surprising Economics of Load-Balanced Systems The M/M/c model may not behave like you expect.

A Story About a Fish Nothing's more boring than a fishing story.

Code Only Says What it Does Only loosely related to what it should do.

Some Virtualization Papers Worth Reading A short, and incomplete, survey.

Reading Research: A Guide for Software Engineers Don't be afraid.

Two Years With Rust I like it.

Firecracker: Lightweight Virtualization for Serverless Applications Our second paper for NSDI'20.

Physalia: Millions of Tiny Databases Avoiding Hard CAP Tradeoffs A few years ago, when I was still working on EBS, we started building a system called Physalia.

Why do we need distributed systems?

Kindness, Wickedness and Safety We must build kind systems.

When Redundancy Actually Helps Redundancy can harm more than it helps.

Is Anatoly Dyatlov to blame?

Some risks of coordinating only sometimes Sometimes-coordinating systems have dangerous emergent behaviors A classic cloud architecture is built of small clusters of nodes (typically one to nine1), with coordination used inside each cluster to provide ava…

Learning to build distributed systems A long email reply A common question I get at work is “how do I learn to build big distributed systems?”.

Control Planes vs Data Planes Are there multiple things here?

Telling Stories About Little’s Law Building Up Intuition with Narrative Little’s Law is widely used as a tool for understanding the behavior of distributed systems.

Availability and availability Translating math into engineering.

Balls Into Bins In Distributed Systems Throwing things can be fun.

Is the Mean Really Useless?

Why Must Systems Be Operated?

Heuristic Traps for Systems Operators What can we learn from avalanche safety?

Is there a CAP theorem for Durability?

CALISDO: Threat Modeling for Distributed Designs Some steps towards a mnemonic threat model for distributed systems.

Sodium Carbonate, and Ramenized Pasta Doing chemistry with baking soda, cabbage and an oven.

The Zero, One, Infinity Disease Numbers are important.

How Amazon Web Services Uses Formal Methods Now in CACM.

Jitter: Making Things Better With Randomness Jitter is a good thing.

Electoral Trouble in Sybilania An Small Town Struggles To Achieve a Fair Vote.

Does Bitcoin Solve Byzantine Consensus?

A Quiet Defense of Patterns Twenty years late to the party.

Make Your Program Slower With Threads How much do context switches matter?

Two Farmers and Common Knowledge A legislative solution to a technical problem.

Exactly-Once Delivery May Not Be What You Want It's hard to get, but that's OK, because you don't want it.

Ice Cream and Distributed Systems Can we serve a fair amount of ice cream?

Harvest and Yield: Not A Natural Cure for Tradeoff Confusion Comments on a 15 year old paper.

The Essential Barbara Liskov Some of my favorite Barbara Liskov publications.

The Space Between Theory and Practice in Distributed Systems How do we learn synthesis?

Use of Formal Methods at Amazon Web Services How we're using TLA+ at AWS Late last year, we published Use of Formal Methods at Amazon Web Services about our experiences with using formal methods at Amazon Web Services (AWS).

CAP and PACELC: Thinking More Clearly About Consistency CAP is confusing.

Two traps in iostat: %util and svctm These commonly-used fields in iostat shouldn't be commonly-used. iostat, from the excellent sysstat suite of utilities, is the go-to tool for evaluating IO performance on Linux.

The Operations Gradient: Improving Safety in Complex Systems Can we improve the safety of complex systems by listening to operators more?

Viewstamped Replication: The Less-Famous Consensus Protocol The first practical consensus protocol may be the least famous.

The Essential Nancy Lynch Some of my favorite Nancy Lynch publications.

Failure Detectors, and Non-Blocking Atomic Commit Non-blocking atomic commit is harder than uniform consensus.

The Essential Leslie Lamport Some of my favourite Leslie Lamport publications.

Snark, Chord, and Trust in Algorithms Good journals, well-known authors and informal proofs are not sufficient.