Distributed systems, explained

A distributed system is any system whose parts run on more than one machine and coordinate over a network. By that definition, almost every modern application qualifies: a mobile app talking to an API, which talks to a database, a cache, a payment provider and a messaging service. Each arrow in that diagram is a network call, and each network call can be slow, fail, or succeed without you finding out.

We distribute systems for good reasons: to handle more load than one machine can, to stay available when a machine fails, and to serve users closer to where they are. The price is complexity. This post covers why distributed systems are hard, the core ideas that make them manageable, and the patterns we use to build reliable ones.

Why they are hard

Peter Deutsch and colleagues at Sun Microsystems famously listed the fallacies of distributed computing: assumptions that feel true on a laptop and break in production. A few of them account for most of the trouble we see:

• The network is reliable: Messages get lost, delayed, duplicated and reordered. Mobile and rural connections make this the normal case, not the exception.
• Latency is zero: A call that takes microseconds in memory takes milliseconds across a data centre and far longer across continents. Chatty designs that ignore this become slow designs.
• Topology does not change: Servers are added, replaced and moved constantly, especially in the cloud. Anything that hard-codes where things live will eventually break.

The deepest difficulty is partial failure. On one machine, a program either works or crashes. In a distributed system, some parts can fail while others carry on, and a timeout cannot tell you whether the other side never got your request or did the work and failed to reply.

The ideas that tame them

1. Consistency and availability: When the network splits, a system must choose between refusing some requests and risking that different parts disagree for a while. This trade-off, usually described through the CAP theorem, is not a one-time choice: a bank balance and a product review can sensibly make different decisions.
2. Replication: Keeping copies of data on several machines provides durability and read capacity, but every write now has to reach several places. How many copies must confirm a write before it counts is one of the most important settings in any database.
3. Consensus: Algorithms such as Raft and Paxos let a group of machines agree on a single value or leader even when some members fail. You rarely implement them yourself, but the databases and coordination services you depend on use them, and their limits shape your system’s.
4. Time and ordering: Clocks on different machines drift, so “which happened first” is harder than it looks. Logical clocks, sequence numbers and careful use of timestamps keep events in a sensible order.

Patterns we rely on in production

• Idempotency: Design operations so that doing them twice has the same effect as doing them once. Payment and messaging callbacks can arrive late, more than once or not at all; an idempotency key turns a dangerous retry into a safe one.
• Timeouts, retries and backoff: Every network call needs a timeout. Retries should back off and add jitter, so a struggling service is not flattened by a wave of identical requests.
• Queues and asynchronous work: Putting a durable queue between components lets each side work at its own pace and survive the other’s downtime, and smooths out sudden spikes in traffic.
• The outbox pattern: Writing a change and the message announcing it in the same database transaction, then publishing from that outbox, avoids the classic bug of saving data but losing the event, or the reverse.
• Offline-first clients: For field teams and customers on unreliable connections, letting apps work locally and sync when the network returns is often the difference between a system people use and one they work around.
• Observability: Structured logs, metrics and distributed traces that follow a request across services are how you find out what actually happened, instead of guessing.

Start simple, distribute deliberately

The most reliable distributed system is often the one with fewer moving parts. A well-structured application on a properly configured managed database will outperform an elaborate web of microservices for most organisations, and be far easier to run. Distribute when there is a concrete reason: load a single machine cannot handle, availability targets one region cannot meet, or teams that need to ship independently.

When we do distribute, we make failure a first-class design input. We ask what happens when each dependency is slow, down or wrong, and we build and test for those cases before they happen in production, not after.