Building resilient systems with Sam Newman

Summary · 4 min read

Sam Newman wrote Building Microservices and says he was in the room when the term microservices was coined. He worked at ThoughtWorks for years, tried three startups, and has worked independently since. His new book, Building Resilient Distributed Systems, is out now.

Newman says microservices fit best when an organization wants team autonomy, and that independent deployability is their defining trait. His three rules of distributed systems are that data takes time to travel, that remote services are sometimes unavailable, and that resource pools are finite. He argues resilience is a matter of degree, set by business context and informed by observability, idempotency and learning from incidents. On AI, he favors hedging across vendors and models, keeping humans connected to the system, and trying software factories inside low-risk modules.

Minute by minute

1. 8:58

   ThoughtWorks and programming as collaboration. Newman describes his years at ThoughtWorks: extreme programming, build pipeline work from about 2004, and a stint at Google teaching automated testing. He says programming is not typing. He describes a program as a theory held by a team, with code a side effect of working together, and says AI does not change that.

2. 20:50

   How microservices emerged from SOA. Newman says James Lewis coined the term after studying companies with fine-grained service-oriented architectures. Newman came to it from continuous delivery and infrastructure automation. He started writing his book in that context. He says Uber and Netflix brought attention, while many firms simply relabeled their SOA as microservices.

3. 26:12

   Microservices are an architecture of last resort. Newman's definition rests on independent deployability and, by default, service boundaries drawn around the business domain. He calls the style a last resort because it pushes state into a more distributed form. His best reason to adopt it is organizational autonomy. Enterprise domains are more complex than those of tech companies.

4. 33:42

   Specs versus code as source of truth. Newman says the industry is debating whether the spec can replace code as the definition of a system, and he does not think everywhere is ready. He cites tacit knowledge, lost shared context and the end of code review as risks. Teams must define what good looks like, and he says production is the real truth.

5. 44:38

   Why he wrote the resilience book. Newman says a paper on resilience engineering baffled him at first. He then saw that many people were building distributed systems anyway, so he wrote a book of foundations such as observability and timeouts. It also covers socio-technical ideas. He praises candid public post-incident reports, with Cloudflare as an example.

6. 51:42

   Three rules of distributed systems. Newman reduces distributed systems problems to three rules. Information cannot travel instantly. The thing you want to call is sometimes not there, as with a fire or rabbits eating cables. Resource pools are not infinite, and he says most outages come down to saturation, often through retry storms.

7. 55:39

   Observability sets how resilient you need to be. Newman says a distributed system must emit structured events, which build into traces, service level indicators and objectives. Resilience is not binary, so teams need that data to decide how much to invest. He cites Monzo's stripped-down standby bank and Zoom's multi-region choice. He adds that engineers must communicate with business stakeholders.

8. 1:07:06

   Idempotency keys make retries safe. Newman explains that an idempotent operation can run repeatedly with one effect, and uses payments as the example. His preferred fix is a unique key generated by the client, such as a UUID, though both sides must support it. A server-side fingerprint of the request fields can be retrofitted, but it risks rejecting legitimate duplicates.

9. 1:15:06

   Thundering herds have different mitigations. Newman defines a thundering herd as too many requests for too few resources. His examples are denial-of-service attacks, cache collapse, and Square's 2017 retry storm against Redis. He also cites healthcare.gov and Twitter, where success caused the overload. Autoscaling suits some cases but can turn an attack into a cost problem.

10. 1:20:57

    Business context decides fail open or closed. Newman says the choice to fail open or closed depends on the business. An e-commerce shop can sell an item it may lack and refund later, while a concert ticket seller cannot. Orosz adds that Uber moved from failing open to failing closed as its focus shifted to profitability. Newman says the decision is made feature by feature with product owners.

11. 1:25:06

    Woods's four concepts of resilience. Newman describes four concepts from David Woods: robustness, rebound, graceful extensibility and sustained adaptability. Most engineers focus on robustness, which adds complexity of its own. Rebound is about recovery time, graceful extensibility is about handling surprise, and sustained adaptability is about learning over time. The last two depend on culture and psychological safety.

12. 1:32:24

    Hedging against AI vendor risk. Newman says conservative economist analyses show AI labs need $2.6 trillion in revenue by 2030, against a $1.4 trillion software market. He advises being multi-vendor and multi-model and splitting agents into workflows that can use deterministic code. He argues LLMs have no concept of causality, so teams should decide where to use them.

13. 1:39:05

    Cognitive debt and cognitive surrender. Newman describes cognitive debt as eroding critical thinking, and as a lost shared mental model when people work separately with AI. Cognitive surrender is approving AI output without reading it, much like approving a pull request without review. He contrasts AI that assists an expert with AI that turns the expert into an approver.

14. 1:44:18

    Modular architecture for experimenting with AI. Newman recommends modular systems where humans design the modules and their connections while AI works inside them. Teams can try a software factory in one low-risk module with specs and verification, and keep a collaborative approach elsewhere. He says you need to define good across developer, operational and business requirements.

Published Oct 7, 2026.