Decide how resilient you need to be before building resilience

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:02:11

   without those building blocks because like i talk about the first chapter of the book nobody says i don't want my system to be resilient yeah no no one but at the same time resiliency is not a binary property it's not like off or on it is a bit like observability it's a degree so then the question is how much do you need because am i having a conversation with you about multi-cloud and, you know, doing complete rewrites of systems for critical stand-ins like Monzo do, for example.

2. 1:02:45

   Yeah, they have a stand-in bank when they go down. They have a whole stand-in bank. So it's sitting there, it's costing them money for the most part, but then just a few weeks ago, their main application went down for a short period of time and they activated that bank, which I'm assuming it costs them millions of pounds or dollars per year just to keep it there. It's not even a slight sidebar, but so firstly, their standby bank It has a real stripped-down list of features.

3. 1:03:09

   It's much more bare-bones, but it's all the core ones. Like, you can freeze cards, you can take money out, you can check your balance. It's a complete rewrite, so it's no shared code because they're worried about a bug. Different cloud vendor. It's also, by design, significantly more cheap to run. So they've got thousands of microservices on the full stack. This is about a tenth the cost of the cloud cost.

4. 1:03:29

   They also run it constantly. So it's always there. But, like, Monzo, know they're Monzo, right? That's how that works. but like there's no point you starting there so actually the reason i put the observatory stuff in is like you need the raw data but also you need to make decisions about this comes back to what good looks like then you need to know okay do i need to worry about multi-cloud vendor dr strategies because that's where i need to be going or am i somewhere else and so i think without that kind of sense of your north staff how resilient do we need to be without that defined, you don't know how deep you need to get. This is why I'm trying to say how secure does my system need to be?

5. 1:04:09

   Have you done a threat model? If you haven't done a threat model, I don't know. And I guess to answer this question of how resilient do I need to be, it greatly helps, and maybe it's actually a requirement to do an informed decision, is how much will it cost me in terms of operating costs, in terms of time to build it, in terms of, for example, latency. Just a simple example is if you say, like, okay, the business is like, oh, would you like multi-cloud?

6. 1:04:31

   And the business is like, yeah, we'd love multi-cloud. like if amazon goes down because we heard they had an outage last week or a few months ago i actually want to be able to run google cloud but then you'll be like okay well first of all this will mean we're paying two cloud bills and if we want full replication it might increase our latency because we need to wait for to write for both of them which might You know, customers might see things slower. Anyway, there's all of these trade-offs. And in my experience, like oftentimes, like you might retrofit that, but if you can have a discussion upfront, that's great. But to have this discussion, you need to know what the trade-offs are. You need to know what's happening in all these dimensions that, you know, you care about, like how long will it take to build? How complex will it be? How much money will it cost to operate?

7. 1:05:16

   But also as technologists, we can't just say, oh, we have also got to get good at communicating with our business stakeholders, with our end users. Oh, yeah. What percentile latency do you want? Like, some developers know what I say when I say that. Not all, but some. How many, you know, experts in their field who are great product owners or subject matter experts but are not software developers would know what I meant by that?

8. 1:05:42

   And so we also have to understand the business domain, the product domain, well enough to do the interpretation and help and to bridge that gap. And it is a dynamic thing as well. I mean, this is the whole thing about resiliency in general, right? A resilient system is one that changes. It changes and it evolves. If you don't change, it evolves. You're a brittle system. You're not a resilient system.

9. 1:06:03

   And so even those targets, that conversation you have to have about what is good enough does change. but like think about something like in the case of say the us east one outage towards the end of last year um there were lots of on aws on aws as a aws us east one region went down it was the biggest region big region lots of people there first because that's often where new services got launched historically when that went down it caused a lot of people to say do we need to be multi-region and some of those companies would have had the conversation before and decided no we don't and then they've seen the fallout and they may have changed their minds and that is fine this is what you mean by a resilient system evolves yeah thanks to the real world but also like it is about what problems you're not going to

10. 1:06:48

    try and solve there were some big name companies that got taken out when US East 1 went down and got severely had severe degradation in terms of what they could offer companies like Zoom for example they weren't taken off entirely but they were severely affected do we think that people working at Zoom are idiots? no Do we think they knew that if a cloud region went down, they'd be impacted? Yes. They probably looked at that and said, well, for us to come out with a multi-region approach, it's going to have this impact.

11. 1:07:15

    It's going to cost us this much money. It's not worth it. Let alone talking multi-cloud. Because multi-cloud versus multi-region is a whole different conversation. And that's okay. And I think as long as you've had the conversation, and as long as, this is the other secret bit of it, of course, is as long as what you think you need to do is in tune with what your users think it needs to be doing. This is why I make the point in the book when talking about the difference between SLAs and SLOs.

12. 1:07:40

    The SLA is what you're going to get sued for if you breach. If you only ever deliver to the SLA, your customers are rarely ever happy. You've got to do more than that, right? So there is the social contract there as well.