A design diagram shows two Availability Zones, a standby database and a load balancer. The document says the service is highly available.
Then an entire failure domain disappears, and the standby service has nowhere to run.
A recovery plan is a hypothesis until its essential steps have been exercised. It needs more than replication. It needs capacity, reachable dependencies, operating procedures and a way to decide when the business can trust the recovered service.
This is a practical SRE guide to evaluating the recovery path, not a claim about a specific customer's undisclosed outage.
Availability and disaster recovery answer different questions
High availability attempts to keep service running through expected component failures. Disaster recovery addresses larger failures that can require restoring data, moving workloads or running a more manual process.
An architecture can support both, but the controls differ.
Two replicas do not provide useful redundancy if they share a failing dependency. Two data copies do not prove that either can be restored. A standby in a different region does not prove the application can authenticate users, fetch secrets, resolve DNS or connect to essential external APIs from that region.
The service boundary matters as much as the compute topology.
Define what recovery success actually means
Recovery Time Objective (RTO) is a business target for how long a service can remain unavailable. Recovery Point Objective (RPO) is the targeted maximum data loss measured in time. Neither becomes a guaranteed result because it appears in a runbook.
For a stateful application, success may require several independent conditions: write operations are accepted, payment callbacks or event processing are functioning, reconciliation is correct, and the system can return to routine operations without losing or duplicating work.
A status page turning green is not necessarily the same thing as application-level recovery.
A realistic test should therefore include actual user journeys, data consistency and downstream effects rather than just checking that a container started.
Spare capacity is part of the recovery design
Consider a multi-zone Kubernetes cluster that normally runs near its resource limits. If one zone is lost, can the remaining zones host its workloads?
If not, recovery may depend on provisioning new capacity. That is a control-plane dependency, and the needed instance type may not be immediately available during a wider incident.
The same question applies to a secondary region. "Autoscaling enabled" is not evidence that emergency capacity will exist when it is needed.
An SRE readiness review should identify:
- Which workloads must survive the loss of a zone or region?
- How much spare capacity exists in the remaining failure domains?
- Can the scheduler place all critical Pods under the active node affinity, taints and disruption constraints?
- Does the database retain sufficient serving capacity and healthy replication state?
- Which external dependencies still route through the failed region?
- What happens if the cloud API needed to create replacements is unavailable?
You may choose to keep warm capacity, reserve it, or accept a slower recovery. Those are business decisions linked to measured recovery objectives, not hidden defaults.
Backups and replicas fail differently
A replica can propagate accidental deletion and application corruption. A backup can be stale, inaccessible or unusable if its restore keys and control plane are unavailable.
Recovery tests should include the ability to create a clean environment and restore an application-consistent data set. Check the catalog, credentials, encryption keys, retention rules and storage permissions before a real incident forces you to.
For PostgreSQL, one might validate a point-in-time recovery path under the actual backup-and-WAL mechanism used by the deployment. Merely seeing WAL archive objects in object storage does not prove the database can be restored to an intended point.
This is a proposed verification method, not evidence that a particular BBB client achieved a specific RPO.
Failover should be a measured experiment
A low-risk initial drill can start with one controlled dependency failure in a non-production environment or a carefully approved production scenario.
Before the drill, identify the customer-visible path, observability checks and a stop condition. Record when the fault was introduced, when traffic was diverted, when the application became functionally useful and when the incident response team confirmed its data state.
Repeat it with at least one dependency that is not a Pod or VM: DNS, identity, secret store, message broker, database promotion, external callback service or cloud API.
Measure the gaps. If the exercise fails, capture what was missing rather than grading the design from the diagram.
The Google SRE workbook and official cloud reliability frameworks provide useful general context, but they cannot validate a specific deployment without actual tests. Faster AI-orchestrated intrusion workflows make rehearsed containment, credential rotation and restoration even more important.
Returning to normal is another failure-prone operation
Failing over and failing back are separate processes. After the original region recovers, the new primary may hold writes that the old environment does not. Traffic, DNS and message consumers may have changed position.
An automatic failback can be more dangerous than remaining on the recovered environment.
A credible plan says who authorizes failback, how state divergence is checked and how the team validates service behavior after restoring normal traffic paths.
Turn the drill into a reliability backlog
A recovery test should produce evidence, owners and next actions:
- A measured recovery timeline, separated from the aspirational RTO.
- A record of data consistency and the observed recovery point.
- Dependencies that blocked or delayed the exercise.
- Configuration changes, capacity reservations or runbook repairs required.
- A proposed repeat test after each important remediation.
A report that only says "passed" is weaker than an exercise that discovers a hidden dependency and then proves it has been fixed.
The result buyers should ask for
The most useful deliverable from an SRE or disaster recovery engagement is a repeatable demonstration that a critical service can recover under defined failure conditions.
If your current plan has never tested regional capacity loss, database restoration and the application path end to end, those are the places to start. Big Bad Byte's DevOps/SRE engineering and infrastructure assessment work can turn recovery assumptions into operational tests.