Kubernetes is an open-source system for deploying, scaling and managing containerized applications. It reconciles the desired state declared through its API: scheduling Pods, maintaining replicas, routing traffic and restarting failed workloads. Production success depends on what that desired state omits as well as what it contains.
A cluster can run while releases fail, stateful data is lost or upgrades stall. This guide covers the workload, operational and recovery decisions that Kubernetes does not make for you.
Managed control plane does not mean managed application
EKS, GKE and AKS can run and upgrade parts of the control plane. RKE2, Rancher and self-managed clusters leave more of that responsibility with the team. In every model, the application owner still has to handle:
- Resource requests, limits and scheduling.
- Readiness, startup and deployment behavior.
- Stateful workloads and data.
- Application configuration and secrets.
- Rollouts, upgrades and rollback.
- Observability and incident response.
- Cost, capacity and recovery.
Start by recording who owns each layer. A cloud provider incident and an application incident can look identical to users even when their responsible teams are different.
Begin with a production workload inventory
For each workload, record:
- Replicas and criticality.
- CPU, memory, disk and network requirements.
- Stateless or stateful behavior.
- Startup time and dependencies.
- Data storage and consistency requirements.
- Deployment and rollback behavior.
- Autoscaling signals and peak patterns.
- Owners, alerts and runbooks.
This inventory exposes where a generic Helm chart or copied resource values will fail. A broker, database, RPC node and stateless web API should not receive identical disruption and scaling settings.
Resource requests and limits affect both reliability and cost
Requests inform scheduling and capacity planning. Limits constrain what a container may use. Copying them between services creates predictable incidents: some pods waste capacity while others are throttled or evicted.
Review:
- Actual p50, p95 and peak usage over representative periods.
- CPU throttling and memory eviction behavior.
- JVM and runtime memory relative to container memory, not only heap size.
- Node allocatable capacity and system-reserved resources.
- HPA and VPA behavior against the actual scaling goal.
- Whether autoscaling reacts before users see degradation.
A larger limit does not automatically make a workload faster. A request that is too high can make scheduling and autoscaling unnecessarily expensive. Test changes against the real workload.
Probes determine whether a broken release receives traffic
Kubernetes uses different probes for different decisions:
- Liveness: should this container be restarted?
- Readiness: should this pod receive traffic?
- Startup: has a slow-starting application finished initialization?
A readiness probe that checks only an open port can mark a broken release ready while real requests fail. That lets a rollout replace healthy pods with failing ones. A liveness probe that depends on a fragile external service can restart healthy containers and amplify an outage.
Probe acceptance should test actual dependencies at the right level, avoid unnecessary external fragility and use startup probes for slow initialization. Validate probe behavior under load, not only with a single manual request.
Rollouts need a tested failure path
Rolling updates replace pods gradually according to deployment settings and readiness. Do not assume that a stalled Deployment automatically restores the previous working version. A progress deadline can mark a rollout as stalled while the failed configuration remains in place.
Review:
maxSurgeandmaxUnavailableagainst capacity and availability requirements.progressDeadlineSecondsand alert behavior.- Readiness and startup behavior during the rollout.
- Revision history and a tested rollback command.
- Database migrations and backward compatibility between application versions.
- Traffic shifting and automated abort behavior when progressive delivery is used.
Acceptance should include a deliberately broken non-production release: prove it does not take down healthy capacity and that rollback restores service.
Stateful workloads require different rules
Databases, brokers, caches and blockchain nodes have identity, storage and sometimes quorum. Treating them as interchangeable replicas can lose data or availability.
For StatefulSets and stateful operators, review:
- Stable identity and ordered rollout requirements.
- Storage class performance and reclaim policy.
- PersistentVolumeClaim lifecycle and what deletion means.
- Topology spread and anti-affinity.
- PodDisruptionBudgets against quorum requirements.
- Node-drain procedure and the number of members that may be unavailable at once.
- Backup, restore and data consistency after recovery.
A PodDisruptionBudget is not a guarantee of availability if two quorum members can still be drained or fail together. Test the failure scenario instead of reading only the manifest.
Storage and recovery are application concerns
Kubernetes can mount and reattach storage, but it does not prove that the application recovered to a consistent state.
Record:
- Which storage class and performance profile each workload uses.
- Whether data remains after pod, PVC or node deletion.
- How snapshots and backups are created and validated.
- How the application behaves while a volume is detached or restored.
- Which region or provider holds the recovery copy.
- How long restoration and reattachment take.
An application-level restore test should validate data and user journeys, not just the existence of a new PVC.
Autoscaling has more than one dimension
Horizontal Pod Autoscaling changes replicas. Vertical scaling adjusts resource requests. Cluster or node autoscaling changes infrastructure capacity. Karpenter, KEDA and HPA/VPA can help where they fit, but they do not remove the need for workload evidence.
Check:
- Whether the scaling metric correlates with user demand.
- How long nodes and application replicas take to become ready.
- Whether a downstream database or queue can absorb the new replicas.
- Whether scale-down removes capacity too quickly.
- Whether Spot or interruptible capacity is appropriate for the workload.
- Cost per successful unit before and after scaling.
Autoscaling can increase cost without improving throughput when the shared bottleneck is elsewhere.
GitOps needs trust boundaries as well as automation
GitOps records desired configuration in Git and lets a controller such as Argo CD reconcile it. This creates a clear audit trail, but write access to the repository or controller can become production access.
Review:
- Who can change deployment repositories and approve changes.
- Which clusters and namespaces each controller may modify.
- How drift is detected and handled.
- How secrets reach workloads without being stored in ordinary Git values.
- How controller failure and rollback are handled.
Kargo can coordinate promotion flows where used, but promotion still needs acceptance criteria and ownership. A Git commit is not proof that the new state is healthy.
Our External Secrets guide covers ownership, refresh and failure modes when secrets are managed outside the cluster.
Network policy and identity limit the blast radius
Every pod does not need access to every other pod or to the internet. Use workload identity, RBAC, network policy and egress rules to reduce what a compromise can reach.
Check:
- Service accounts and their actual API permissions.
- Default-deny or appropriately scoped network rules.
- Egress to databases, metadata services and external systems.
- Public ingress, load balancers and administrative endpoints.
- TLS, certificates and renewal.
- Cloud IAM mapped to Kubernetes identities.
Scanning an image is useful, but runtime identity and exposure determine what happens next. The container security boundary article explains that distinction.
Observability should connect cluster and workload signals
A useful investigation can move between user experience and the underlying cause:
- Application latency, errors and dependency behavior.
- Pod readiness, restarts and eviction.
- CPU throttling, memory pressure and node capacity.
- Deployment, scaling and drain events.
- Stateful workload health and storage behavior.
- Cloud load balancer, network and external dependency signals.
Alerts should identify the responsible team and link a runbook. Do not alert on every threshold simply because the metric exists. Include deployment and scaling events so a responder can tell whether behavior changed at the same time as an incident.
Upgrades include the add-ons and the workloads
A cluster upgrade can affect API versions, admission webhooks, CNI, CSI, ingress, service mesh, autoscaling, operators and application manifests. Upgrading only the control plane leaves those dependencies untested.
A safe sequence is:
- Record the current control-plane, node and add-on versions.
- Check release notes and removed or deprecated APIs.
- Validate CRDs, operators and workload manifests for compatibility.
- Test stateful behavior, rollout and restoration in a representative environment.
- Upgrade one supported step at a time with a rollback and recovery plan.
- Drain and replace nodes according to workload disruption constraints.
- Verify user journeys, telemetry and data after each step.
The Kubernetes EOL dependency guide covers why add-ons and version skew belong in the upgrade plan.
Recovery needs both cluster and application proof
A new cluster with old manifests is not automatically a recovered service. Recovery can require:
- Rebuilding control-plane and node infrastructure from code.
- Restoring secrets, configuration and deployment identities.
- Reattaching or restoring data and validating consistency.
- Restoring load balancing, DNS and external dependencies.
- Verifying critical user journeys and alerting.
- Returning to the original environment safely after failover.
Measure recovery time and data loss. The DR and failover guide explains why failback belongs in the exercise.
Cost comes from workload shape, not only node price
Kubernetes cost includes compute, storage, networking, control plane, observability and operations. Review:
- Requests versus actual use and idle allocatable capacity.
- Autoscaling and scale-down behavior.
- Stateful storage, replication and backup costs.
- Cross-zone and external data transfer.
- Logging and metrics volume.
- The operational cost of self-managing components.
A cheaper node pool can increase total cost if it reduces throughput or creates more incidents. Compare price per successful unit using the FinOps model.
A practical Kubernetes production review
Workloads
- Are requests, limits and probes based on real behavior?
- Are stateful storage, quorum and drain scenarios tested?
- Does a deliberately broken release fail safely and roll back?
Platform
- Are controller, repository, service-account and cloud permissions scoped?
- Are network and egress rules intentional?
- Do GitOps and secret handling preserve clear ownership?
Operations
- Do alerts connect workload and cluster evidence?
- Are autoscaling and scale-down tested under realistic load?
- Are add-ons and workloads included in upgrade tests?
Recovery and cost
- Have restoration and failback been measured?
- Are cost and successful performance compared per useful unit?
- Can the team operate and recover the platform without one specific person?
Big Bad Byte's DevOps and Kubernetes consulting covers these production concerns across EKS, GKE, AKS, RKE2 and self-managed clusters, with the code, dashboards and runbooks left in the client's own systems.