DevOps in production: CI/CD, Kubernetes, SRE and platform engineering

DevOps is the combination of engineering practices, automation and shared responsibility that moves software from a source change to reliable production operation. CI/CD makes delivery repeatable, infrastructure as code makes environments reproducible, and SRE adds measurable reliability targets and incident practice. Tools support those outcomes, but installing a tool is not the outcome itself.

This guide focuses on what has to work in production: delivery flow, Kubernetes safety, observability, security, recovery, cost and ownership.

DevOps is not a tool purchase or a job title

A team does not become DevOps by deploying Jenkins, Kubernetes or Terraform. Those tools can automate an unsafe or poorly understood process just as effectively as a good one.

A working DevOps system answers practical questions:

  • How does a reviewed change reach production?
  • What proves it is ready to deploy?
  • Which identity can deploy each component, and what else can that identity access?
  • How is the infrastructure reproduced and changed?
  • What tells users and engineers that production is degraded?
  • How does the team roll back, recover data and learn from an incident?
  • Who owns the platform after the initial project ends?

A single contractor can help answer these questions, but a team still needs shared access, documentation and operational knowledge. Otherwise the answers leave with that person.

Map the delivery path before automating it

Start with one representative change, from pull request to production and back again if necessary. Record the systems and trust boundaries it crosses:

  1. Source repository and review.
  2. Build and test runner.
  3. Dependency, artifact and container registries.
  4. Deployment identity and cloud account.
  5. GitOps controller or release automation.
  6. Kubernetes cluster, runtime and data stores.
  7. Telemetry, alerting and incident ownership.
  8. Rollback and recovery path.

This map exposes unnecessary manual steps and dangerous privileges. It also prevents a pipeline from being designed only around the successful path. A failed build, invalid artifact, unavailable dependency and rejected deployment should all have defined behavior.

Establish a delivery and reliability baseline

Before changing the process, measure enough to know whether it improves. Where the data exists, start with the four DORA delivery metrics and a small operational baseline.

Measure What it tells the team What to avoid
Deployment frequency How often production receives changes Increasing volume by shipping unchecked changes
Lead time for changes How long reviewed work takes to reach production Measuring only coding time and ignoring queues
Change failure rate How often a production change causes degradation or needs remediation Hiding rollbacks and hotfixes outside the definition
Failed deployment recovery time How long the service takes to recover from a failed deployment Measuring detection without recovery or user impact
Alert coverage and quality Whether important failures reach someone who can act Counting alerts rather than useful signals
Cost per useful unit Infrastructure cost per transaction, customer, environment or completed build Comparing only instance hourly prices

The definitions must be consistent. Two teams can report different deployment frequency while counting completely different events.

CI/CD: build once, verify and deploy the same artifact

Continuous integration checks and packages changes. Continuous delivery keeps the application releasable, while continuous deployment automates some or all releases. The exact boundary matters less than artifact integrity and a controlled promotion path.

A sound pipeline normally:

  1. Builds from a reviewed source revision.
  2. Runs appropriate unit, integration, security and performance checks.
  3. Produces an identified, immutable artifact or image.
  4. Records its source, test results and deployment eligibility.
  5. Promotes that same artifact through environments.
  6. Deploys with a scoped identity and an explicit rollback path.

Rebuilding independently for every environment can introduce differences that tests were supposed to rule out. A green pipeline should identify what exactly is green.

The runner is a security boundary

CI runners execute code and often handle registry, cloud and deployment credentials. Pull-request tests, release builds and production deployment should not share one implicit trust level.

Separate execution environments, cache scopes and identities. Prefer narrow, short-lived OIDC credentials over broad static keys. Test unauthorized actions with harmless canary resources rather than assuming the boundary works.

Our GitHub Actions runner security guide explains why ephemeral registration alone does not guarantee a clean execution environment. The runner cost analysis adds the other necessary dimension: compare cost per successfully completed build, not just the hourly VM price.

Infrastructure as code and GitOps

Infrastructure as code describes infrastructure in version-controlled files so environments can be reviewed and reproduced. Terraform, OpenTofu, Pulumi and Ansible are common implementation choices, but the repository structure and change process matter more than the brand.

A useful infrastructure change has:

  • An identifiable owner and review.
  • A plan or preview that people can understand.
  • Explicit state and secret handling.
  • A way to detect drift from the intended configuration.
  • A rollback or recovery procedure for failed changes.

GitOps extends that model to deployment: Git records the desired state and a controller such as Argo CD reconciles it. The controller needs boundaries too. Write access to a deployment repository can become production access, so reviews, branch protection and identity scope are part of platform security.

Do not store production secrets in ordinary Git values just because the rest of the configuration is there. Our External Secrets guide covers ownership, RBAC, refresh and failure modes when Kubernetes consumes externally managed secrets.

Kubernetes: production behavior matters more than manifests

Kubernetes can schedule and restart workloads, but it does not make an application reliable by default. Production readiness includes the workload's behavior during rollout, autoscaling, upgrade and node failure.

Check:

  • CPU and memory requests and limits against real usage.
  • Startup and readiness probes against actual dependencies.
  • Deployment surge and unavailable limits.
  • PodDisruptionBudgets and topology spread for workloads that need quorum.
  • Storage classes, reclaim policies and volume behavior during node drains.
  • Autoscaling metrics and the time required for new capacity to become useful.
  • Add-on, operator and deprecated API compatibility before cluster upgrades.

A managed control plane does not manage these application responsibilities. EKS, GKE, AKS, RKE2 and self-managed clusters all still need workload-level decisions.

Stateful workloads are not ordinary Deployments

Databases, brokers and caches can lose data or quorum when treated like stateless replicas. Their rollout order, storage, anti-affinity and failure behavior need explicit testing. A probe that checks only an open port may allow a broken release to keep replacing healthy pods.

This is also why a Kubernetes upgrade is not only a control-plane button. The Kubernetes upgrade dependency guide shows why add-ons, CRDs, APIs and stateful behavior belong in the same plan.

DevOps, SRE and platform engineering are connected, not interchangeable

Discipline Primary concern Typical outputs
DevOps How software is built, delivered and operated CI/CD, infrastructure as code, deployment automation, shared operations
SRE How reliability is measured and protected SLIs, SLOs, error budgets, incident practice, toil reduction, capacity work
Platform engineering How teams get safe self-service delivery Golden paths, templates, internal developer platforms, reusable platform capabilities

A small team does not need three departments. It does need the three questions answered. Platform engineering without reliability can standardize failure. SRE without delivery automation can become manual incident response. DevOps without user-oriented reliability can optimize deployment count while production gets worse.

SLOs and error budgets connect reliability to decisions

An SLI is a measured indicator, such as successful request ratio or latency for a user journey. An SLO is the target for that indicator over a defined period. An error budget is the amount of failure the target permits.

A useful SLO is based on user experience, not only server uptime. For example, a checkout can be "up" at the load balancer while a downstream dependency makes every order fail.

Error budgets support a practical trade-off:

  • While the service is comfortably within budget, normal delivery can continue.
  • When the budget is being consumed too quickly, reliability work takes priority.
  • The response should follow a written policy rather than an argument after every incident.

SLOs should be few enough to guide action. They do not replace detailed diagnostics; they tell the team when a technical issue matters to users.

Observability should answer incident questions

Metrics, logs and traces are useful when they help someone understand and resolve a problem. More telemetry is not automatically better, especially when its cost grows faster than its diagnostic value.

Start with questions such as:

  • Which user journeys are failing or slowing down?
  • Which release, replica, node, dependency or tenant is affected?
  • Is the problem reaching the application, or is it stuck in a queue or network path?
  • What changed immediately before the failure?
  • Can the responder see the relevant evidence from the alert itself?

Alerts should map to actionable conditions and named owners. Duplicate threshold alerts create noise without improving detection. Runbooks should link the signal to safe first checks, escalation and rollback.

Our observability cost guide explains why telemetry should be evaluated by useful diagnostic signal, not only ingestion volume or storage price.

Incidents, runbooks and postmortems complete the loop

An incident process defines how a team detects, responds, communicates, recovers and learns. It does not require a large organization or a complicated status-page product.

A usable runbook includes:

  1. The symptom and why it matters.
  2. Safe initial checks and evidence to collect.
  3. The owner and escalation path.
  4. Known rollback, failover or mitigation actions.
  5. Validation that service and data are healthy again.
  6. A post-incident record of causes, contributing conditions and corrective work.

Postmortems should change the system, not assign blame. Corrective actions need owners and verification, otherwise the same incident becomes a recurring meeting.

Disaster recovery is a test, not a document

A recovery plan should state what must be restored, in which order and to what measurable target. Recovery time and recovery point objectives belong to business impact, not a generic template.

A meaningful DR exercise can include:

  • Restoring infrastructure and configuration from code.
  • Recovering data and validating its consistency.
  • Switching traffic and proving critical user journeys.
  • Measuring actual recovery time and data loss.
  • Returning to the normal operating state safely.

The DR testing and failover guide shows why the way back matters as well as failover. A plan that has never been exercised is a hypothesis.

DevSecOps belongs inside the delivery path

Security controls work best when they are part of ordinary delivery rather than a separate review after deployment. Depending on the risk, this includes dependency and image scanning, secret detection, least-privilege access, network policy, workload identity and evidence collection.

The controls must be tuned to a clear owner and an exception process. A scanner that produces thousands of unowned findings does not make delivery secure. Formal independent penetration testing and certification also remain separate activities; a pipeline cannot declare itself compliant.

See our DevSecOps service and the container scanning boundary for the distinction between build-time checks and runtime risk.

Cost and performance belong in the same engineering decision

DevOps decisions change both reliability and cost. Autoscaling, runner pools, managed databases, observability backends and cloud providers should be compared using real workload behavior.

Measure:

  • Price per successful transaction, completed build or other useful unit.
  • Performance under the expected workload, not only instance specifications.
  • The operational work added when a managed service is replaced with self-management.
  • Data transfer, storage, logging and support costs that sit outside compute.
  • The rollback cost if the target architecture turns out to be wrong.

In an UpCloud migration, architecture simplification reduced cost by 60-70% versus the initially proposed architecture. After migration, the remaining monthly bill fell to one-half to one-third. The important sequence was simplification, measured migration and verification, not a universal claim that one provider is always cheaper. Read the UpCloud migration case for the actual context.

Fixed task, project or ongoing DevOps responsibility

Not every problem needs an ongoing service.

  • Fixed task: a pipeline, Kubernetes upgrade, DR test, security boundary or recurring incident with a defined acceptance result.
  • Defined project: a migration, platform build, observability rollout or broader remediation with staged deliverables.
  • Ongoing responsibility: continuous upgrades, incidents, security, cost, performance and documentation shared by a team.

In every model, code, dashboards and runbooks should remain accessible to the client. Ongoing DevOps as a service is about shared operational knowledge, not creating dependency on one external engineer.

Our DevOps and Kubernetes consulting covers defined engineering work, while DevOps as a service covers ongoing responsibility from a senior team.

A practical DevOps review checklist

Use these questions to find the next highest-value improvement:

Delivery

  • Can one change be traced from review to the artifact running in production?
  • Are pull-request, release and deployment identities separated?
  • Can the team deploy and roll back without one specific person's laptop?
  • Are DORA metrics defined consistently enough to compare over time?

Platform

  • Can environments be reproduced from infrastructure as code?
  • Are state, secrets and deployment credentials scoped and reviewed?
  • Have rollout, node-drain, upgrade and stateful-workload failure modes been tested?
  • Do developers have a safe, documented path for ordinary delivery?

Reliability

  • Do SLOs reflect real user journeys?
  • Do alerts lead to actionable evidence and an owner?
  • Are runbooks current and incidents followed by verified corrective actions?
  • Has recovery, including data and the return path, been measured end to end?

Economics and ownership

  • Is cost compared per useful unit and measured performance?
  • Are data transfer, telemetry and operational work included in platform choices?
  • Does the team keep its code, accounts, dashboards and runbooks?
  • Is there a named owner for the platform after the engagement changes or ends?

The strongest next step is rarely "install more DevOps". It is usually to find one broken handoff, unsafe trust boundary, unmeasured reliability target or untested recovery path and fix it with evidence.

Tell us about your setup

Describe what you run and what isn’t working: the cloud bill, response times, releases or recurring incidents. We reply within one business day with a suggested first step, usually a short call or an audit.

Prefer email? Write to info@bigbadbyte.com. Prefer to talk? Schedule a call.