DevSecOps in production: CI/CD, Kubernetes and compliance evidence

DevSecOps integrates security decisions into the same systems that build, deploy and operate software. It covers code and dependencies, but also runner identities, artifacts, secrets, Kubernetes permissions, network paths, runtime behavior and the evidence behind them. A scanner is only one control, not a security boundary by itself.

This guide follows an unreviewed change from source to production and shows which controls and acceptance tests matter at each stage.

Start by mapping how an unreviewed change could reach production

Threat modeling does not need to begin with a complex framework. Trace one ordinary change and ask where trust is granted:

  1. Who can modify source, workflow and deployment configuration?
  2. Which code runs on the CI runner, and under which identity?
  3. Which dependencies and actions execute during the build?
  4. Which caches and artifacts can later be trusted?
  5. Which identity publishes and deploys the result?
  6. Which cluster, cloud account and network paths can it reach?
  7. Which evidence proves that approved delivery works and unauthorized work is denied?

Prioritize gaps by exploitability and impact. An internet-reachable unauthenticated service or a low-trust job with production credentials usually deserves attention before a scanner-configuration nicety.

Establish ownership and risk thresholds before adding scanners

Security tools generate findings. They do not decide what the business should do with them.

For each control, define:

  • Which risk it addresses.
  • Which findings block delivery and which create review work.
  • Who owns triage and remediation.
  • How exceptions are approved, documented and revisited.
  • Which evidence proves the control still works.

A pipeline that blocks every low-severity issue trains developers to bypass it. A pipeline that reports thousands of findings with no owner produces the same result. Tune gates around exploitable risk and delivery context.

Code, dependency and secret scanning

Common CI controls include:

  • Static application security testing for dangerous code patterns.
  • Software composition analysis for vulnerable dependencies.
  • Secret detection for credentials accidentally added to source.
  • Container image scanning for known package vulnerabilities.
  • Dynamic testing against a controlled running environment.

Tools already used by the Big Bad Byte team include Semgrep, SonarQube, Trivy, Grype, Gitleaks, TruffleHog, OWASP ZAP and DefectDojo. Tool choice should fit the stack and ownership model rather than creating an isolated scanner for every category.

Deduplicate findings and route them to the team that can fix them. Record affected version, exposure, exploitability evidence and remediation state. A CVE count alone does not describe production risk.

The runner is part of the supply-chain boundary

CI runners execute repository code, third-party actions and build scripts. A compromised or untrusted job can attempt to read secrets, modify artifacts, poison caches or assume deployment roles.

Separate:

  • Pull-request validation from release builds.
  • Read-only analysis from artifact publication.
  • Deployment from other CI work.
  • Cache read and write privileges by trust level.
  • Short-lived cloud identities by repository, environment and operation.

Ephemeral runner registration does not prove a clean execution environment. Use clean compute per job and check what persists through disks, processes, network paths and credentials.

Our GitHub Actions runner security guide covers untrusted checkout, runner groups, OIDC, cache trust and harmless acceptance tests.

Use short-lived, narrowly scoped deployment identity

Long-lived cloud keys turn a CI compromise into a cloud compromise. OIDC can replace them with short-lived credentials scoped to an explicit trust relationship.

Review:

  • Which repository, branch, environment and workflow can assume each role.
  • Which resources and actions the role permits.
  • Whether pull-request code can influence role claims or deployment input.
  • How role use is logged and alerted.
  • How quickly access expires and can be revoked.

OIDC does not fix an unsafe workflow that executes untrusted code in a privileged context. It reduces credential exposure after the workflow and identity boundaries are already designed correctly.

Secrets need ownership, delivery and rotation

A secret manager stores credentials, but the full lifecycle matters:

  • Who owns each secret and its rotation?
  • Which workload may read it?
  • How is it delivered to runtime without entering Git or ordinary CI logs?
  • Is access audited and revocable?
  • Does a refresh reach the application safely?
  • What happens when the secret manager or controller is unavailable?

HashiCorp Vault, OpenBao and cloud secret managers can store secrets. External Secrets Operator can deliver them to Kubernetes workloads with explicit ownership and RBAC. The External Secrets guide covers refresh and failure modes.

When a secret is exposed, masking the log is not enough. Revoke and rotate it, investigate use and remove it from future builds.

Artifacts need provenance and controlled promotion

A release should deploy an identified artifact built from a reviewed revision, not an ambiguous image assembled later.

Record:

  • Source revision and build identity.
  • Dependency and build inputs where required.
  • Test and scan results relevant to release policy.
  • Registry and immutable artifact identity.
  • Promotion and deployment history.

Minimal base images, pinned and verified build inputs and controlled registries reduce ambiguity. Where the risk requires it, provenance, SBOM and signing can add evidence, but they should solve a defined verification problem rather than becoming decorative metadata.

The same artifact should move through environments. Rebuilding independently can introduce changes that were never tested.

Kubernetes hardening is runtime work

A clean image scan does not limit what a compromised workload can do. Runtime controls determine the blast radius.

Review:

  • Service-account RBAC and cloud workload identity.
  • Network policies, ingress and default-deny or scoped egress.
  • Public endpoints and administrative interfaces.
  • Admission and configuration policy.
  • Secrets and certificate handling.
  • Node, control-plane and add-on patch status.
  • Logging and response ownership.

A database or metadata service should not be reachable simply because a pod runs in the same cluster. The container security boundary article explains why build-time and runtime controls answer different questions.

For rollout, stateful and upgrade safety, see the Kubernetes production guide.

Dynamic tests need controlled scope

Dynamic application security testing can validate running behavior, but it needs explicit targets and rate limits. OWASP ZAP can automate checks against an approved test environment. It should not be pointed at arbitrary production systems.

DAST does not replace an independent penetration test. A formal report by a specialized firm remains a separate activity. Technical teams can fix findings and produce configuration evidence afterward.

Compliance evidence should come from normal operations

Technical controls can support PCI DSS, ISO 27001, SOC 2, NIS2 and other assessments, but a pipeline cannot certify itself.

Useful evidence includes:

  • Access reviews and scoped identity configuration.
  • Change history from Git and protected approvals.
  • Deployment and artifact records.
  • Vulnerability findings, ownership and exception status.
  • Logging, retention and incident records.
  • Backup, restore and disaster-recovery exercises.

The evidence should be generated by ordinary workflows where possible. A screenshot assembled once before an audit does not prove that the control continues to work.

An accredited auditor or independent assessor decides whether those controls meet the formal requirement.

AI agents add tool and content boundaries

AI coding and operations agents process untrusted content and may hold powerful tools. Apply the same DevSecOps model:

  • Separate external content from privileged instructions.
  • Scope read and write tools independently.
  • Require exact approval for high-impact actions.
  • Make operations idempotent and auditable.
  • Test prompt injection, memory poisoning and retry behavior.

The prompt-injection guide covers why a system prompt is not an authorization boundary. The AI-orchestrated attack review separates documented task delegation from claims of independent criminal intent.

Acceptance tests should prove denial and normal delivery

A control is credible when its behavior can be tested. Use harmless canary resources and synthetic identities:

  • A low-trust job attempts to assume a deployment role and must fail.
  • An untrusted job attempts to write a trusted cache and must fail.
  • A workload attempts to reach an unrelated internal service and must fail.
  • An expired or incorrect secret must not be accepted.
  • The approved build, release and deployment path must still succeed.
  • A restore or rollback must return the service to the expected state.

Do not use real customer data or destructive tests for this. The point is to verify the boundary without creating a security incident.

A practical DevSecOps review checklist

Source and pipeline

  • Are workflow and deployment changes reviewed?
  • Are trust levels, runner environments and caches separated?
  • Are scanner thresholds, owners and exceptions explicit?
  • Are deployment identities short-lived and narrowly scoped?

Artifacts and secrets

  • Can the running artifact be traced to a reviewed revision?
  • Are build inputs and promotion controlled?
  • Are secrets owned, rotated, delivered and audited?

Runtime

  • Are workload identity, RBAC, ingress and egress limited?
  • Are public endpoints and management interfaces intentional?
  • Are runtime alerts and response owners defined?

Evidence and acceptance

  • Do denial canaries fail safely while approved delivery works?
  • Are findings, exceptions, changes and recovery evidence current?
  • Are formal independent testing and certification kept in their proper scope?

Big Bad Byte's DevSecOps service implements these controls inside existing CI/CD and Kubernetes platforms, with documented runbooks, exception processes and harmless acceptance tests rather than unsupported compliance promises.

Tell us about your setup

Describe what you run and what isn’t working: the cloud bill, response times, releases or recurring incidents. We reply within one business day with a suggested first step, usually a short call or an audit.

Prefer email? Write to info@bigbadbyte.com. Prefer to talk? Schedule a call.