A load test that ends with "performance is worse" has found a problem, but it has not finished the job.
For a development team preparing a release, the useful outcome is a failure that can be reproduced, a clear account of the conditions that trigger it, and enough evidence to tell whether a fix worked. That was the distinction in one Java/Kubernetes investigation by the Big Bad Byte team.
One hot replica, then another
The team was performing pre-release load tests against a service running with multiple replicas on Kubernetes.
At first, the instances behaved normally. As the test continued, exactly one replica began approaching CPU and memory saturation while the other instances became less busy. The symptom did not follow a particular pod: terminating the overloaded one caused a different replica to take over the same pattern.
This made a simple "one bad machine" explanation less convincing. The replicas used the same application code, yet the load distribution changed whenever one instance slowed.
Thread dumps and profiling offered little useful direction. The next experiment changed the investigation.
A heap dump changed the measured system
During a load test, an engineer captured a Java heap dump. Immediately, the observed load dropped across all replicas.
The heap dump itself did not reveal the suspected bug. The important observation was what happened while it was being captured: request processing was interrupted, and the supposedly independent replicas all became quiet.
The engineers reproduced that interruption using SIGSTOP on a JVM. They also deliberately introduced CPU work on a different replica. The overload pattern followed the slowed instance.
These were not production mitigations or recommended ways to operate a healthy service. They were controlled diagnostic experiments, useful because the resulting changes were repeatable.
The load generator was part of the explanation
The test exercised sequential user flows. A virtual user waited for a response before progressing to its next action.
When the sequence hit a slowed or stopped replica, it could not keep generating the following requests at the expected pace. That helped explain why the workload seemed to concentrate on one replica while the others were relieved. The team's original account described round-robin request distribution; the insight was the relationship between the request path, sequential waits and the observed load, rather than a claim that every Kubernetes load balancer behaves the same way.
In other words, the experiment exposed a coupling between the application and the test workload. More CPU on one replica was a symptom, not necessarily the place where a fix should be made.
The original engineering narrative, with graphs, is available in A cursed replica.
From a reproducible profile to an application fix
The performance engineers shared the failure profile with the development team.
A developer then found a feature implementation that consumed more CPU than it had previously. The developers corrected that application code; the performance team did not claim to have written the fix.
The workload was tested again. The earlier degradation was no longer observed, and the release received its performance sign-off.
There are no published latency percentiles, exact throughput deltas or financial figures for this investigation. The observed outcome is narrower and more credible: the team characterized the problem, the developers fixed the implementation, and retesting no longer reproduced the failure.
What belongs in a useful performance gate
This case suggests five questions for teams designing a pre-release performance gate:
- Is the test realistic enough to expose the risk? Use a representative mix of user journeys, concurrency and data rather than a single synthetic endpoint.
- Does the report show where the failure happens? Preserve workload conditions, resource behavior, timing and the failure profile rather than only an overall pass/fail number.
- Can the result be challenged by an experiment? Try isolating instances, introducing controlled delays or changing a relevant workload parameter when safe.
- Who owns the next diagnostic step? Infrastructure engineers can characterize a failure without being the authors of an application code fix.
- Was the fix measured against the same failure profile? A release gate should not turn green until the original regression has been checked again.
The general method applies beyond Kubernetes: understand the load, reproduce the symptom, isolate a mechanism, deliver actionable evidence to the responsible team, and retest after the change.
Performance engineering that helps a team ship
Stopping a release is sometimes the correct outcome. But an effective performance gate should do more than block the deployment. It should shorten the path from an unexplained test result to a fix that engineers can verify.
That is the goal of Big Bad Byte's performance engineering work: turning load tests and production-like experiments into concrete engineering decisions.