The node configuration looked reasonable for the workload we had been promised. The application team said it would not make queries that needed certain Solana account indexes. So some indexes were deliberately left out of the RPC nodes we operated for an anonymous crypto platform.
Then the nodes started becoming unstable after particular requests.
At that point, there were two different descriptions of the same system. The infrastructure team saw nodes hanging or slowing. The developers believed their queries stayed within the configuration's limits. Increasing machine size might have postponed a symptom, but it would not settle which description was correct.
We needed to see the actual requests.
An infrastructure choice depends on an application contract
An RPC node does not have one universal cost or capacity profile. Memory, storage, indexing, historical data, network throughput and request concurrency all depend on the methods clients call.
That is particularly important when a team intentionally omits indexes to control resource usage. Such a decision can be suitable for a known workload, but only if the consuming applications really honor its limits.
Solana uses JSON-RPC over HTTP. Each request carries a method name and optional parameters. The public Solana RPC method reference describes different kinds of reads, from lightweight metadata to account and transaction queries. That reference explains the protocol; it does not establish which exact methods triggered our client's incident.
In this project, the expected method mix and the method mix reaching the nodes had diverged.
Why the usual dashboards were not enough
A dashboard showing that the RPC process is running, CPU is busy or latency is rising can establish that the service is unhealthy. It cannot by itself tell you whether the caller used a method that depends on unavailable indexes.
The issue also crosses team boundaries. The developer may inspect application code and find no obvious forbidden call. The operator may inspect node logs and find expensive or stalled requests. Neither is a complete picture of what crossed the API boundary.
We therefore changed the observation point rather than debating assumptions.
We placed HAProxy in front of the Solana RPC nodes. A custom Lua script observed the incoming JSON-RPC method names and counted requests. This gave us a method-level view of actual traffic.
The observation revealed calls that did not match the agreed workload, including requests that depended on indexes excluded from the node configuration.
That was the missing evidence. There was no need to assume a general Solana client defect or guess which node setting should be re-enabled.
What the infrastructure team changed, and what developers fixed
Our team's contribution was the RPC-facing instrumentation and diagnosis. We showed which kinds of requests were reaching the service so the application team could compare them with the intended integration behavior.
The developers then changed their application code. After that change, the particular node problem stopped recurring.
This division of responsibility matters. We did not fix the application's code, and the incident did not prove that all non-indexed Solana configurations are unsuitable. It showed that a resource-conscious infrastructure design and a consumer's actual behavior must be checked against one another.
We have not retained the exact method names, omitted index flags, or a controlled benchmark for this story. Publishing guessed method names would make the article sound more technical while making it less reliable.
How to repeat the diagnostic approach
For a team facing a similar disagreement, the workflow is more reusable than any single Lua snippet.
1. Write down the supported RPC contract
List the method families the application is allowed to use, and distinguish optional features such as particular indexes or historical query availability. Record where the assumptions came from. A statement like "the application never uses this" is a hypothesis until traffic confirms it.
2. Observe the boundary
Use a gateway or another suitable observation layer to classify real incoming RPC methods. Where JSON-RPC requests are batched, the observation layer must treat the batch carefully instead of assuming every HTTP request contains exactly one call.
The useful output is a method-level count, ideally separated by authorized application or integration. This is an approach for other teams, not a claim that the original customer used every one of these dimensions.
Do not turn the debugging proxy into a source of sensitive request-payload logs. Method counting and carefully controlled metadata are often enough; review access controls, payload retention and request-size limits before deploying any inspection logic.
3. Compare traffic with provisioning decisions
Find the method families that violate the assumed contract. If the workload genuinely needs them, re-evaluate the node configuration and resource model. If those methods were unintended, fix the caller instead.
The correct response is not automatically "increase RAM", "add all indexes" or "move to a different RPC provider".
4. Confirm the fix at the same boundary
After the application changes, confirm that the unwanted method pattern is gone and that the node behaves as expected under representative traffic. Measure relevant failure and latency signals rather than stopping at a successful deployment.
The operational lesson
The boundary between blockchain infrastructure and application code is an API, not an organizational wall.
When node operators and developers disagree about what is happening, better request-level evidence usually beats a larger server and a longer meeting. In our Solana case, method-level observability identified an application-to-infrastructure mismatch, developers corrected the caller, and the observed node problem ceased.
For teams running their own RPC infrastructure, this is one part of blockchain node operations: understanding what the applications actually demand and building the node fleet around that evidence.