The number was more than 40 node instances, all within one crypto client's project. These included RPC, full and archive workloads across multiple bare-metal providers.
That is a meaningful fleet, but the node count does not tell you whether the systems are operationally interchangeable. An execution client, an archive endpoint and a Solana RPC node do not have the same definition of healthy. Nor does the count mean 40 different blockchains or 40 nodes running simultaneously. We do not have that inventory breakdown available to publish.
The useful engineering lesson from operating this fleet was that a common platform can provide consistency, but chain-specific operations still require deliberate choices.
A running process is not a working RPC service
For an API customer, a node can appear alive while failing the queries they actually depend on.
An Ethereum execution endpoint can be connected to peers yet be behind the current chain head. A node intended to answer historical queries must also have the data required for those calls. On another network, recent synchronization may be acceptable but a query path may depend on optional indexes that the node lacks.
This means a generic process-health check should be only the beginning. A practical RPC service check has to answer three questions:
- Is the service process reachable and responding?
- Is its chain state sufficiently current for the application's use case?
- Can it perform the actual API methods the consumer needs, including any historical lookups or index-dependent calls?
Those three checks are an operational recommendation; they are not a claim that every test was deployed in this customer fleet.
Our team tailored monitoring to the different chains and created custom metrics where generic infrastructure metrics were insufficient.
Sync and snapshots change the recovery plan
It is tempting to treat blockchain nodes as stateless containers: replace one, restore a persistent volume or download a snapshot, and let it catch up.
In practice, recovery depends on the network and node role.
For some networks in this engagement, a sufficiently fresh usable snapshot was not readily available. We had to create snapshots ourselves or source a suitable alternative. That changed the operational work from "start another machine" to "verify how much trustworthy state can be restored, and what remains to synchronize."
An archive node may need different historical data from a standard full node. A stale snapshot may also have very different catch-up implications from a recent one.
For a team planning recovery, a runbook should therefore identify the node role, expected data set, snapshot source and integrity check, catch-up observation, and the first successful real query that proves it is fit for use.
We are deliberately not publishing snapshot sizes or recovery-time promises: they are not supported by the original case record.
Upgrade calendars are not uniform across chains
There is no single release cadence covering the whole blockchain ecosystem.
During this engagement, keeping up with version changes and required upgrades meant following different projects' release announcements and interpreting uneven upgrade guidance. Some changes required timely operational attention; others needed testing for compatibility and impact on the RPC service.
A reusable upgrade checklist helps, but it still has to link to chain-specific source material:
- What is the exact version or protocol requirement?
- Is the change optional, recommended or required for chain continuity?
- Does the current client support the needed behavior?
- Is the data-format change reversible?
- Can the service continue handling requests throughout the maintenance?
- What is the fallback if synchronization or serving queries fails after the update?
The point is not that every network has poor release practices. The point is that one infrastructure team cannot safely assume one network's upgrade process applies to another.
Client choice should follow actual methods
The team also compared execution clients such as Geth and Reth for different request patterns.
It would be convenient to declare one client universally faster. We did not establish such a result. The practical choice depends on how the application uses the interface, the data retained by the node, the resource envelope and the operations required around it.
A benchmark of one RPC method is not a benchmark of the whole node. Nor does a fast response prove that a client handles every required method or recovery scenario.
A useful evaluation would test the methods that matter to consumers, at realistic concurrency, and account for sync behavior, storage and memory use, tail latency and upgrade/restart behavior. These are evaluation criteria, not invented measurements from the project.
The network is part of the node service
The workloads ran on bare-metal infrastructure from multiple providers. Choosing providers required balancing cost and reliability, and in some cases the team had to investigate or escalate networking problems.
That is different from simply buying the host with the most RAM per euro. A busy RPC service depends on stable peer connectivity, throughput toward consumers and predictable storage and network behavior. Poor external connectivity can make a correctly configured node unusable for the application.
Our public account does not identify the affected providers or establish a market-wide ranking. Those conclusions would require data from comparable locations, time windows and workloads.
A real example of an application mismatch
One Solana workload exposed a different class of problem: the application sent queries that needed indexes deliberately omitted from the RPC nodes.
The developers believed the queries were not used. We added HAProxy with custom Lua method monitoring to observe the requests actually reaching the service. That evidence enabled the developers to fix the application call pattern, after which the observed node issue stopped.
This was not a case where adding nodes solved an indexing mismatch. It was a case where the contract between an RPC service and its consumers needed to be measured.
The full technical story is in our companion draft, The Solana RPC calls our infrastructure was never supposed to receive (link after that page is actually published).
Standardize the platform, not the assumptions
The fleet benefited from shared operational thinking: inventory, ownership, alerting conventions, changes, backups, escalation and verified recovery procedures. But "node is green" could not have one universal meaning.
For a CTO or RPC service owner, the review question is straightforward: can the team show that each kind of node performs its required role, remains current enough, and can be recovered and upgraded without relying on a generic container restart?
That is the operational value of blockchain node management. The goal is not to make every network look identical. It is to make the differences observable and manageable.