Performance and limits

For architects and buyers.

In short

Every performance figure on this page comes from a test that runs on every change and prints what it measured. None of them is an estimate, and none of them was taken on a machine sized to flatter the result. They are figures from a development machine, so read them as orders of magnitude for your own sizing, not as a guarantee.

The short version: the API answers in milliseconds and the indexer keeps up with a busy chain, but one worker carries about 25 transactions a second from one sending identity, against a target of 50 that it does not meet. A second worker adds throughput. Whether that is enough is a decision the product owner has not taken yet; this page does not promise a fix.

What was measured

A submission is answered 202 in single-digit milliseconds at the 95th percentile over 200 authenticated submissions with gas supplied, against a budget of 200 ms; the API accepts about 390 submissions a second from one identity.

Throughput from one sending identity is measured on every change, against PostgreSQL, a real node and authentication on: on a development machine one worker carries about 25 submissions a second through to broadcast, against a target of 50 that one worker does not meet.

A second worker on the same identity adds throughput (about 34 submissions a second against about 25 with one, on a development machine) and the nonce sequence stays dense: no gap, no repeat.

One indexer kept up with 500 subscribed logs per block at a 2-second block time: 6,000 logs delivered once each under 6,000 distinct ids, never a block of lag against a bound of depth plus two.

How it was measured

Three choices make the transaction figures worth trusting, and each one is a way the number could have been made to look better:

• Against PostgreSQL and a real node (a local Anvil chain), not an in-memory store or a stub. A memory store would measure a hash map, and the point of the requirement is the one place where work has to queue behind a lock.
• With authentication on, and a real API key presented on every call. A budget measured without the credential check is a budget for something the service never does in production.
• From one sending identity. Spreading the load over several identities removes the serialization point and measures the machine instead of CryptoRail.

The API, the worker and the signing gate run as separate processes over the shared database, as a deployment runs them. The test submits 150 transfers at once with the fees supplied, waits until every one of them has been broadcast, and then checks that the nonces form a dense sequence: a fast figure with lost or duplicated work would not be a throughput figure.

sequenceDiagram
    participant T as Test
    participant A as API process
    participant DB as PostgreSQL
    participant W as Worker
    participant G as Signing gate
    participant N as Anvil node
    T->>A: 150 authenticated submissions at once
    A->>DB: persist each intent
    A-->>T: 202 for each (accepted per second)
    loop one transaction at a time per worker
        W->>DB: claim, prepare, take the nonce
        W->>DB: signing request
        G->>DB: sign under the gate's checks
        W->>N: broadcast
        W->>DB: persist every transition
    end
    T->>DB: all broadcast? nonces dense? (broadcast per second)

The indexer measurement runs a contract that emits 500 subscribed Transfer logs per block, with Anvil mining a block every 2 seconds, one indexer and a decoded subscription. Over twelve loaded blocks it delivered 6,000 events under 6,000 distinct ids, never fell a block behind, and took 43 ms median per pass to hand off a 500-log block. The 250 logs a second it sustained are the chain’s rate, not the indexer’s ceiling.

Reproducing it

The measurements need Foundry’s anvil and a Docker daemon for PostgreSQL; without them the suites skip and say why.

pnpm --filter @cryptorail/service exec vitest run test/throughput.test.ts                  # NFR-PERF-01
pnpm --filter @cryptorail/service exec vitest run test/latency.test.ts                     # NFR-PERF-03
pnpm --filter @cryptorail/integration-tests exec vitest run test/indexer-throughput.anvil.test.ts   # NFR-PERF-02, NFR-PERF-04

Each prints its figures on every run. The build fails only on a collapse (below 10 a second on a development machine, below 1.5 on a shared CI runner, which measures about 4 a second because it is slow at everything), not on a slowdown: CI keeps the measurement honest and running, it is not a benchmark.

Where the time goes

The bottleneck of the worker has been located with measurements, not guesses:

• Not accepting the work. The API takes about 390 submissions a second.
• Not the nonce lock. Allocating a nonce holds the lock for two database round trips and nothing else: a median of about 5 ms against roughly 42 ms each transaction spends in the pipeline.
• The per-transaction work itself. Each transaction is prepared, given a nonce, signed and broadcast, with every state change persisted before anything depends on it. That is fifteen to twenty round trips to the database and the node, in sequence, and a worker carries one transaction at a time.

The nonce lock is held for two database round trips and nothing else, measured on every run: a median of about 5 ms, an eighth of the time a transaction spends in the pipeline.

Every operation follows one state machine written as a transition table: each change of state is persisted with its cause and time before anything depends on it, and a transition the table does not allow is refused.

The persistence before every side effect is what makes an acknowledged operation impossible to lose across a crash; it is also what costs the time. That trade is deliberate, and the life of a transaction explains it.

An operation CryptoRail has acknowledged is never lost: not across crashes and restarts, not through a node outage of up to ten minutes.

What scales by adding workers

Workers share the queue through the database. Each transaction is claimed by the worker that picks it up, so two workers never prepare the same one, and the nonce sequence of an identity stays dense whatever the number of workers.

flowchart LR
    API[API processes] --> DB[(PostgreSQL)]
    DB --> W1[Worker 1]
    DB --> W2[Worker 2]
    DB --> Wn[Worker n]
    W1 & W2 & Wn --> L{nonce lock per identity}
    L --> N[Nodes]

• More workers on one identity: one worker measured about 25 a second, two about 34. The nonce lock per identity is held briefly, so it is not what stops the second worker; how far more workers go before the lock or the database becomes the limit has not been measured.
• More identities: separate sending identities do not wait on each other’s nonce lock, which is why the measurement deliberately uses only one.
• More API processes: they share only the database. One process already accepts about 390 submissions a second, so the API is rarely where a deployment runs out.

Several workers on the same database without sending anything twice.

One process, or separate API, worker, signing gate and indexer processes sharing only the database, each with a database role limited to what it needs.

Before the claim per transaction existed, two workers measured slower than one (19 a second) and, worse, left gaps in the nonce sequence. That defect was fixed before anything was made faster.

The open decision

NFR-PERF-01 asks one worker to sustain 50 submissions a second from one identity. It is a Should requirement, and it is not met by one worker: the figure is reported rather than the target lowered. Getting one worker past it would mean letting a worker have more than one transaction in flight at a time; the claim per transaction makes that safe to attempt, and it has not been attempted.

Whether about 25 a second per worker, and more with more workers or more identities, is enough is a decision of the product owner that is still open. Until it is taken, size a deployment on the measured figures, and tell us during a pilot what rate from one identity your use case needs.

Known limits

An atomic multi-call carries two to sixteen calls, executed all or none by the actor's smart account in one UserOperation; it is refused in plain transaction mode.

Per-tenant limits set by the operator, with refusals that name the limit reached.

Per API key rate limiting with standard rate-limit headers and a 429 with Retry-After, counted in each API process; a count shared across several copies of the API is not built.

Per node endpoint, a rate and a concurrency limit enforced before a request leaves the process: a burst above the limit waits in a queue and is served, and fails as retryable only after a timeout.

Wallets ask for sponsorship through the standard ERC-7677 interface, authenticated per application and held to quotas.

Several indexers sharing the subscriptions of one deployment: one indexer per deployment is the rule. Not supported

Limits that come from the network rather than from CryptoRail, such as a provider’s cap on eth_getLogs ranges or the block gas limit, are absorbed or reported, not lifted: a provider that throttles makes the indexer slower, not wrong.

What it does not do

• One worker does not reach 50 transactions a second from one identity. About 25 is measured; whether that is enough is open.
• The figures are not a capacity guarantee. They come from a development machine against a local chain; a public network’s block time and your node provider’s limits bound what any deployment can send.
• It does not share a rate-limit count across several API copies. Each process counts its own.
• It does not shard subscriptions across indexers. One indexer per deployment.
• It does not spread calls across healthy node endpoints. Calls go to the first healthy one, and the others are there for failover.

Several node endpoints per network: calls go to the first healthy one, an operation stays on the endpoint it started on, and a dead or lagging node is left aside without losing anything.

Evidence

Every claim above links to its evidence on the evidence page. The main ones: