Operations and multi-tenancy

For operations teams.

In short

CryptoRail is one container image and one configuration file. Each process starts in a role, checks its whole configuration before doing anything, and then tells you the truth about itself: a health answer for “is it alive”, a readiness answer for “can it work right now”, and traces, metrics and JSON logs through OpenTelemetry into the monitoring tools you already run. Nodes fail over without your systems noticing. Several companies can share one deployment, each seeing only its own resources, and that isolation is checked on every route of the API on every change. The daily work (pausing an identity, redelivering a webhook, exporting a history, onboarding a tenant) is a command line over the same API, and each situation that recurs has a runbook.

This page is for the team that will run it. The architecture deep dive explains the processes; the security deep dive explains what each one may reach.

What you run

flowchart LR
    subgraph CR[CryptoRail, one image]
        API[api: HTTP]
        W[worker: pipeline, trackers, deliveries]
        G[signing gate]
        IX[indexer: subscriptions]
    end
    DB[(PostgreSQL)]
    API --> DB
    W --> DB
    G --> DB
    IX --> DB
    W --> N[Node endpoints]
    IX --> N
    CR -->|OTLP| OT[Your OpenTelemetry collector]
    OT --> MON[Your traces, metrics and logs backends]
    ORC[Your orchestrator] -->|health, readiness| API

The same image runs as one process for development and small deployments, or as separate API, worker, signing gate and indexer processes. They share only the database: no process calls another, so a worker can be added or restarted without anyone else noticing. Schema migrations are an explicit command, never a side effect of starting.

One process, or separate API, worker, signing gate and indexer processes sharing only the database, each with a database role limited to what it needs.

Several workers on the same database without sending anything twice.

Runs in your infrastructure, as a container image over your PostgreSQL, with schema migrations applied by an explicit command.

Runs on Node.js 22 and 24, both tested on every change; the service ships as an OCI container image over PostgreSQL, tested on PostgreSQL 17.

Starting, health and readiness

A process validates its whole configuration before it builds anything, and lists every problem at once, with the path of each in the file, so a mistake is fixed in one round rather than one restart per error. Then it resolves its secrets (references only, never values), connects to the database and refuses a schema that is behind its build, asks every endpoint which network it serves, checks the pinned contracts, and opens its signers.

Once running, it answers two different questions:

A node being down is a readiness failure, not a health failure: restarting CryptoRail would not bring the node back. Each readiness check is bounded in time and the report names the failing check, so the answer says what is wrong, not only that something is.

Health and readiness that tell the truth, and a configuration checked at startup with every problem listed at once.

Secrets passed by reference and never written to logs, traces, metrics or responses.

A stopping process finishes its broadcasts and deliveries in flight within a grace period, and whatever it could not finish is picked up again on restart.

Logs, traces and metrics

CryptoRail emits through the OpenTelemetry API only and never ships a vendor’s agent: you point it at your collector and choose the backends.

• Logs are JSON lines with the trace and span ids, the tenant, the network and the operation they concern. Secrets, keys and signed payloads never reach a line; a secret reference prints as redacted.
• Traces follow an operation for its whole life. The trace started at submission is stored with the operation, so a signature that arrives hours later, in another process, lands in the same trace. Transactions of one business operation hang from a common root.
• Metrics carry functional dimensions (operation type, contract, function, network, tenant) and never a unique id, so their cardinality stays bounded.

Traces, metrics and structured logs through OpenTelemetry, into the monitoring tools you already run.

One trace per operation, from submission to confirmation, across processes and restarts; spans carry the operation type, contract and function, so a search by operation type returns the whole business flow, and the reference dashboard drills down from a failed operation to its trace.

Metrics for transactions by state and network, time spent per state, attempts, nonce gaps, payer balances, paymaster deposits, outbox depth, deliveries, indexer lag, and each node endpoint's health, lag and queue.

Your business operation travels with its transactions: its type, its id and its step appear in queries, events and traces.

Alerts are events of their own: they reach your systems through the same channel as every other event, and your monitoring through metrics and logs. Each is raised once when a condition is crossed, not on every pass, so a condition that persists does not page twice.

Alerts for a payer low on funds, a paymaster deposit running down, a nonce gap, a signature from the wrong key, a reorganisation deeper than the finality rule and an indexer falling behind, each raised once per crossing and delivered as an event, a metric and a log line.

Failover between node endpoints

Each network takes a priority list of endpoints: your own nodes, a provider’s, or both. Every call goes to the first healthy one. A sequence of work (one transaction from nonce reservation through broadcast and tracking, one tracker cycle, one indexer pass) stays on the endpoint it started on, and moves only when that endpoint stops being usable, carrying the same signed bytes and the same cursor.

sequenceDiagram
    participant W as Worker
    participant P as Primary node
    participant F as Fallback node
    W->>P: nonce, broadcast, tracking
    Note over P: goes down
    W--xP: transient error
    W->>F: retried once, the sequence continues here
    W->>W: probes mark the primary unhealthy (log line, metric, chain status)
    Note over P: back, but blocks behind
    W->>W: still unhealthy while it lags
    Note over P: caught up
    W->>W: healthy again, new sequences return to the configured order

An endpoint that answers but is behind the others is treated as unhealthy: a node that lags misses logs and misjudges confirmations. The status of every endpoint, by name and never by URL, is readable through the API with its state, lag, queue and the reason it is unhealthy. With every endpoint of a network down, submissions are still accepted and wait; nothing acknowledged is lost.

Several node endpoints per network: calls go to the first healthy one, an operation stays on the endpoint it started on, and a dead or lagging node is left aside without losing anything.

An endpoint counts as healthy only when it answers in time and its head is within a set number of blocks of the most advanced endpoint; rate and concurrency limits per endpoint, and provider limits such as the eth_getLogs block range, are respected on CryptoRail's side.

An operation CryptoRail has acknowledged is never lost: not across crashes and restarts, not through a node outage of up to ten minutes.

Tenants

A tenant is a company using a deployment. Every resource (identities, contracts, transactions, approvals, subscriptions, webhook endpoints, API keys) belongs to exactly one tenant, and the tenant is taken from the API key, never from the request: a body or a query that names a tenant is refused rather than quietly ignored. A single-company deployment is a deployment with one tenant.

What tenants share is the deployment: the processes, the database, the configured networks and the signer adapters. Isolation is enforced by the service on every call, not by a separate database per tenant.

A tenant sees only its own resources, proven on every route of the API on every change.

Who may do what is checked per call: read-only, submitter, signer, sponsor, operator and admin roles.

How isolation is proven

A claim of isolation is only as good as its coverage, so the proof does not list routes by hand. A test gives one tenant one of every kind of resource, then reads the OpenAPI document the service serves and calls every route in it with a second tenant’s admin key and the first tenant’s ids. No route may answer with success, and no answer, lists included, may carry an id, an address or a label of the first tenant.

flowchart LR
    A[Tenant A: one of every resource] --> D[The served OpenAPI document]
    D --> R[Every route, called with tenant B's admin key and A's ids]
    R --> C{Success on A's id, or A's data in any answer?}
    C -->|yes| X[Test fails, naming the route]
    C -->|no| OK[Isolated]
    N[A new route] --> D

A route added tomorrow is in the proof the day it is added. A path parameter that does not name a tenant’s resource (a network id, a content-addressed ABI) must be declared with its reason, and the test fails when a new one appears without one. The test runs on every change; journey J44 shows the same thing to a reader, with an onboarded second tenant receiving 404s for the first one’s ids.

Onboarding, quotas and offboarding

sequenceDiagram
    participant O as Operator (admin of the operator tenant)
    participant S as CryptoRail
    participant T as New tenant's admin
    O->>S: create an admin key for tenant "globex"
    S-->>O: credential, shown once
    O->>S: set globex's quotas
    O-->>T: the credential, over a channel fit for a secret
    T->>S: create its own keys, one role per system

Only the deployment’s operator tenant can onboard another tenant or change its quotas. Quotas limit resources (identities, contracts, subscriptions, webhook endpoints) and work in flight (transactions not yet final, sponsored operations pending), with defaults in the configuration and per-tenant overrides recorded in the audit log. A refusal names the quota and its limit, and a retried submission is never counted twice. Rate limits per API key are a separate, optional guard.

Only the admins of the deployment's operator tenant can give a new tenant its first key or change another tenant's quotas; a request that names a tenant in its body or query is refused rather than ignored.

Per-tenant limits set by the operator, with refusals that name the limit reached.

Rate limits per API key, off by default, announced with standard rate-limit headers and refused with 429 and a retry time; health and readiness are never limited.

Every approval, signature and decision is recorded with who, when and what, and every change made through the API lands in an audit log.

Offboarding a tenant deletes nothing: its identities are paused, its history exported for audit and its keys revoked.

The operator command line

cryptorail is a thin client over the API, run with the operator’s own key. It holds no business logic: every command is one or a few calls to routes the service already serves, and the service’s roles decide what the operator may do. Every mutating command sends an idempotency key, so a command retried after a network error is not applied twice.

An export ends with a summary line; the command checks it is there and fails when it is not, because an export without it is incomplete, whatever it looks like.

An operator command line for the daily work: pause an identity, redeliver a webhook, reconcile nonces, export a history, onboard a tenant.

The complete history of any operation, exported in a machine-readable file for audit.

An identity can be paused: new operations are accepted and wait without taking a nonce until it is resumed.

The runbooks

Each situation an on-call engineer meets more than once has a page: what the signal means, what to look at, what to do, and what not to do.

Runbooks for the situations that recur (a dead-lettered delivery, an unhealthy endpoint, a lagging indexer, a paused identity, a signing gate down, onboarding, offboarding and key rotation of a tenant), each naming the requirement behind the behaviour and the journey that shows it.

Deployment profiles, briefly

What a deployment may load depends on who operates it: you (self-hosted), CryptoRail’s operator for you (run for you), or CryptoRail’s operator for a pilot (sandbox). The profile is checked in every process when it starts. The security deep dive explains what each profile admits and refuses.

The hosted profile refuses to start if it is given a key of its own, listing every reason.

A sandbox says it is one: in its logs, on its health endpoint and on every screen built over it. It is never supported for value.

What it does not do

• It does not give each tenant its own database. Tenants are isolated by the service on every call and proven route by route; a company that needs physical separation runs its own deployment.
• Load spread across healthy node endpoints: by design one endpoint is used at a time, in the configured order. Not supported A sequence that hops between nodes with different views of the mempool misjudges nonces and confirmations.
• It does not repair a nonce gap on its own. A gap is an alert; filling it spends gas and is an operator’s decision.
• It does not delete a tenant’s history. Offboarding stops new work and hands over the history; the history is the audit trail.
• It does not ship a full alerting rule set. The reference observability stack has one dashboard with a functional view; further dashboards and alert rules are yours to build on the metrics.
• It has no deposit top-up command. A paymaster’s automatic top-up is configuration the service applies; the command line never signs or sends a transaction itself.

Evidence

Every claim above links to its evidence on the evidence page. The main ones: