Events and integration

For integration engineers.

In short

Your back office needs to know two things: what happened to the operations it asked for, and what happened on the chain to the contracts it cares about. CryptoRail delivers both through the same channel, as signed webhooks, with the same guarantees: nothing is lost, every event carries an id you can deduplicate on, each stream arrives in order, and a receiver that is down for a weekend finds everything waiting when it comes back.

Contract events arrive decoded into named fields, only once they are deep enough in the chain to trust, and a reorganisation of the chain is handled for you rather than left for your systems to notice. When an auditor asks what happened, the history comes out as a file.

Subscribe to your contracts' events and receive them decoded into named fields, in chain order, once they are deep enough to trust.

At-least-once delivery through a transactional outbox, with a deterministic id on every event so your systems can drop duplicates.

Two directions, one mechanism

Events travel out, everything CryptoRail does (an operation changing state, an approval arriving, an identity paused, an alert), and in, the logs your contracts emit on the chain, which the indexer turns into events. Both are written to the same outbox and delivered by the same loop.

flowchart LR
    subgraph CR[CryptoRail]
        W[API and worker: a state change] -->|same database transaction| O[(outbox)]
        IX[indexer: decoded logs and its cursor] -->|same database transaction| O
        O --> D[delivery loop, per target]
    end
    N[Network] --> IX
    D -->|signed POST| R[Your webhook receiver]
    D --> P[A configured publisher]
    R -->|2xx| OK[delivered]
    R -->|anything else| RT[retried on a schedule, then dead letter]

Every change of state of your own operations arrives through the same channel as the contract events.

Subscriptions to contract events

A subscription asks the indexer to watch a contract and deliver its events. You name a contract registered under its ABI, the event, optional filters on its indexed parameters (only the transfers to this address, say), where to deliver, how deep a block must be before its events count, and where to start: the latest block, a given block, or the contract’s deployment block. History before the start of the live stream is backfilled at a limited pace, so a new subscription over years of history does not exhaust your node.

A subscription names a registered contract, an event and optional filters on its indexed parameters, and delivers to one of your webhook endpoints or a configured publisher.

A subscription starts from the latest block, a given block or the contract's deployment block; earlier history is backfilled at a limited pace before the subscription goes live.

An ABI that declares only part of a contract limits what you can subscribe to decoded, not what you can receive: a raw subscription by event topic takes the rest and starts decoding it from the block a later version of the ABI declares it. A contract behind an upgradeable proxy is registered once, and each event decodes with the ABI that was active at its block.

Events your ABI does not declare can be received raw by topic, and are decoded from the block a later version of the ABI declares them.

Contracts behind upgradeable proxies, and new versions of their ABI, without losing history.

Each subscription shows where it stands: its state (backfilling, live, paused), its cursor and how many blocks it is behind the head. That tells you whether an event you have not received is late or simply not confirmed yet. Pausing keeps the cursor; resuming delivers what was missed, in order.

Each subscription shows its cursor and how many blocks it is behind the head, as a field and as a metric, with an alert above your threshold.

A subscription pauses and resumes without losing its place: what was missed while paused arrives, in order, on resume.

What an event looks like

Every event, whatever its source, is one envelope: an id, a type, a schema version, a severity (info or alert), when it happened, the tenant, the network, your business operation reference when the transaction carried one, and a typed payload. A webhook endpoint can narrow what it receives by type.

The id is deterministic: chainId:blockHash:logIndex for a contract event, the transaction id and its history position for a state change. A duplicate delivery always carries the same id, so your receiver drops it with a lookup. A contract event also carries the block number and time, the transaction hash, the contract, the raw topics and data, and the decoded event name and named parameters, with quantities as decimal strings and addresses checksummed.

Your business operation travels with its transactions: its type, its id and its step appear in queries, events and traces.

Reorganisations

A block your node reports today may be replaced tomorrow by a competing one. The indexer compares the hash of the block at its cursor with the chain’s on every pass; on a mismatch it rewinds to the newest block it still recognises and processes again from there. What you see depends on the depth each subscription asked for.

flowchart TB
    R[two blocks replaced] --> D{subscription's depth}
    D -->|0: tentative| T["contract.event-reverted for each event that fell off,<br/>with its original id; then the new events, new ids"]
    D -->|deeper than the reorg| N[nothing: those events were never delivered]
    D -->|shallower than the reorg, but not 0| A["alert.indexer-deep-reorg with the block it rewound to;<br/>the region processed again"]

• Depth zero receives events at once, as tentative, and is told by id which ones a reorganisation removed.
• A depth that covers the network’s reorganisations never sees one. On networks with deterministic finality, such as a Besu QBFT consortium, following the network’s finality means depth zero with nothing ever reverted.
• A reorganisation deeper than the depth is an alert, never a silent change. What was delivered as confirmed is not retracted; the alert names the block to reconcile from, using the event ids. A subscription that raises it repeatedly has a depth too shallow for its network.

The cursor and the events it covers are written in one database transaction, so a crash between the two can neither lose a log nor deliver it twice, and a restart resumes where the last pass ended.

Chain reorganisations handled: no event lost or delivered from a discarded block, across restarts.

A subscription at depth zero receives events at once, as tentative, and a revert message carrying the original id when a reorganisation removes them.

A reorganisation deeper than a subscription's depth raises an alert naming the block it rewound to, so you reconcile on the event ids instead of finding out later.

Each network declares its finality rule: a number of blocks on top, the node's finalized block, or instant on networks with deterministic finality, with coherent defaults for public and permissioned networks.

The transactional outbox

A state change and the event that announces it are written in the same database transaction. There is no moment at which CryptoRail has changed an operation’s state but not yet recorded that it must tell you, so a crash cannot swallow an event. A separate loop then drains the outbox to each target.

• One stream per aggregate. Events about the same subscription, transaction, identity or paymaster stay in order. A batch of up to 100 events is accepted or retried as a whole, so a receiver never sees a later event of a stream before an earlier one.
• Targets are independent. A receiver that refuses delays only what is addressed to it; other endpoints and other streams keep flowing.
• Several processes may deliver. Delivery is claimed per target under a lease, so two processes never send the same batch.
• At least once, never exactly once. Over a network a lost acknowledgement looks like a lost message, so no system can honestly promise exactly once. What CryptoRail promises is that no event is lost and every duplicate carries the same id.

Events arrive in order per stream (a subscription, a transaction, an identity, a paymaster), and a target that refuses its deliveries delays only what is addressed to it.

Several workers on the same database without sending anything twice.

Signed webhooks

A delivery is one HTTPS POST of a JSON body holding the batch, with the headers of the Standard Webhooks convention: the delivery id (the same on every retry), a timestamp, and an HMAC-SHA256 signature over the id, the timestamp and the body. An attempt number and the event count come alongside. Any Standard Webhooks library verifies it; CryptoRail also ships a verifier.

sequenceDiagram
    participant CR as CryptoRail
    participant R as Your receiver
    participant S as Your system
    CR->>R: POST batch, webhook-id, webhook-timestamp, webhook-signature
    R->>R: timestamp within five minutes? if not, 401
    R->>R: HMAC over id, timestamp and body matches, in constant time? if not, 401
    R->>S: hand over each event, skipping ids already seen
    S-->>R: stored
    R-->>CR: 2xx only now

The endpoint’s secret is shown once when you register it, stored sealed under a key from your secret manager, and rotated without a gap: during the overlap every delivery carries both signatures. Endpoints must use HTTPS unless the deployment has explicitly allowed otherwise. The security model covers the sealing and both rotations.

Webhooks follow the Standard Webhooks convention, so a receiver verifies them with an existing library; the delivery id stays the same on every retry.

Each webhook endpoint gets a secret shown once, sealed at rest under a key from your secret manager, and rotated without a gap; endpoints must be HTTPS unless the deployment explicitly allows otherwise.

Retries and dead letters

Anything other than a 2xx, a timeout or a refused connection is one failed attempt. Attempts are retried on an exponential schedule, from one second up to an hour between attempts, for about 72 hours by default: long enough for a receiver that is down over a weekend. After that the delivery is dead-lettered: kept, with every attempt and what your receiver answered each time, until a person acts.

stateDiagram-v2
    [*] --> pending
    pending --> delivered: 2xx
    pending --> failed: other status, timeout, refused
    failed --> pending: next attempt on the schedule
    failed --> dead: schedule exhausted, about 72 hours
    dead --> pending: operator redelivers, with a reason
    delivered --> [*]

A dead letter is not a lost event. An operator lists the dead letters and puts one back, with a reason that goes into the audit log; it arrives with the next attempt number, so your receiver can tell. A redelivered dead letter arrives after whatever was delivered while it was dead: a receiver that depends on order reconciles on the event ids. Every delivery can be read with its attempts, per endpoint or per subscription.

Signed webhooks, retried for about three days, then held in a dead-letter queue an operator can redeliver from.

Every delivery can be read with its status and each attempt, with what your receiver answered, per webhook endpoint or per subscription.

An operator command line for the daily work: pause an identity, redeliver a webhook, reconcile nonces, export a history, onboard a tenant.

Exports for audit

An auditor does not read a database. The complete history of every transaction in a period, every state with its cause and time, every attempt, the receipt, the cost, the failure report and the policy decisions, exports as NDJSON, one transaction per line. The same exists for a subscription: the subscription, then every delivery of its stream with its attempts. Both stream, so neither side holds a month in memory, and both end with a summary line stating the file is complete: an export without that line is incomplete, whatever it looks like. A range too large to export is refused with a number to act on, never cut short.

The complete history of any operation, exported in a machine-readable file for audit.

The history of a subscription, every delivery with its attempts, exported as NDJSON whose last line states that the file is complete.

Every approval, signature and decision is recorded with who, when and what, and every change made through the API lands in an audit log.

Integrating a receiver

What a receiver must do, in order:

1. Verify the timestamp and the signature before believing anything in the body.
2. Keep both secrets during a rotation.
3. Deduplicate on each event’s id, not on the delivery id: a batch can be redelivered.
4. Answer 2xx only once the batch is safely stored; anything else is retried.
5. Treat contract.event-reverted as an instruction to undo the event it names, if you subscribed at depth zero.
6. Route alerts, such as a deep reorganisation or indexer lag, to whoever acts on them.
7. Reconcile on ids after an operator redelivers a dead letter, if your processing depends on order.

What it does not do

• It does not promise exactly-once delivery. It promises that nothing is lost and that every duplicate carries the same id.
• It does not order events across streams. Each subscription, transaction or identity is in order; two streams relative to each other are not.
• It does not index a whole network. Subscriptions are per registered contract or address; every transfer of every token on a chain is an indexer product of its own and out of scope.
• It does not retract a confirmed event. A reorganisation deeper than a subscription’s depth is an alert for you to reconcile, not a silent rewrite.
• It does not keep a redelivered dead letter in its original place in the stream; it arrives when it is put back.
• Events delivered to Kafka, NATS or AMQP brokers. On request

Evidence

Every claim above links to its evidence on the evidence page. The main ones: