Skip to content

Architecture

The Aetelier SDK is a Cargo workspace layered so each level depends on the ones below and not on the ones above. The story runs from raw exchange WebSocket traffic at the bottom to per-datatype Parquet at the top, with one canonical schema layer in the middle.

Ingestion has two generations that live side by side in aetelier-connect: a legacy per-venue path and a framework path. The framework path is the default for managed order-book and trade streams. Both decode the same wire types and emit the same normalized Orderbook and Trade, so downstream consumers see one schema regardless of which produced a stream.

Crate-level dependency graph

flowchart TB
    types["<b>aetelier-types</b><br/>schema<br/>(no internal deps)"]

    connect["<b>aetelier-connect</b><br/>WebSocket clients,<br/>ingestion framework,<br/>workers, sinks"]
    io["<b>aetelier-io</b><br/>Parquet / CSV / JSON<br/>readers + writers"]
    telemetry["<b>aetelier-telemetry</b><br/>OpenTelemetry<br/>instrumentation"]
    sdk["<b>aetelier-sdk</b><br/>umbrella façade<br/>(re-exports)"]

    types --> connect
    types --> io

    telemetry -.->|feature: telemetry| connect
    io --> connect

    types --> sdk
    connect --> sdk
    io --> sdk

Diagram: aetelier-types is the dependency-free schema foundation every other crate builds on. aetelier-connect and aetelier-io are the connectivity and persistence layers. aetelier-telemetry cross-cuts aetelier-connect behind its telemetry feature. aetelier-sdk is an umbrella façade re-exporting types, connect, and io.

aetelier-types has zero internal dependencies — it is the foundation every other crate depends on. aetelier-connect and aetelier-io are the connectivity and persistence layers. aetelier-telemetry cross-cuts: behind the telemetry feature it instruments aetelier-connect's workers and sinks, and is otherwise absent from the build graph. aetelier-sdk is the umbrella crate — depend on it for one import that pulls in the schema, connectivity, and I/O layers.

The two ingestion generations

Both generations decode the same wire types and emit the same normalized Orderbook and Trade. They differ in where per-venue behaviour lives.

Generation Per-venue surface Shared surface Reconstruction
Legacy path A full per-source WssClient + WssDecoder + ExchangeEvent per venue Reconnect / backoff plumbing Ad hoc, per client (BookInitializer pipeline)
Framework path ProtocolHooks + Normalizer + a ReconstructionModel choice, registered as one ExchangeAdapter WssTransport, the drive loop, the reconstruction runtime, the checksum kit, the symbol codec SourcedOrderBook / SourcedTradeBook, model-selected

A worker runs the framework path when framework_ingest = true is set under [collect] in its config and the requested venue resolves in the framework registry; otherwise it runs the legacy path. The framework path is the default for managed order-book and trade streams. The legacy WssClient / DataWorker / MarketWorker surface remains exported for bring-your-own callers and datatypes the framework does not model.

The framework path is documented across four pages:

  • Ingestion framework — the ExchangeAdapter model, ProtocolHooks, WssTransport, Normalizer, and the registry.
  • Feeds and Books — the Feed lifecycle and the SourcedOrderBook / SourcedTradeBook Book lifecycle.
  • Reconstruction models — FullRefresh, SeqDelta, ChecksumDelta, and L3, with their continuity predicates and checksum recipes.
  • Venue coverage — the coverage matrix for the twelve registered adapters.

The runtime data path

sequenceDiagram
    autonumber
    participant Exch as Exchange feed
    participant Ingest as Ingestion<br/>(framework adapter<br/>or legacy client)
    participant Worker as DataWorker /<br/>MarketWorker
    participant Sync as MarketSynchronizer
    participant Sinks as OutputSinkSet
    participant IO as aetelier-io<br/>(Parquet)

    Exch->>Ingest: WebSocket frames
    Ingest->>Worker: normalized Orderbook / Trade
    Worker->>Sync: event with timestamp
    Sync->>Worker: MarketSnapshot at grid tick
    Worker->>Sinks: MarketSnapshot
    Sinks->>IO: decomposed per-datatype rows
    IO-->>IO: per-datatype Parquet files

Diagram: exchange frames enter through either the framework adapter or a legacy client, are normalized, synchronized onto a grid into MarketSnapshots, then decomposed to per-datatype Parquet by the sinks.

Parquet on disk is where the SDK's responsibility ends. Any consumer that reads those files runs as a separate process, on its own schedule, and can run on a separate machine.

Reading the layers

Layer 1 — schema (aetelier-types)

Pure type definitions. Orderbook, Trade, MarketSnapshot, synchronizer modes, error variants. No I/O, no networking, no async. The single source of truth for what a "trade" looks like across the SDK.

A change here ripples upward by ordinary type-checking — there is no risk of two crates inventing their own Trade.

See aetelier-types.

Layer 2 — connectivity (aetelier-connect)

Exchange WebSocket clients with reconnect, backoff, and circuit breakers. The ingestion framework: per-venue ExchangeAdapters behind a shared WssTransport, normalization, model-selected reconstruction, and a registry. Two worker types: DataWorker (raw passthrough) and MarketWorker (synchronized snapshots). Output sinks: channel, terminal, Parquet.

See aetelier-connect for the crate, and Ingestion framework for the framework path in depth.

Layer 3 — persistence (aetelier-io)

Readers and writers for every top-level type, primarily Parquet. Three extension traits — FlushToParquet, FlushObSyncToParquet, FlushAggregateToParquet — that workers call into to persist streaming output, gated behind the connect + parquet features. A filename convention ({symbol}_{datatype}_{mode}_{ts}.parquet) that downstream tools parse to locate the right files for a given symbol and time range.

See aetelier-io.

Cross-cutting — telemetry (aetelier-telemetry)

OpenTelemetry instrumentation, wired into aetelier-connect behind the telemetry feature. A fixed set of metric names (MESSAGES_RECEIVED, EVENT_LATENCY_MS, WORKER_CONNECTION_STATE, SINK_QUEUE_DEPTH) shared across workers and sinks so a single operator dashboard can consume telemetry from anywhere in the SDK.

See aetelier-telemetry.

Umbrella — façade (aetelier-sdk)

The umbrella crate. Depending on aetelier-sdk pulls in aetelier-types, aetelier-connect, and aetelier-io through one import, and forwards the parquet and connect feature flags to aetelier-io.

See SDK overview.

Reading paths

Most readers want one of:

For everything else, the API reference is the exhaustive view, organized by crate.