Skip to content

Architecture

The Aetelier SDK is a Cargo workspace layered so each level depends on the ones below and not on the ones above. The story runs from raw exchange WebSocket traffic at the bottom to fitted quantitative models at the top, with one canonical schema layer in the middle.

Ingestion has two generations that live side by side in atelier-connect: a legacy per-venue path and a framework path. The framework path is the default for managed order-book and trade streams. Both decode the same wire types and emit the same normalized Orderbook and Trade, so downstream consumers see one schema regardless of which produced a stream.

Crate-level dependency graph

flowchart TB
    types["<b>atelier-types</b><br/>schema<br/>(no internal deps)"]

    connect["<b>atelier-connect</b><br/>WebSocket clients,<br/>ingestion framework,<br/>workers, sinks"]
    io["<b>atelier-io</b><br/>Parquet / CSV / JSON<br/>readers + writers"]
    telemetry["<b>atelier-telemetry</b><br/>OpenTelemetry<br/>instrumentation"]
    sdk["<b>atelier-sdk</b><br/>umbrella façade<br/>(re-exports)"]
    agent["<b>atelier-agent</b><br/>(binary)<br/>remote runner"]

    types --> connect
    types --> io

    telemetry -.->|feature: telemetry| connect
    io --> connect

    types --> sdk
    connect --> sdk
    io --> sdk

    connect -->|feature: gateway| agent

Diagram: atelier-types is the dependency-free schema foundation every other crate builds on. atelier-connect and atelier-io are the connectivity and persistence layers. atelier-telemetry cross-cuts atelier-connect behind its telemetry feature. atelier-sdk is an umbrella façade re-exporting types, connect, and io. atelier-agent is a deployable binary built on atelier-connect's gateway feature.

atelier-types has zero internal dependencies — it is the foundation every other crate depends on. atelier-connect and atelier-io are the connectivity and persistence layers. atelier-telemetry cross-cuts: behind the telemetry feature it instruments atelier-connect's workers and sinks, and is otherwise absent from the build graph. atelier-sdk is the umbrella crate — depend on it for one import that pulls in the schema, connectivity, and I/O layers. atelier-agent is excluded from the SDK workspace and built from the monorepo root, because its gateway feature pulls in atelier-proto.

Companion crates

atelier-data (Arrow-backed columnar pipeline) and atelier-quant (point-process models) are documented alongside the SDK crates. They ship on their own release cadence rather than as members of the SDK workspace. See atelier-data and atelier-quant.

The two ingestion generations

Both generations decode the same wire types and emit the same normalized Orderbook and Trade. They differ in where per-venue behaviour lives.

Generation Per-venue surface Shared surface Reconstruction
Legacy path A full per-source WssClient + WssDecoder + ExchangeEvent per venue Reconnect / backoff plumbing Ad hoc, per client (BookInitializer pipeline)
Framework path ProtocolHooks + Normalizer + a ReconstructionModel choice, registered as one ExchangeAdapter WssTransport, the drive loop, the reconstruction runtime, the checksum kit, the symbol codec SourcedOrderBook / SourcedTradeBook, model-selected

A worker runs the framework path when framework_ingest = true is set under [collect] in its config and the requested venue resolves in the framework registry; otherwise it runs the legacy path. The framework path is the default for managed order-book and trade streams. The legacy WssClient / DataWorker / MarketWorker surface remains exported for bring-your-own callers and datatypes the framework does not model.

The framework path is documented across four pages:

  • Ingestion framework — the ExchangeAdapter model, ProtocolHooks, WssTransport, Normalizer, and the registry.
  • Feeds and Books — the Feed lifecycle and the SourcedOrderBook / SourcedTradeBook Book lifecycle.
  • Reconstruction models — FullRefresh, SeqDelta, ChecksumDelta, and L3, with their continuity predicates and checksum recipes.
  • Venue coverage — the coverage matrix for the twelve registered adapters.

The runtime data path

sequenceDiagram
    autonumber
    participant Exch as Exchange feed
    participant Ingest as Ingestion<br/>(framework adapter<br/>or legacy client)
    participant Worker as DataWorker /<br/>MarketWorker
    participant Sync as MarketSynchronizer
    participant Sinks as OutputSinkSet
    participant IO as atelier-io<br/>(Parquet)
    participant Quant as Point-process fit<br/>(offline)

    Exch->>Ingest: WebSocket frames
    Ingest->>Worker: normalized Orderbook / Trade
    Worker->>Sync: event with timestamp
    Sync->>Worker: MarketSnapshot at grid tick
    Worker->>Sinks: MarketSnapshot
    Sinks->>IO: decomposed per-datatype rows
    IO-->>IO: per-datatype Parquet files
    Note over IO,Quant: offline boundary
    IO->>Quant: read interarrivals from Parquet
    Quant->>Quant: fit Hawkes / Poisson via MLE
    Quant-->>Quant: model artifact on disk

Diagram: exchange frames enter through either the framework adapter or a legacy client, are normalized, synchronized onto a grid into MarketSnapshots, decomposed to per-datatype Parquet by the sinks, then — across an offline boundary — read back for point-process fitting.

The offline boundary is intentional: the live worker loop and the model-fitting loop are separate processes, on separate schedules, and can run on separate machines. Parquet on disk is the contract between them.

Reading the layers

Layer 1 — schema (atelier-types)

Pure type definitions. Orderbook, Trade, MarketSnapshot, synchronizer modes, error variants. No I/O, no networking, no async. The single source of truth for what a "trade" looks like across the SDK.

A change here ripples upward by ordinary type-checking — there is no risk of two crates inventing their own Trade.

See atelier-types.

Layer 2 — connectivity (atelier-connect)

Exchange WebSocket clients with reconnect, backoff, and circuit breakers. The ingestion framework: per-venue ExchangeAdapters behind a shared WssTransport, normalization, model-selected reconstruction, and a registry. Two worker types: DataWorker (raw passthrough) and MarketWorker (synchronized snapshots). Output sinks: channel, terminal, Parquet.

See atelier-connect for the crate, and Ingestion framework for the framework path in depth.

Layer 3 — persistence (atelier-io)

Readers and writers for every top-level type, primarily Parquet. Three extension traits — FlushToParquet, FlushObSyncToParquet, FlushAggregateToParquet — that workers call into to persist streaming output, gated behind the connect + parquet features. A filename convention ({symbol}_{datatype}_{mode}_{ts}.parquet) that downstream tools parse to locate the right files for a given symbol and time range.

See atelier-io.

Cross-cutting — telemetry (atelier-telemetry)

OpenTelemetry instrumentation, wired into atelier-connect behind the telemetry feature. A fixed set of metric names (MESSAGES_RECEIVED, EVENT_LATENCY_MS, WORKER_CONNECTION_STATE, SINK_QUEUE_DEPTH) shared across workers and sinks so a single operator dashboard can consume telemetry from anywhere in the SDK.

See atelier-telemetry.

Umbrella — façade (atelier-sdk)

The umbrella crate. Depending on atelier-sdk pulls in atelier-types, atelier-connect, and atelier-io through one import, and forwards the parquet, connect, and torch feature flags to atelier-io.

See SDK overview.

Deployable — agent (atelier-agent)

Binary-only. Remote agent that connects to the Aetelier Gateway over JWT, accepts work assignments, spawns workers, and streams telemetry back. Built on atelier-connect's gateway feature and documented in Operations → atelier-agent since it has no library API.

Reading paths

Most readers want one of:

For everything else, the API reference is the exhaustive view, organized by crate.