Architecture¶
The Aetelier SDK is a Cargo workspace layered so each level depends on the ones below and not on the ones above. The story runs from raw exchange WebSocket traffic at the bottom to per-datatype Parquet at the top, with one canonical schema layer in the middle.
Ingestion has two generations that live side by side in
aetelier-connect: a legacy per-venue path and a framework path.
The framework path is the default for managed order-book and trade
streams. Both decode the same wire types and emit the same normalized
Orderbook and Trade, so downstream consumers see one schema
regardless of which produced a stream.
Crate-level dependency graph¶
flowchart TB
types["<b>aetelier-types</b><br/>schema<br/>(no internal deps)"]
connect["<b>aetelier-connect</b><br/>WebSocket clients,<br/>ingestion framework,<br/>workers, sinks"]
io["<b>aetelier-io</b><br/>Parquet / CSV / JSON<br/>readers + writers"]
telemetry["<b>aetelier-telemetry</b><br/>OpenTelemetry<br/>instrumentation"]
sdk["<b>aetelier-sdk</b><br/>umbrella façade<br/>(re-exports)"]
types --> connect
types --> io
telemetry -.->|feature: telemetry| connect
io --> connect
types --> sdk
connect --> sdk
io --> sdk
Diagram: aetelier-types is the dependency-free schema foundation every
other crate builds on. aetelier-connect and aetelier-io are the
connectivity and persistence layers. aetelier-telemetry cross-cuts
aetelier-connect behind its telemetry feature. aetelier-sdk is an
umbrella façade re-exporting types, connect, and io.
aetelier-types has zero internal dependencies — it is the foundation
every other crate depends on. aetelier-connect and aetelier-io are the
connectivity and persistence layers. aetelier-telemetry cross-cuts:
behind the telemetry feature it instruments aetelier-connect's workers
and sinks, and is otherwise absent from the build graph. aetelier-sdk is
the umbrella crate — depend on it for one import that pulls in the schema,
connectivity, and I/O layers.
The two ingestion generations¶
Both generations decode the same wire types and emit the same normalized
Orderbook and Trade. They differ in where per-venue behaviour lives.
| Generation | Per-venue surface | Shared surface | Reconstruction |
|---|---|---|---|
| Legacy path | A full per-source WssClient + WssDecoder + ExchangeEvent per venue |
Reconnect / backoff plumbing | Ad hoc, per client (BookInitializer pipeline) |
| Framework path | ProtocolHooks + Normalizer + a ReconstructionModel choice, registered as one ExchangeAdapter |
WssTransport, the drive loop, the reconstruction runtime, the checksum kit, the symbol codec |
SourcedOrderBook / SourcedTradeBook, model-selected |
A worker runs the framework path when framework_ingest = true is set
under [collect] in its config and the requested venue resolves in the
framework registry; otherwise it runs the legacy path. The framework path
is the default for managed order-book and trade streams. The legacy
WssClient / DataWorker / MarketWorker surface remains exported for
bring-your-own callers and datatypes the framework does not model.
The framework path is documented across four pages:
- Ingestion framework — the
ExchangeAdaptermodel,ProtocolHooks,WssTransport,Normalizer, and the registry. - Feeds and Books — the
Feedlifecycle and theSourcedOrderBook/SourcedTradeBookBook lifecycle. - Reconstruction models —
FullRefresh,SeqDelta,ChecksumDelta, andL3, with their continuity predicates and checksum recipes. - Venue coverage — the coverage matrix for the twelve registered adapters.
The runtime data path¶
sequenceDiagram
autonumber
participant Exch as Exchange feed
participant Ingest as Ingestion<br/>(framework adapter<br/>or legacy client)
participant Worker as DataWorker /<br/>MarketWorker
participant Sync as MarketSynchronizer
participant Sinks as OutputSinkSet
participant IO as aetelier-io<br/>(Parquet)
Exch->>Ingest: WebSocket frames
Ingest->>Worker: normalized Orderbook / Trade
Worker->>Sync: event with timestamp
Sync->>Worker: MarketSnapshot at grid tick
Worker->>Sinks: MarketSnapshot
Sinks->>IO: decomposed per-datatype rows
IO-->>IO: per-datatype Parquet files
Diagram: exchange frames enter through either the framework adapter or a
legacy client, are normalized, synchronized onto a grid into
MarketSnapshots, then decomposed to per-datatype Parquet by the sinks.
Parquet on disk is where the SDK's responsibility ends. Any consumer that reads those files runs as a separate process, on its own schedule, and can run on a separate machine.
Reading the layers¶
Layer 1 — schema (aetelier-types)¶
Pure type definitions. Orderbook, Trade, MarketSnapshot,
synchronizer modes, error variants. No I/O, no networking, no async. The
single source of truth for what a "trade" looks like across the SDK.
A change here ripples upward by ordinary type-checking — there is no risk
of two crates inventing their own Trade.
See aetelier-types.
Layer 2 — connectivity (aetelier-connect)¶
Exchange WebSocket clients with reconnect, backoff, and circuit breakers.
The ingestion framework: per-venue ExchangeAdapters behind a shared
WssTransport, normalization, model-selected reconstruction, and a
registry. Two worker types: DataWorker (raw passthrough) and
MarketWorker (synchronized snapshots). Output sinks: channel, terminal,
Parquet.
See aetelier-connect for the crate, and
Ingestion framework for the framework path in depth.
Layer 3 — persistence (aetelier-io)¶
Readers and writers for every top-level type, primarily Parquet. Three
extension traits — FlushToParquet, FlushObSyncToParquet,
FlushAggregateToParquet — that workers call into to persist streaming
output, gated behind the connect + parquet features. A filename
convention ({symbol}_{datatype}_{mode}_{ts}.parquet) that downstream
tools parse to locate the right files for a given symbol and time range.
See aetelier-io.
Cross-cutting — telemetry (aetelier-telemetry)¶
OpenTelemetry instrumentation, wired into aetelier-connect behind the
telemetry feature. A fixed set of metric names (MESSAGES_RECEIVED,
EVENT_LATENCY_MS, WORKER_CONNECTION_STATE, SINK_QUEUE_DEPTH) shared
across workers and sinks so a single operator dashboard can consume
telemetry from anywhere in the SDK.
See aetelier-telemetry.
Umbrella — façade (aetelier-sdk)¶
The umbrella crate. Depending on aetelier-sdk pulls in aetelier-types,
aetelier-connect, and aetelier-io through one import, and forwards the
parquet and connect feature flags to aetelier-io.
See SDK overview.
Reading paths¶
Most readers want one of:
- "I want to collect live data." Start with Getting started, then Tutorial 1: Bybit → Parquet.
- "I want to integrate or understand a venue." Read Ingestion framework and Venue coverage, then Tutorial 2: multi-exchange sync.
- "I want to read the data I already collected." Skip the
connectivity layer and read
aetelier-io.
For everything else, the API reference is the exhaustive view, organized by crate.