aetelier-connect¶
Real-time exchange connectivity for the Aetelier SDK. Manages
WebSocket connections to four cryptocurrency exchanges, decodes
exchange-native frames into typed ExchangeEvents, runs them through
configurable worker loops, and emits the result via pluggable
output sinks.
If aetelier-types is the schema, aetelier-connect is the engine
that fills the schema with live data. It is the SDK's single largest
crate by both surface area and complexity.
Supported exchanges¶
| Exchange | API | Order books | Trades | Liquidations | Funding | Open interest |
|---|---|---|---|---|---|---|
| Bybit | WSS v5 (linear) | yes | yes | yes* | yes* | yes* |
| Binance | WSS + REST (spot) | yes | yes | — | — | — |
| Coinbase | WSS (Adv. Trade) | yes | yes | — | — | — |
| Kraken | WSS v2 (spot) | yes | yes | — | — | — |
Dashes indicate the exchange does not expose that data type on its
current WebSocket endpoint. Entries marked * are decoded by the
legacy WssClient path only — the framework adapter (DomainEvent)
normalizes order books and trades; see Venues for the
framework support matrix. Bybit currently has the most complete
coverage; expanding the others to perpetual / futures endpoints is a
roadmap item.
Each exchange ships with a typed decoder (WssDecoder impl), a
response model deserialized via serde::Deserialize, and an event
classifier that maps raw frames into the unified ExchangeEvent
enum. Binance additionally has a REST client used by the
BookInitializer pipeline stage to seed orderbook state.
The data path¶
Exchange WebSocket
│
▼
Exchange client (WssClient<D>) ─── reconnect, backoff, health check
│
▼
WssDecoder ─── exchange-native JSON → typed ExchangeEvent
│
▼
Event pipeline (PassthroughPipeline | BookInitializer | …)
│
▼
Worker (DataWorker | MarketWorker)
│
▼
OutputSinkSet ─── ChannelSink, TerminalSink, ParquetSink (fan-out)
Connection management¶
| Component | Purpose |
|---|---|
WssClient<D> |
Generic WebSocket client over a WssDecoder |
WssClientBuilder |
Fluent construction with URL, headers, decoder |
HttpClient |
Rate-limited HTTP, per-exchange base URL, timeout |
ConnectionManager |
Lifecycle tracker — state transitions and timestamps for every connection |
ReconnectPolicy |
Jittered exponential backoff (100 ms → 10 s) with circuit breaker |
DisconnectReason |
Classifies low-level errors as retryable vs non-retryable so transient TCP resets don't kill the worker |
The disconnect classifier is the load-bearing piece for production reliability — it distinguishes "the network blinked" from "the exchange rejected us," ensuring the former triggers backoff-and-retry while the latter surfaces as an error.
Workers¶
Two worker types do end-to-end data collection. Both are built on a
shared IngestionCore that handles the connection lifecycle, stale-
connection detection, per-topic gap tracking via GapDetector /
GapDetectorSet, and health monitoring.
DataWorker — raw event ingestion¶
Lightweight option for collecting raw feed data with minimal
processing. Decodes exchange messages into typed events and delivers
them through a pluggable OutputSink pipeline. TOML-driven
configuration via DataWorkerManifest. Reconnection, backoff, and
gap detection are automatic.
Use when: you want the raw event stream and will do your own synchronization downstream.
MarketWorker — synchronized snapshots¶
Extends DataWorker's ingestion with a MarketSynchronizer that
bins heterogeneous events onto a uniform nanosecond grid, producing
MarketSnapshot objects at each tick. Snapshots flow through the
same OutputSink pipeline and can be flushed to Parquet
automatically.
Use when: you want time-aligned snapshots ready for downstream analytics or model fitting.
ObSynchronizer¶
Time-grid sampler specifically for orderbook snapshots. Reduces data volume via configurable snapshot intervals and handles missing data points gracefully. Useful when you only care about orderbook state on a regular cadence rather than every delta.
Synchronizer clock modes¶
MarketSynchronizer supports four ways of advancing the grid clock:
| Mode | Grid driven by |
|---|---|
OrderbookDriven |
Orderbook timestamp crossings (default) |
TradeDriven |
Trade timestamps |
LiquidationDriven |
Liquidation timestamps |
ExternalClock |
Explicit nanosecond on_time() calls |
State-based feeds (orderbook, funding rate, open interest) carry their latest value forward across grid ticks. Event-based feeds (trades, liquidations) collect all events that fell within each grid period.
Output sinks¶
The OutputSink trait defines where worker output goes. Multiple
sinks run simultaneously via OutputSinkSet (fan-out):
| Sink | What it does |
|---|---|
ChannelSink |
Wraps TopicRegistry broadcast channels for in-process pub/sub |
TerminalSink |
Tracing-formatted terminal output for debugging |
ParquetSink |
Buffers MarketSnapshots, decomposes them, flushes per-datatype Parquet files |
ParquetSink is feature-gated behind the parquet feature.
Quick start — Bybit MarketWorker¶
Workers are configured from a TOML manifest and driven with a shutdown signal:
use aetelier_connect::{
config::workers::MarketWorkerManifest, workers::market_worker::MarketWorker,
};
use tokio::sync::watch;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let manifest = MarketWorkerManifest::from_toml("worker.toml")?;
let config = manifest.resolve_all().into_iter().next().unwrap();
let (shutdown_tx, shutdown_rx) = watch::channel(false);
tokio::spawn(async move {
tokio::signal::ctrl_c().await.expect("ctrl-c handler");
let _ = shutdown_tx.send(true);
});
let worker = MarketWorker::from_config(config)?;
let report = worker.run(shutdown_rx).await?;
println!("{} snapshots", report.snapshots_produced);
Ok(())
}
The manifest names the exchange, symbols, data types, sync grid, and
outputs — see Getting started for a complete
one, or the md_worker example in the repo for per-exchange variants.
Multi-exchange fan-out¶
ConnectionManager is the per-connection reconnect state machine
(connect → backoff → resume) that every exchange client runs on
internally — it is not a multi-source registry. Fanning out across
exchanges is done by resolving a multi-worker manifest and spawning
one MarketWorker per stream, as in the multi_sync example. See the
Multi-exchange sync tutorial.
Features¶
parquet(optional) — enablesParquetSinkfor high-volume persistence.- All exchange clients and in-process workers are available without features.
Where to go next¶
- Tutorial: Bybit → Parquet
- Tutorial: multi-exchange sync
- API reference for
aetelier-connect aetelier-types— the wire types this crate emits.aetelier-io— whatParquetSinkwrites to.