aetelier-io¶
Persistence for the Aetelier SDK: Parquet, CSV, and JSON readers and writers for every top-level type. Designed for both online persistence (a worker streaming live data straight to Parquet) and offline analysis (loading Parquet datasets into a research notebook).
The crate is intentionally narrow. It does not transform data, does
not synchronize, and does not subscribe to anything. It moves bytes
between in-memory aetelier-types and on-disk files.
File format support¶
Parquet — columnar, the canonical format¶
| Data type | Writer | Reader |
|---|---|---|
| Order books | orderbooks::write_ob_parquet |
orderbooks::read_ob_parquet |
| Trades | trades::write_trades_parquet_timestamped (auto filename) or trades::write_trades_parquet (exact path) |
trades::read_trades_parquet |
| Funding rates | funding::write_funding_parquet_timestamped |
funding::read_funding_parquet |
| Liquidations | liquidations::write_liquidations_parquet_timestamped |
liquidations::read_liquidations_parquet |
| Open interest | open_interest::write_oi_parquet_timestamped |
open_interest::read_oi_parquet |
| Market aggregates | snapshots::write_market_aggregate_parquet |
snapshots::read_market_aggregate_parquet |
Every function lives in its per-type module (aetelier_io::trades,
aetelier_io::orderbooks, …), not at the crate root. Parquet is
recommended for any non-trivial dataset.
CSV & JSON — human-readable, narrower scope¶
Only orderbooks have text writers (orderbooks::write_csv /
orderbooks::write_json, operating on an OrderbookDelta). The
matching read_csv / read_json are not implemented yet (no-op
stubs); use Parquet for round-trips. The other types return
UnsupportedFormat for CSV/JSON. Extending text I/O is a roadmap item.
Core traits¶
| Trait | Purpose |
|---|---|
FlushToParquet |
Single-shot flush for the market synchronizer's output |
FlushObSyncToParquet |
Specialized flush for orderbook-synchronizer output, batched |
FlushAggregateToParquet |
Optimized path for MarketAggregate snapshots |
All three are re-exported at the crate root
(aetelier_io::FlushToParquet, …) and are gated behind both the
parquet and connect features (the whole aetelier_io::flush module
requires connect). They are implemented for synchronizer/aggregate
types, not for bare Orderbook/Trade values — for those, call the
module writers above directly.
Filename convention¶
The timestamped writers produce files in this shape:
MODE is sync for grid-aligned data or raw for unprocessed
captures. Symbols containing / (e.g. Kraken's BTC/USDT) are
sanitized to - in the filename (BTC-USDT) while the Parquet data
retains the original symbol string.
Examples:
bybit_BTCUSDT_ob_sync_20260226_153000.123.parquet
bybit_ETHUSDT_trades_raw_20260226_160000.456.parquet
kraken_BTC-USDT_ob_sync_20260226_153000.123.parquet
Files are organized into per-datatype subdirectories: orderbooks/,
trades/, liquidations/, fundings/, open_interests/.
Quick start — write & read trades¶
use std::path::Path;
use aetelier_io::trades::{read_trades_parquet, write_trades_parquet};
// `trades: Vec<Trade>` — collected live or produced earlier.
write_trades_parquet(&trades, Path::new("data/trades/bybit_BTCUSDT.parquet"))?;
let loaded = read_trades_parquet(Path::new("data/trades/bybit_BTCUSDT.parquet"))?;
for t in loaded.iter().take(10) {
println!("{} {:?} {} @ {}", t.source_trade_ts_us, t.side, t.amount, t.price);
}
write_trades_parquet takes an exact file path; use
write_trades_parquet_timestamped when you want the writer to build the
{exchange}_{symbol}_trades_{mode}_{ts}.parquet name for you.
Quick start — write & read an orderbook¶
use std::path::Path;
use aetelier_io::orderbooks::{read_ob_parquet, write_ob_parquet};
use aetelier_types::orderbooks::OrderbookTarget;
// `books: Vec<Orderbook>` — collected or reconstructed. write_ob_parquet
// writes one timestamped file into the directory, tagging the mode
// ("sync" for grid-aligned, "raw" for unprocessed).
write_ob_parquet(&books, Path::new("data/orderbooks"), "sync")?;
// Read a written file back as Vec<Orderbook> snapshots.
let data = read_ob_parquet(
Path::new("data/orderbooks/bybit_BTCUSDT_ob_sync_20260226_153000.123.parquet"),
OrderbookTarget::Snapshot,
)?;
OrderbookTarget::Delta reads into the BTreeMap-based OrderbookDelta
form instead; the result is an OrderbookTargetData enum you match on.
Persisting live market snapshots¶
You rarely hand-build a MarketSnapshot to persist it. Live collection
persists through the worker: a MarketWorker with a parquet output in
its manifest drives ParquetSnapshotFlusher, which decomposes each
snapshot and calls the per-type writers above. See
Getting started for that flow. Offline, the
aggregate writers (snapshots::write_market_aggregate_parquet /
read_market_aggregate_parquet) round-trip MarketAggregate data.
Feature flags¶
| Flag | Effect |
|---|---|
parquet |
Enable Arrow / Parquet I/O. Pulls in arrow and parquet crates |
connect |
Enable the flush traits + ParquetSnapshotFlusher bridge to aetelier-connect workers |
parquet is the one most users want. The flush traits and the
worker bridge need connect and parquet.
Integration patterns¶
Online → Parquet — a live worker persisting straight to disk:
MarketWorker (manifest: [[collect.output]] type = "parquet")
│
▼
ParquetSnapshotFlusher (aetelier-io, needs `connect` + `parquet`)
│
▼
Parquet files (orderbooks/, trades/, …)
│
▼
aetelier-io module readers (offline)
Batch processing — research notebooks consuming archived data:
Parquet files
│
▼
aetelier-io module readers (read_trades_parquet, read_ob_parquet, …)
│
▼
Vec<Trade> / Vec<Orderbook> / …
│
▼
Analytics / backtesting / model fitting
Where to go next¶
- API reference for
aetelier-io - Tutorial: Bybit → Parquet — end-to-end persistence via the worker manifest.
aetelier-connect— what produces the data this crate persists.aetelier-types— the schema all writers serialize.