Skip to content

aetelier-io

Persistence for the Aetelier SDK: Parquet, CSV, and JSON readers and writers for every top-level type. Designed for both online persistence (a worker streaming live data straight to Parquet) and offline analysis (loading Parquet datasets into a research notebook).

The crate is intentionally narrow. It does not transform data, does not synchronize, and does not subscribe to anything. It moves bytes between in-memory aetelier-types and on-disk files.

File format support

Parquet — columnar, the canonical format

Data type Writer Reader
Order books orderbooks::write_ob_parquet orderbooks::read_ob_parquet
Trades trades::write_trades_parquet_timestamped (auto filename) or trades::write_trades_parquet (exact path) trades::read_trades_parquet
Funding rates funding::write_funding_parquet_timestamped funding::read_funding_parquet
Liquidations liquidations::write_liquidations_parquet_timestamped liquidations::read_liquidations_parquet
Open interest open_interest::write_oi_parquet_timestamped open_interest::read_oi_parquet
Market aggregates snapshots::write_market_aggregate_parquet snapshots::read_market_aggregate_parquet

Every function lives in its per-type module (aetelier_io::trades, aetelier_io::orderbooks, …), not at the crate root. Parquet is recommended for any non-trivial dataset.

CSV & JSON — human-readable, narrower scope

Only orderbooks have text writers (orderbooks::write_csv / orderbooks::write_json, operating on an OrderbookDelta). The matching read_csv / read_json are not implemented yet (no-op stubs); use Parquet for round-trips. The other types return UnsupportedFormat for CSV/JSON. Extending text I/O is a roadmap item.

Core traits

Trait Purpose
FlushToParquet Single-shot flush for the market synchronizer's output
FlushObSyncToParquet Specialized flush for orderbook-synchronizer output, batched
FlushAggregateToParquet Optimized path for MarketAggregate snapshots

All three are re-exported at the crate root (aetelier_io::FlushToParquet, …) and are gated behind both the parquet and connect features (the whole aetelier_io::flush module requires connect). They are implemented for synchronizer/aggregate types, not for bare Orderbook/Trade values — for those, call the module writers above directly.

Filename convention

The timestamped writers produce files in this shape:

{EXCHANGE}_{SYMBOL}_{DATATYPE}_{MODE}_{TIMESTAMP}.parquet

MODE is sync for grid-aligned data or raw for unprocessed captures. Symbols containing / (e.g. Kraken's BTC/USDT) are sanitized to - in the filename (BTC-USDT) while the Parquet data retains the original symbol string.

Examples:

bybit_BTCUSDT_ob_sync_20260226_153000.123.parquet
bybit_ETHUSDT_trades_raw_20260226_160000.456.parquet
kraken_BTC-USDT_ob_sync_20260226_153000.123.parquet

Files are organized into per-datatype subdirectories: orderbooks/, trades/, liquidations/, fundings/, open_interests/.

Quick start — write & read trades

use std::path::Path;

use aetelier_io::trades::{read_trades_parquet, write_trades_parquet};

// `trades: Vec<Trade>` — collected live or produced earlier.
write_trades_parquet(&trades, Path::new("data/trades/bybit_BTCUSDT.parquet"))?;

let loaded = read_trades_parquet(Path::new("data/trades/bybit_BTCUSDT.parquet"))?;
for t in loaded.iter().take(10) {
    println!("{}  {:?} {} @ {}", t.source_trade_ts_us, t.side, t.amount, t.price);
}

write_trades_parquet takes an exact file path; use write_trades_parquet_timestamped when you want the writer to build the {exchange}_{symbol}_trades_{mode}_{ts}.parquet name for you.

Quick start — write & read an orderbook

use std::path::Path;

use aetelier_io::orderbooks::{read_ob_parquet, write_ob_parquet};
use aetelier_types::orderbooks::OrderbookTarget;

// `books: Vec<Orderbook>` — collected or reconstructed. write_ob_parquet
// writes one timestamped file into the directory, tagging the mode
// ("sync" for grid-aligned, "raw" for unprocessed).
write_ob_parquet(&books, Path::new("data/orderbooks"), "sync")?;

// Read a written file back as Vec<Orderbook> snapshots.
let data = read_ob_parquet(
    Path::new("data/orderbooks/bybit_BTCUSDT_ob_sync_20260226_153000.123.parquet"),
    OrderbookTarget::Snapshot,
)?;

OrderbookTarget::Delta reads into the BTreeMap-based OrderbookDelta form instead; the result is an OrderbookTargetData enum you match on.

Persisting live market snapshots

You rarely hand-build a MarketSnapshot to persist it. Live collection persists through the worker: a MarketWorker with a parquet output in its manifest drives ParquetSnapshotFlusher, which decomposes each snapshot and calls the per-type writers above. See Getting started for that flow. Offline, the aggregate writers (snapshots::write_market_aggregate_parquet / read_market_aggregate_parquet) round-trip MarketAggregate data.

Feature flags

Flag Effect
parquet Enable Arrow / Parquet I/O. Pulls in arrow and parquet crates
connect Enable the flush traits + ParquetSnapshotFlusher bridge to aetelier-connect workers

parquet is the one most users want. The flush traits and the worker bridge need connect and parquet.

Integration patterns

Online → Parquet — a live worker persisting straight to disk:

MarketWorker  (manifest: [[collect.output]] type = "parquet")
    │
    ▼
ParquetSnapshotFlusher  (aetelier-io, needs `connect` + `parquet`)
    │
    ▼
Parquet files  (orderbooks/, trades/, …)
    │
    ▼
aetelier-io module readers  (offline)

Batch processing — research notebooks consuming archived data:

Parquet files
    │
    ▼
aetelier-io module readers  (read_trades_parquet, read_ob_parquet, …)
    │
    ▼
Vec<Trade> / Vec<Orderbook> / …
    │
    ▼
Analytics / backtesting / model fitting

Where to go next