Data Governance¶
Governance tag: overwatch
Data Governance lets a tenant govern their own artifact streams: the feeds an agent produces (orderbook, trades, funding, open interest, liquidations) are reconciled against what was declared, scored for completeness and quality, and tracked file-by-file with content-addressed lineage. Governance is scoped to the streams a tenant owns — one tenant never sees another's.
Two surfaces¶
| Surface | What it covers | Where it runs |
|---|---|---|
| Streaming telemetry + lineage | Stream inventory, freshness, declared-vs-observed reconciliation, completeness and quality scorecards, network placement alerts | Continuously, over the live stream |
| At-rest ingestion | Parquet-to-ClickHouse ingestion with path-derived lineage, sha256 provenance, and exactly-once landing into a queryable system-of-record | On the tenant's archived Parquet |
Per-tenant artifact-stream model¶
A stream is one (exchange, market_type, symbol, datatype) tuple an agent
produces. Every governance dimension keys on that tuple. Each stream carries a
binding_id — the deploy that owns it — so all reconciliation, scoring, and
lineage stays isolated to the owning tenant.
flowchart LR
A[Agent artifact streams] --> D[Declared streams]
A --> O[Observed reality]
D --> C[Catalog reconciliation]
O --> C
C --> S[Completeness + Quality scorecards]
O --> AL[Network placement alerts]
A -.archived Parquet.-> IN[At-rest ingestion]
IN --> SOR[Queryable system-of-record]
Dimensions¶
-
Catalog
Stream inventory: declared-vs-observed reconciliation, freshness, presence.
-
Completeness
Sequence and bucket scoring — how much of the expected stream actually landed.
-
Quality
Per-datatype check families over the stored records.
-
Alerts
Network placement alerts with a debounced firing-to-resolved lifecycle.
Surfaces¶
| Surface | What it is | Reference |
|---|---|---|
| Governance API | Read the catalog, per-stream scorecards, and resolve a dataset to its files | api.md |
| At-rest ingestion | Parquet to ClickHouse with lineage and exactly-once landing | ingestion.md |
What a tenant gets for their own streams¶
| Question | Answered by |
|---|---|
| Which streams do I own, and are they fresh? | Catalog |
| Did every expected bucket arrive? | Completeness |
| Is the stored data valid? | Quality |
| Is a collector deployed in a high-latency region? | Alerts |
| Where are the files behind a dataset? | API — datasets |
| Can I query my persisted streams at rest? | At-rest ingestion |