Skip to content

Data Governance

Governance tag: overwatch

Data Governance lets a tenant govern their own artifact streams: the feeds an agent produces (orderbook, trades, funding, open interest, liquidations) are reconciled against what was declared, scored for completeness and quality, and tracked file-by-file with content-addressed lineage. Governance is scoped to the streams a tenant owns — one tenant never sees another's.

Two surfaces

Surface What it covers Where it runs
Streaming telemetry + lineage Stream inventory, freshness, declared-vs-observed reconciliation, completeness and quality scorecards, network placement alerts Continuously, over the live stream
At-rest ingestion Parquet-to-ClickHouse ingestion with path-derived lineage, sha256 provenance, and exactly-once landing into a queryable system-of-record On the tenant's archived Parquet

Per-tenant artifact-stream model

A stream is one (exchange, market_type, symbol, datatype) tuple an agent produces. Every governance dimension keys on that tuple. Each stream carries a binding_id — the deploy that owns it — so all reconciliation, scoring, and lineage stays isolated to the owning tenant.

flowchart LR
  A[Agent artifact streams] --> D[Declared streams]
  A --> O[Observed reality]
  D --> C[Catalog reconciliation]
  O --> C
  C --> S[Completeness + Quality scorecards]
  O --> AL[Network placement alerts]
  A -.archived Parquet.-> IN[At-rest ingestion]
  IN --> SOR[Queryable system-of-record]
An agent's own streams flow into a declared set and an observed reality; their reconciliation feeds completeness and quality scorecards; observed network latency feeds placement alerts; archived Parquet is separately ingested into a queryable system-of-record.

Dimensions

  • Catalog


    Stream inventory: declared-vs-observed reconciliation, freshness, presence.

    Read catalog →

  • Completeness


    Sequence and bucket scoring — how much of the expected stream actually landed.

    Read completeness →

  • Quality


    Per-datatype check families over the stored records.

    Read quality →

  • Alerts


    Network placement alerts with a debounced firing-to-resolved lifecycle.

    Read alerts →

Surfaces

Surface What it is Reference
Governance API Read the catalog, per-stream scorecards, and resolve a dataset to its files api.md
At-rest ingestion Parquet to ClickHouse with lineage and exactly-once landing ingestion.md

What a tenant gets for their own streams

Question Answered by
Which streams do I own, and are they fresh? Catalog
Did every expected bucket arrive? Completeness
Is the stored data valid? Quality
Is a collector deployed in a high-latency region? Alerts
Where are the files behind a dataset? API — datasets
Can I query my persisted streams at rest? At-rest ingestion