Skip to content

Data Governance

Governance tag: overwatch · Min version: overwatch-serve / overwatch-ingest 0.0.1

Data Governance lets a tenant govern their own artifact streams: the feeds an agent produces (orderbook, trades, funding, open interest, liquidations) are reconciled against what was declared, scored for completeness and quality, and tracked file-by-file with content-addressed lineage. Governance is scoped to the streams a tenant owns — one tenant never sees another's.

Two tiers

Tier What it covers Where it runs You get it for
Universal telemetry + lineage Stream inventory, freshness, declared-vs-observed reconciliation, completeness and quality scorecards, firing/resolved alerts Continuously, over the live stream Every governed artifact stream
Premium at-rest Parquet-to-ClickHouse ingestion with path-derived lineage, sha256 provenance, and exactly-once landing into a queryable system-of-record On the tenant's archived Parquet Streams a tenant elects to persist at rest

Per-tenant artifact-stream model

A stream is one (exchange, market_type, symbol, datatype) tuple an agent produces. Every governance dimension keys on that tuple. Each stream carries a binding_id — the deploy that owns it — so all reconciliation, scoring, and lineage stays isolated to the owning tenant.

flowchart LR
  A[Agent artifact streams] --> D[Declared streams]
  A --> O[Observed reality]
  D --> C[Catalog reconciliation]
  O --> C
  C --> S[Completeness + Quality scorecards]
  S --> AL[Alerts firing to resolved]
  A -.archived Parquet.-> IN[At-rest ingestion]
  IN --> SOR[Queryable system-of-record]
An agent's own streams flow into a declared set and an observed reality; their reconciliation feeds completeness and quality scorecards, which raise alerts; archived Parquet is separately ingested into a queryable system-of-record.

Dimensions

  • Catalog

    Stream inventory: declared-vs-observed reconciliation, freshness, presence.

    Read catalog →

  • Completeness

    Sequence and bucket scoring — how much of the expected stream actually landed.

    Read completeness →

  • Quality

    Per-datatype check families over the stored records.

    Read quality →

  • Alerts

    Firing-to-resolved lifecycle, debounced, over the notification channels.

    Read alerts →

Surfaces

Surface What it is Reference
Governance API Read the catalog, per-stream scorecards, and resolve a dataset to its files api.md
At-rest ingestion The premium tier: Parquet to ClickHouse with lineage and exactly-once ingestion.md

What a tenant gets for their own streams

Question Answered by
Which streams do I own, and are they fresh? Catalog
Did every expected bucket arrive? Completeness
Is the stored data valid? Quality
What broke, and has it recovered? Alerts
Where are the files behind a dataset? API — datasets
Can I query my persisted streams at rest? At-rest ingestion