Skip to content

Governance API

Governance tag: overwatch · Min version: overwatch-serve 0.0.1 · Anchor: #api

overwatch-serve is the read surface over a tenant's governance state and their Parquet lake. Responses are JSON. Every query value is bound as a typed parameter; dataset routes take an opaque dataset_id, never a raw path.

Method Path Returns
GET /health Liveness (ok)
GET /api/collectors/catalog Deployable collector options
GET /api/collectors/live Reconciled catalog rows
GET /api/streams/{exchange}/{market_type}/{symbol}/{datatype}/scorecard Completeness + quality for one stream
GET /api/collectors/datasets File-level dataset catalog
GET /api/collectors/datasets/{dataset_id}/resolve Files behind a dataset id

GET /api/collectors/catalog

The deployable option universe: every supported venue with its native symbol format, market types, orderbook-depth ladder and per-datatype deployability. Static capability data — what a caller CAN deploy, independent of what is currently collected.

Response — venues[]:

Field Type Meaning
exchange string Venue id as accepted by deploy requests
label string Human-readable venue name
symbol_format · symbol_example string Native wire symbol form
market_types[] string Market types deployable today
orderbook_depths[] · default_orderbook_depth u32 Valid depth choices
datatypes[] array name · available · pending_on (when gated) 
GET /api/collectors/catalog

{
  "venues": [
    {
      "exchange": "binance", "label": "Binance",
      "symbol_format": "concatenated uppercase", "symbol_example": "BTCUSDT",
      "market_types": ["spot"],
      "orderbook_depths": [5, 10, 20, 50, 100, 500, 1000],
      "default_orderbook_depth": 50,
      "datatypes": [
        { "name": "orderbook", "available": true },
        { "name": "trades", "available": true },
        { "name": "funding", "available": false,
          "pending_on": "perpetual endpoint routing" }
      ]
    }
  ]
}

GET /api/collectors/live

Every declared stream reconciled with observed reality. See Catalog.

Query parameters (all optional, exact-match filters):

Param Meaning
exchange Filter by exchange
market_type Filter by spot / perp
symbol Filter by symbol
datatype Filter by datatype
presence Filter by present / absent / stale

Response — array of rows:

Field Type Meaning
exchange · market_type · symbol · datatype string Stream identity
kind string state · event
expected_cadence_ms u32 Grid quantum; 0 = event-based
is_active u8 Active in reconciliation
row_count u64 Observed rows
first_seen_ms · last_seen_ms u64 Earliest / latest event ts
measured_cadence_ms u32 Median inter-row delta
observed_at u64 Refresh time
presence string present · absent · stale 
GET /api/collectors/live?exchange=binance&presence=present

[
  {
    "exchange": "binance", "market_type": "perp",
    "symbol": "BTC-USDT", "datatype": "orderbook",
    "kind": "state", "expected_cadence_ms": 250, "is_active": 1,
    "row_count": 143220, "first_seen_ms": 1750000000000,
    "last_seen_ms": 1750003600000, "measured_cadence_ms": 250,
    "observed_at": 1750003605000, "presence": "present"
  }
]

GET /api/streams/{exchange}/{market_type}/{symbol}/{datatype}/scorecard

Completeness plus quality for one stream, over its latest run, a pinned run, or a live trailing window. See Completeness and Quality.

Query parameters:

Param Meaning
run_id Pin the scorecard to a specific completeness run (default: latest)
window_secs Recompute completeness live over a trailing window of this many seconds

run_id and window_secs are alternatives: with window_secs, completeness is recomputed live and window.live is true; without it, the latest (or pinned) stored run is read.

Response:

Field Type Meaning
stream object exchange · market_type · symbol · datatype
window.run_id string The completeness run behind this scorecard
window.start_ms · window.end_ms u64 Window bounds
window.run_at u64 Run time (stored-run reads)
window.kind string state · event (stored-run reads)
window.window_secs · window.live u64 · bool Live-window reads
completeness.status string ok · degraded · critical · absent
completeness.completeness_pct f64 0.0–1.0; -1 = N/A (event stream)
completeness.expected_buckets · observed_buckets · missing_buckets u64 Bucket counts
completeness.max_gap_ms u64 Longest contiguous silence
completeness.rows_in_window u64 Raw event count
quality.exception_count int Checks that fired
quality.max_severity string critical · warn · none
quality.checks[] array check_name · severity · observed · detail · sample_ts_ms
gaps[] array gap_start_ms · gap_end_ms · gap_ms · missing_buckets 
GET /api/streams/binance/perp/BTC-USDT/orderbook/scorecard?window_secs=3600

{
  "stream": {
    "exchange": "binance", "market_type": "perp",
    "symbol": "BTC-USDT", "datatype": "orderbook"
  },
  "window": {
    "run_id": "run_20260623_1200", "start_ms": 1750000000000,
    "end_ms": 1750003600000, "window_secs": 3600, "live": true
  },
  "completeness": {
    "status": "ok", "completeness_pct": 0.998,
    "expected_buckets": 14400, "observed_buckets": 14371,
    "missing_buckets": 29, "max_gap_ms": 1500, "rows_in_window": 14371
  },
  "quality": {
    "exception_count": 1,
    "max_severity": "warn",
    "checks": [
      { "check_name": "empty_side", "severity": "warn",
        "observed": 12, "detail": "snapshot with an empty book side",
        "sample_ts_ms": 1750002200000 }
    ]
  },
  "gaps": [
    { "gap_start_ms": 1750001000000, "gap_end_ms": 1750001001500,
      "gap_ms": 1500, "missing_buckets": 6 }
  ]
}

No run yet

A stream with no completeness run returns 404 for stored-run reads. A live read (window_secs) computes fresh over the window regardless.

GET /api/collectors/datasets

The file-level catalog over the Parquet lake, one entry per (venue, symbol, data_type), richest (most files) first.

Response — array of entries:

Field Type Meaning
dataset_id string Opaque, traversal-safe id (see below)
name string venue · symbol · data_type label
venue · symbol · data_type string Dataset identity
file_count int Parquet files in the group
size_bytes u64 Total bytes
mtime string | null Latest file modification, RFC 3339 
GET /api/collectors/datasets

[
  {
    "dataset_id": "3f9a1c22e0b47d51",
    "name": "binance · BTC-USDT · orderbook",
    "venue": "binance", "symbol": "BTC-USDT", "data_type": "orderbook",
    "file_count": 362, "size_bytes": 3985210,
    "mtime": "2026-06-23T12:00:00+00:00"
  }
]

The dataset_id is the first 8 bytes of sha256("<venue>/<symbol>/<data_type>"), hex-encoded — deterministic, so resolution is a re-scan-and-match and the same tuple hashes to the same id platform-wide.

GET /api/collectors/datasets/{dataset_id}/resolve

Resolve an opaque dataset_id to its concrete, root-relative files.

Response:

Field Type Meaning
dataset_id string The requested id
venue · symbol · data_type string Dataset identity
file_count int Files behind the id
files[] string Root-relative Parquet paths 
GET /api/collectors/datasets/3f9a1c22e0b47d51/resolve

{
  "dataset_id": "3f9a1c22e0b47d51",
  "venue": "binance", "symbol": "BTC-USDT", "data_type": "orderbook",
  "file_count": 2,
  "files": [
    "collected/binance/orderbooks/binance_BTC-USDT_ob_sync_20260623_120000.000.parquet",
    "collected/binance/orderbooks/binance_BTC-USDT_ob_sync_20260623_120500.000.parquet"
  ]
}

Ids in, never paths

A client only ever sends a dataset_id. Resolution re-scans the root, recomputes ids, and matches; a raw filename or .. never reaches the filesystem, and every resolved path is asserted to stay within the datasets root. A malformed id returns 400; an unknown id returns 404.