Skip to content

Completeness

Governance tag: overwatch · Min version: overwatch-serve 0.0.1 · Anchor: #completeness

Completeness measures how much of a stream's expected output actually landed over a trailing window. State streams score against a fixed cadence (bucket math); event streams have no fixed expectation and score on silence and presence.

Two scoring modes

kind Datatypes Signal completeness_pct
state orderbook, funding, open_interest Observed buckets ÷ expected buckets 0.0–1.0
event trades, liquidations max_gap_ms vs a per-symbol baseline + presence -1 sentinel (N/A)

State scoring (bucket math)

A window is diced into fixed-cadence buckets. Each bucket is either observed or missing; the score is the observed fraction.

Quantity Definition
expected_buckets (window_end_ms - window_start_ms) / expected_cadence_ms
observed_buckets Distinct buckets that carried at least one event
completeness_pct min(1.0, observed_buckets / expected_buckets)
missing_buckets expected_buckets - observed_buckets
max_gap_ms Longest contiguous run of missing buckets, in ms
rows_in_window Raw event count in the window 
flowchart LR
  W[Trailing window] --> B[Dice into cadence buckets]
  B --> O[Mark buckets observed or missing]
  O --> P[completeness_pct = observed / expected]
  O --> G[max_gap_ms = longest missing run]
A state stream's window is diced into fixed-cadence buckets; the completeness percentage is the observed fraction and the max gap is the longest contiguous missing run.

Status thresholds

State streams resolve to a status from their observed fraction:

status Condition
ok observed / expected >= 0.999
degraded observed / expected >= 0.95
critical below 0.95 with some observed
absent zero observed buckets

Event streams resolve to absent when no rows landed in the window, otherwise ok; their headline signal is max_gap_ms against a per-symbol baseline.

Gaps

Each contiguous missing run is recorded as a gap, so a scorecard can point at where a stream fell silent:

Field Type Meaning
gap_start_ms u64 First missing bucket start
gap_end_ms u64 Last missing bucket end
gap_ms u64 Gap duration
missing_buckets u64 Buckets in the run

Runs and windows

A run is one audit invocation over a fixed trailing window; each run stores a scorecard row per stream plus its gaps. A scorecard read can also recompute completeness live over a caller-chosen trailing window rather than reading a stored run — see the API's window_secs parameter.

Read it

The Governance API returns the completeness block — status, percentage, bucket counts, max gap, and the gap list — for one stream, either from the latest (or pinned) run or recomputed live.

  • Catalog — catches the zero-row case bucket math cannot see.
  • Quality — validates the records inside the observed buckets.
  • API — scorecard — the HTTP surface.