Skip to content

Alerts

Governance tag: overwatch · Anchor: #alerts

An alert is one (stream, check) condition tracked through a firing-to-resolved lifecycle. Each condition is keyed on a stable alert_id, so it transitions in place — that is the debounce: one row per breach, not one per audit run.

What fires today

The alert dimension currently covers network placement: the misplaced check compares each collector's actual latency (WebSocket RTT, falling back to feed lag) against the best achievable latency for its exchange from the region benchmark. It fires at warn severity when the gap exceeds the threshold (default 40 ms) and the collector is not already in the ideal region.

Completeness and quality breaches surface in the per-stream scorecards — they do not create alert rows yet.

Lifecycle

stateDiagram-v2
  [*] --> firing: check breaches
  firing --> resolved: condition clears
  resolved --> firing: condition re-breaches
An alert condition moves from firing to resolved when it clears and back to firing if it re-breaches, all under one stable alert id.
State Meaning
firing The condition is currently breached (resolved_at = 0)
resolved The condition cleared (resolved_at set)

The latest state per condition wins (ReplacingMergeTree on updated_at), so a recovery updates the existing row rather than adding a new one.

Delivery

Firing alerts surface in the Governance panel, which reads state = firing. The channel and notified fields are reserved schema for a push notifier (Telegram / webhook / digest); no push delivery ships in the current deployment — alerts are pull-only.

Alert record

Field Type Meaning
alert_id string Stable hash of (stream + check_name)
exchange · market_type · symbol · datatype string The stream
check_name string The originating quality/completeness check
severity string critical · warn
state string firing · resolved
value f64 Offending value/count at fire time
detail string Human note
fired_at u64 Unix ms first observed firing
resolved_at u64 Unix ms cleared (0 = still firing)
channel string Reserved (telegram · webhook · digest); empty today
notified u8 Reserved for push delivery; 0 today
updated_at u64 Unix ms; latest state wins
  • Quality — critical/warn exceptions, surfaced in scorecards.
  • Completeness — coverage scores, surfaced in scorecards.
  • Catalog — the stream identity an alert keys on.