Skip to content

Alerts

Governance tag: overwatch · Min version: overwatch-serve 0.0.1 · Anchor: #alerts

An alert is one (stream, check) condition tracked through a firing-to-resolved lifecycle. Each condition is keyed on a stable alert_id, so it transitions in place — that is the debounce: one notification per breach, not one per audit run.

Lifecycle

stateDiagram-v2
  [*] --> firing: check breaches
  firing --> resolved: condition clears
  resolved --> firing: condition re-breaches
An alert condition moves from firing to resolved when it clears and back to firing if it re-breaches, all under one stable alert id.
State Meaning
firing The condition is currently breached (resolved_at = 0)
resolved The condition cleared (resolved_at set)

The latest state per condition wins (ReplacingMergeTree on updated_at), so a recovery updates the existing row rather than adding a new one.

Channels

channel Used for
telegram Critical breaches, pushed immediately
webhook Critical breaches, pushed immediately
digest Warnings, rolled into a periodic digest

A notifier reads state = firing AND notified = 0, pushes on the channel for the severity, then marks notified = 1 so each breach notifies once.

Alert record

Field Type Meaning
alert_id string Stable hash of (stream + check_name)
exchange · market_type · symbol · datatype string The stream
check_name string The originating quality/completeness check
severity string critical · warn
state string firing · resolved
value f64 Offending value/count at fire time
detail string Human note
fired_at u64 Unix ms first observed firing
resolved_at u64 Unix ms cleared (0 = still firing)
channel string telegram · webhook · digest
notified u8 Push delivered
updated_at u64 Unix ms; latest state wins
  • Quality — the critical/warn exceptions that become alerts.
  • Completeness — absent and critical scores that fire alerts.
  • Catalog — the stream identity an alert keys on.