Catalog¶
Governance tag: overwatch · Anchor: #catalog
The catalog is the inventory of a tenant's own streams: what each stream was declared to produce, reconciled against what actually landed, with a headline freshness signal. It is the only dimension that catches a stream producing zero rows — completeness scoring is blind to a stream with no buckets to check.
Two layers reconciled¶
| Layer | Answers | Keyed on |
|---|---|---|
| Declared | What the stream is meant to produce | (exchange, market_type, symbol, datatype) |
| Observed | What actually landed | (exchange, market_type, symbol, datatype) |
| Reconciliation | Declared LEFT JOIN observed → presence |
same tuple |
The reconciliation is declared-driven: it shows exactly the streams a tenant
declared, not whatever else happened to land. A stream declared but never
observed still appears — flagged absent — which is how a dead producer is
caught.
Declared row¶
| Field | Type | Meaning |
|---|---|---|
exchange · market_type · symbol · datatype |
string | The stream identity |
expected_cadence_ms |
u32 | Grid quantum for a state stream; 0 = event-based |
kind |
string | state or event |
is_active |
u8 | 1 = live in the reconciliation |
binding_id |
string | The deploy that owns the stream |
deleted |
u8 | Tombstone: 1 = producer released |
updated_at |
u64 | Unix ms; latest declaration wins |
Latest declaration per stream wins (ReplacingMergeTree on updated_at); a
released producer tombstones its row (deleted = 1) and drops out of the view.
binding_id and deleted are internal reconciliation fields — they shape
which rows the API returns but are not included in
/api/collectors/live responses.
Observed row¶
| Field | Type | Meaning |
|---|---|---|
first_seen_ms · last_seen_ms |
u64 | Earliest / latest event timestamp |
row_count |
u64 | Rows present for the stream |
measured_cadence_ms |
u32 | Median inter-row delta (state streams) |
observed_at |
u64 | Unix ms of this refresh |
Presence signal¶
Reconciliation derives one headline presence per stream:
presence |
Condition |
|---|---|
covered |
Declared capability with no collector launched yet (seed rows carry binding_id = '') |
absent |
row_count = 0, no observed row, or tombstoned (deleted = 1) |
stale |
observed_at - last_seen_ms > 60000 (over 60s since last event) |
present |
fresh and non-empty |
flowchart TD
D[Declared stream] --> B{Launched?}
B -- capability only --> CO[covered]
B -- launched --> J{Observed?}
J -- no rows --> AB[absent]
J -- rows, last event > 60s ago --> ST[stale]
J -- rows, recent --> PR[present]
Read it¶
The Governance API returns one reconciled row per
declared stream, filterable by any of exchange, market_type, symbol,
datatype, presence.
Cross-links¶
- Completeness — scores the streams the catalog lists as present.
- Quality — validates the records behind each stream.
- API — catalog — the HTTP surface.