ADR-0034: Alerting and notification model¶
Context¶
Several decisions have produced things that need to notify someone, and each was designed in isolation: a parse-failure spike, a rule contradiction, an abandoned work item, a backup that did not run. Left as one-off notifications, they arrive on the same surface with the same weight, and the ones that matter get lost among the ones that do not.
Two problems in particular forced the question.
An abandoned item is not a fault. A client ignoring their own approval queue is a commercial signal, not a bug, and alerting ourselves about it in the same channel as a broken deploy means it gets triaged like a bug and dismissed.
Some conditions are only visible across tenants. One tenant's parse failures could be their prompt. Every tenant's at once is a provider-side model change, and nothing looking tenant by tenant would ever see it.
Decision¶
1. An alert is a rule over aggregates¶
Same {metric, operator, value} shape as ADR-0004, evaluated on a schedule rather than per work item.
| Rules (ADR-0004) | Alerts | |
|---|---|---|
| Act on | One work item | A metric over a window |
| Evaluated | At decision time | On a schedule |
| Effect | Set a field, narrow actions | Notify |
Reusing the predicate shape means there is one config language rather than two, and it makes count-versus-percentage a choice of metric (abandoned_count or abandoned_rate) rather than a special case.
No expression language, no eval, for the same reasons as ADR-0004.
2. An alert definition¶
| Field | |
|---|---|
metric |
From a published catalogue, as with rule facts |
operator, threshold |
|
window |
Rolling period the metric is computed over |
scope |
tenant or global |
destination |
Console, email, Teams |
kind |
incident or advisory |
cooldown |
Minimum interval before re-firing |
3. Two scopes on this mechanism, and a third kept separate¶
| Scope | Evaluated over | Who sees it | Example |
|---|---|---|---|
| Tenant | One tenant's aggregates | Us, optionally them | This client's abandonment rate |
| Global | Across all tenants | Us only | Parse failures spiking everywhere |
| Platform | Host metrics | Us only | Disk at 80%, a container restarting |
Global is what justifies the mechanism. A per-tenant-only design would never surface a provider-side model change, because no single tenant's numbers would look unusual enough.
Platform monitoring is not built on this. It reads host metrics rather than aggregates over work items, and it has no tenant dimension at all. Forcing both into one language means expressing "abandonment above 15% for this tenant" and "disk above 80% on bos1" in the same shape, which produces a language that serves neither. Consistent with uptime checking already living off-box (OI-048).
4. kind distinguishes "fix this" from "talk to someone"¶
kind |
Means | Response |
|---|---|---|
incident |
Something is broken | Fix it, today |
advisory |
Something is drifting | A conversation, not a deploy |
This must be designed in rather than added later. Every Console alert produced so far means "broken, fix today". The abandonment alert means "ring the client". On a shared surface with no distinction, the advisory is triaged as a bug and dismissed, and the churn signal it carries is lost.
The two render differently in the Console and may route differently.
5. Cooldown is mandatory, not optional¶
A condition true for a week notifies once, not on every evaluation.
Alert fatigue kills monitoring faster than missing alerts do. A system that fires repeatedly is one people mute, and a muted system catches nothing.
6. Ship a default set¶
The mechanism is configurable, and a set of alerts is defined and enabled out of the box.
A configurable system nobody configures is worse than a fixed one, because it carries the cost of the configuration surface and delivers nothing.
7. Client visibility is allowed and not the default¶
Tenant-scoped alerts may be surfaced to the client, and are not by default.
"Your approval queue is backing up" is fair to tell a client, and it is also the alert most worth raising as a conversation rather than a notification. Console-only until the alert set has run long enough to be trusted.
Consequences¶
One config language covers per-item rules and scheduled alerts, so the vocabulary someone learns for one transfers to the other.
Because kind exists from the start, the Console can separate "something is broken" from "a client relationship is drifting", and the second is not lost among the first.
Because scope includes global, conditions that are invisible tenant by tenant become detectable, which is the main reason this is a mechanism rather than a handful of hard-coded notifications.
The cost is a metric catalogue to maintain alongside the fact catalogue, and a scheduled evaluation path that did not previously exist. Both are small, and the alternative is a growing set of bespoke notifications with no shared cooldown, routing or severity.
Platform monitoring remaining separate means two monitoring surfaces rather than one. Accepted, because a single language covering both would serve neither well.
Alternatives considered¶
- Hard-coded notifications per condition is rejected: no shared cooldown, no severity, no routing, and every new condition is a code change.
- One severity, no
kindis rejected: it is precisely how a churn signal gets triaged as a bug and dismissed. - Platform metrics on the same mechanism is rejected: no tenant dimension, different data source, and a language stretched to cover both would be worse at each.
- A blank configuration surface with no defaults is rejected: the cost of configurability with none of the benefit.
- Client-visible tenant alerts by default is rejected for now, since the alert set is unproven and a noisy first month would teach a client to ignore it.