Skip to content

ADR-0036: Alert delivery

Context

ADR-0034 decides what raises an alert and when: a rule over aggregates, with a kind and a cooldown. It says nothing about how the notice reaches a person. The first alerts now exist (Junior: watchdog, stage 6), and they need a sender.

Three constraints shape the answer.

Constraint Why
Never from a client's mailbox A client's mailbox is theirs. Sending from it puts our operational mail in their Sent folder and needs a send permission the Junior must never hold
One credential must not reach every mailbox Graph Mail.Send as an application permission, granted in Entra, lets the app send as anyone in the tenant. A leaked secret would then impersonate any Octopodia person
Many hosts, more over time Every tenant host runs its own alerts (ADR-0030). Whatever holds the send credential is repeated per host until something central sends for them

Decision

flowchart LR
    subgraph Now["Now: each host sends directly"]
        W1["Tenant host<br/>watchdog"] -->|"client credentials<br/>secret for this host"| G["Microsoft Graph<br/>sendMail"]
        W2["Another host<br/>watchdog"] -->|"its own secret"| G
    end
    subgraph Later["Later: a central notifier"]
        T1["Tenant host"] -->|"alert event<br/>no mail secret"| N["Notifier<br/>control plane"]
        T2["Another host"] --> N
        N --> G2["Microsoft Graph<br/>sendMail"]
    end
    G --> A["alerts@octopodia.com.au"]
    G2 --> A

1. One sender identity: alerts@octopodia.com.au

A shared mailbox in Octopodia's Microsoft 365 tenant. No licence, no interactive sign-in. Every ABOS alert, from any product or host, comes from this address. Recipients learn one sender, can filter on it, and a reply lands somewhere we read.

2. One Entra app, restricted to that one mailbox

App registration ABOS alerts, client credentials flow, Graph Mail.Send granted through Exchange RBAC for Applications with a management scope covering only alerts@. The permission is not granted in Entra as well: Entra grants are tenant-wide and add to the RBAC grant, so granting it in both places undoes the restriction.

A leaked secret can send as alerts@ and nothing else. It cannot read any mailbox.

3. One secret per host, in that host's Ansible vault, rotated
Where it lives The host's host_vars vault file (ADR-0027, machine secret), as vault_abos_alerts_client_secret
One per host Each host gets its own client secret on the same app, named after the host. Revoking one host's secret leaves the others sending
Where it is used An n8n OAuth2 API credential on that host (grant type Client Credentials), Allowed HTTP Request Domains: graph.microsoft.com only
Rotation Every 12 months at most, and at once if a host is suspected compromised. The procedure is in the runbook
Human copy None needed. A lost secret is replaced, not recovered

4. The destination list is config, not code

Who receives alerts is a setting (alert_recipients), never written into a workflow. Today it lives in each n8n instance's settings table. It moves to the registry with the other settings (OI-095).

5. Later, tenant hosts post alert events to a central notifier

A notifier in the control plane receives alert events (rule, kind, mailbox or tenant reference, value, threshold, times, never content) and does the sending. Tenant hosts then hold no mail credential at all, and the per-host secrets are revoked.

This is needed anyway: ADR-0034's global scope (a condition visible only across tenants) cannot be evaluated on any one tenant host, so something central must see every tenant's alert counts. One component serves both.

Until it exists, each host sends directly with its own secret (§3). Tracked as OI-104.

Consequences

Alerts come from one recognisable address, and the credential that sends them cannot be turned against any other mailbox.

One secret per host means a rotation touches every host, and a forgotten host stops alerting when its secret expires: an alert path that fails silently. Expiry is the main operational risk of this decision until the notifier removes per-host secrets (OI-105).

Holding the send credential on tenant hosts is an accepted interim cost. The blast radius is bounded by §2: the worst case is spoofed alerts from alerts@, not access to anyone's mail.

The notifier sits in the control plane, which holds the map and never client content (ADR-0006). Alert events are counts and references, so this holds.

Alternatives considered

  • Send from the client's mailbox is rejected: it is theirs, and the Junior holds no send permission by design.
  • SMTP with a mailbox password is rejected: basic SMTP auth is being retired in Microsoft 365, a password is a human credential, and it cannot be scoped below the whole mailbox's sign-in.
  • Mail.Send granted in Entra alone is rejected: tenant-wide, so one leaked secret sends as any person in Octopodia.
  • Application access policies (the older way to restrict an app to some mailboxes) are rejected in favour of RBAC for Applications, which replaces them.
  • A third-party mail service (SendGrid, Postmark) is deferred: another processor, another account and DNS records to own, for volumes measured in a few emails a week.
  • One secret shared by every host is rejected: one leak means rotating everywhere, and revoking it silences every host at once.
  • Building the central notifier first is rejected for now: the first alerts need a sender today, and the direct path is small and is removed cleanly when the notifier arrives.