Alert delivery¶

How ABOS alerts get from a host to a person: the alerts@octopodia.com.au mailbox, the Entra app ABOS alerts, the scope that keeps that app to one mailbox, and the per-host secret. The decision and its reasons are in ADR-0036.

The inventory moved, 16 October 2026

The live inventory is now its own private repository, octopodia-abos-inventory, cloned beside the code (ADR-0042). Paths below written as infra/inventory/... or inventory/... are now in that repository, and secrets no longer go in group_vars/all/vault.yml: a host's go in host_vars/<host>/vault.yml (vault ID core), a tenant's in tenants/<slug>/vault.yml (the tenant's own vault ID), each value added with ansible-vault encrypt_string. See the inventory repository runbook.

flowchart LR
    W["Host: n8n watchdog"] -->|"1. client id + this host's secret"| E["Entra token endpoint"]
    E -->|"2. token"| W
    W -->|"3. POST /users/alerts@/sendMail"| G["Microsoft Graph"]
    G -->|"RBAC scope: alerts@ only"| A["alerts@octopodia.com.au"]
    A --> R["alert_recipients"]

Part	Done	Where
Mailbox	Once	Microsoft 365
App and scope	Once	Entra, Exchange Online PowerShell
Secret and n8n credential	Per host	Entra, the host's vault, that host's n8n
Rotation	Every 12 months per host, or at once if a host is suspect	As above

Before this: a Microsoft 365 global admin (or Exchange admin plus Application admin) account, and PowerShell with the ExchangeOnlineManagement module.

1. The mailbox (once)¶

Microsoft 365 admin centre, Teams & groups, Shared mailboxes, Add a shared mailbox:

Field	Value
Name	ABOS alerts
Email	`alerts@octopodia.com.au`

Or in Exchange Online PowerShell:

Connect-ExchangeOnline
New-Mailbox -Shared -Name "ABOS alerts" -PrimarySmtpAddress alerts@octopodia.com.au

A shared mailbox needs no licence and nobody signs in to it. Give El Full Access so replies to alerts can be read (Members on the mailbox). Never give it a password.

2. The Entra app (once)¶

Entra admin centre, App registrations, New registration:

Field	Value
Name	ABOS alerts
Supported account types	This organisational directory only
Redirect URI	None

Then, on the app:

API permissions: remove the default User.Read. Do not add Mail.Send. The permission comes from the Exchange scope in section 3. An Entra grant is tenant-wide and adds to the scope, so the app could then send as anyone.
Note three values (not secret, but keep them with the runbook notes, not in chat):

Value	Where	Used for
Application (client) ID	App registration, Overview	The scope, every host's credential
Directory (tenant) ID	App registration, Overview	The token URL
Object ID of the enterprise application	*Enterprise applications, ABOS alerts, Overview*	`New-ServicePrincipal` only

Two different object IDs

The app registration and the enterprise application each have an Object ID. New-ServicePrincipal needs the enterprise application's. With the app registration's, the scope does not apply to the app: section 4 catches it.

3. Scope the app to `alerts@` (once): the three Exchange commands¶

Connect-ExchangeOnline

# 1. Tell Exchange about the app.
New-ServicePrincipal -AppId <client-id> -ObjectId <enterprise-app-object-id> -DisplayName "ABOS alerts"

# 2. A scope that matches one mailbox and nothing else.
New-ManagementScope -Name "ABOS alerts mailbox only" -RecipientRestrictionFilter "PrimarySmtpAddress -eq 'alerts@octopodia.com.au'"

# 3. Give the app Mail.Send, inside that scope only.
New-ManagementRoleAssignment -App <client-id> -Role "Application Mail.Send" -CustomResourceScope "ABOS alerts mailbox only"

Allow up to two hours before the grant takes effect: Exchange caches app permissions.

4. Test the scope¶

In Exchange, straight away:

Test-ServiceAuthorization -Identity <client-id> -Resource alerts@octopodia.com.au
Test-ServiceAuthorization -Identity <client-id> -Resource el@octopodia.com.au

Resource	Must show
`alerts@octopodia.com.au`	`Application Mail.Send`, `InScope: True`
`el@octopodia.com.au` (any other mailbox)	`InScope: False`

Against Graph, once the cache has caught up. Use the first host's secret from section 5. On octo-ctl, so the secret does not reach shell history:

TENANT=<tenant-id>; CLIENT=<client-id>; read -rs SECRET
TOKEN=$(curl -s https://login.microsoftonline.com/$TENANT/oauth2/v2.0/token \
  -d client_id=$CLIENT --data-urlencode client_secret="$SECRET" \
  -d scope=https://graph.microsoft.com/.default -d grant_type=client_credentials | jq -r .access_token)
send() { curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  https://graph.microsoft.com/v1.0/users/$1/sendMail -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message":{"subject":"ABOS alerts scope test","body":{"contentType":"Text","content":"Scope test."},"toRecipients":[{"emailAddress":{"address":"el@octopodia.com.au"}}]}}'; }
send alerts@octopodia.com.au   # must print 202, and the email arrives from alerts@
send el@octopodia.com.au       # must print 403
unset SECRET TOKEN; clear

A 202 on the second line means the app can send as anyone: stop. Check that Mail.Send was not granted in Entra (section 2) and that the scope filter names alerts@ exactly.

5. Each host: secret, vault, n8n credential¶

Per host, so revoking one host leaves the others sending.

Entra, ABOS alerts, Certificates & secrets, New client secret. Description <host> <YYYY-MM> (for instance lite-dev 2026-10), expiry 12 months. Copy the value now: Entra never shows it again.

Vault, on octo-ctl in ~/octopodia-abos/infra:

ansible-vault edit inventory/group_vars/all/vault.yml

vault_abos_alerts_client_secret: "<the value>"

While lite-dev is the only host, its secrets are in group_vars/all/vault.yml, which every host inherits. Before a second host joins the inventory they move to inventory/host_vars/<host>/vault.yml, and from then on each host's secret goes in its own file (known trap 15, OI-124).

The console role writes it into the Console's quiet link check (below). The n8n credential is still entered by hand (OI-105). 3. n8n on that host, Credentials, Add, OAuth2 API:

Field	Value
Name	`Graph: ABOS alerts (<host>)`
Grant Type	Client Credentials
Access Token URL	`https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token`
Client ID	The application (client) ID
Client Secret	The value from step 1
Scope	`https://graph.microsoft.com/.default`
Authentication	Send credentials in body
Allowed HTTP Request Domains	Specific domains: `graph.microsoft.com`

Select the credential on Send alert email and Send resolved email in Junior: watchdog (and on any later product's send node).
Recipients: the settings table, row alert_recipients: comma-separated addresses.
Test: force not polling on the watchdog (a last_poll_ok_at older than the overdue threshold, for instance an hour old), run it by hand, then make last_poll_ok_at fresh and run it again. Two emails arrive from alerts@, and the alerts row reads sent <time> to <recipients> in both notified_open and notified_resolve.
Diary the expiry (12 months less two weeks). Nothing reminds anyone yet (OI-105).

The Console's quiet link alert (OI-119)¶

The Console on a host emails from alerts@ when a tenant's link stops pushing for 15 minutes, and again when it is heard. It uses that host's existing secret (vault_abos_alerts_client_secret, section 5), so there is nothing new to create.

Inventory, in the host's console entry in inventory/hosts.yml, the two IDs from section 2 (not secret):

console_alerts_tenant_id: "<Directory (tenant) ID>"
console_alerts_client_id: "<Application (client) ID>"
# console_alert_recipients: "el@octopodia.com.au,lachlan@octopodia.com.au"   (default: El)

Run the console playbook. It writes the secret to the Console's alerts folder (mode 0400, mounted read-only), migrates the registry and schedules check-links every 5 minutes in root's crontab. Without the IDs the check still runs and records not configured.
Test outside a test round: unpublish Junior: tenant link for 20 minutes. One email link quiet arrives from alerts@; publish it again and one link heard again follows within 5 minutes. On the host, sudo journalctl -t console-check-links shows each send.

Every run logs one line, so the journal shows the check is alive and whether it can send:

check-links: 1 link checked, 0 quiet, alerts configured yes

Links checked counts tenants whose link has pushed at least once; quiet counts those silent for longer than the limit now. The registry records only changes (a link going quiet or being heard), in link_alert_events; its notified column says whether that email went, or not sent: alert email is not configured on this Console.

The outside watcher on `octo-ctl`¶

Everything else that alerts runs on the host it watches, so a host that is down raises nothing. infra/scripts/host_watch.py runs on octo-ctl instead: every 5 minutes it asks each host's n8n /healthz over the tailnet, and emails from alerts@ after two failed runs in a row, and once more when the host answers again.

Entra, ABOS alerts, Certificates & secrets, New client secret, description octo-ctl <YYYY-MM>, 12 months. It is octo-ctl's own secret, so revoking it leaves every host sending (ADR-0036 §3).
On octo-ctl, as the user that runs the watcher:
```
install -d -m 700 ~/.config/abos ~/.local/state/abos
read -rs SECRET
```
Paste the secret when it waits (on its own line), then:
```
printf '%s' "$SECRET" > ~/.config/abos/alerts_client_secret && chmod 600 ~/.config/abos/alerts_client_secret
unset SECRET; clear
cp ~/octopodia-abos/infra/scripts/host_watch.example.json ~/.config/abos/host_watch.json
```
Put the two IDs in ~/.config/abos/host_watch.json. recipients is a list, one address per item: ["el@octopodia.com.au", "lachlan@octopodia.com.au"], never one string "a,b". The watcher refuses to start otherwise and says why. Keep a record of the secret in the core vault, like other machine credentials (ADR-0027 §7). 3. Check the probe answers from octo-ctl: curl -s -o /dev/null -w '%{http_code}\n' https://lite-dev.tail34193f.ts.net/healthz prints 200. 4. Prove the send without an outage:
```
python3 ~/octopodia-abos/infra/scripts/host_watch.py --test-email
```
It prints test email: sent, and [BOS] Test: octo-ctl watcher can send arrives from alerts@ to every recipient. It touches no watcher state. On failure it prints Graph's error code and message (never the token or the secret); see the table at the end. 5. Cron (crontab -e):
```
*/5 * * * * /usr/bin/python3 $HOME/octopodia-abos/infra/scripts/host_watch.py >> $HOME/.local/state/abos/host_watch.log 2>&1
```
Test outside a test round: stop lite-dev's n8n for 15 minutes. Within 10 minutes one email lite-dev n8n not answering arrives from alerts@; start n8n and one answering again follows. This is also OI-119's closing test, together with the Console's.
Diary the expiry (12 months less two weeks), with the hosts' (OI-105).

octo-ctl must stay on

The watcher is only as awake as octo-ctl. If it is off or off the tailnet, nothing reports a host that is down (OI-119).

Rotating a host's secret¶

Every 12 months at most, before the old one expires, and at once if the host may be compromised.

Entra: New client secret, description <host> <YYYY-MM>, 12 months. Keep the old one for now.
Vault: replace vault_abos_alerts_client_secret in that host's vault.yml.
n8n: open Graph: ABOS alerts (<host>), paste the new secret, save.
Test: section 4's Graph test with the new secret (expect 202 from alerts@), or section 5 step 6 on the host.
Entra: delete the old secret, by its description. Until this step the old one still works.
Diary the new expiry.

Suspected compromise: do step 5 first (delete the host's secret), then steps 1 to 4. The host stops alerting for the minutes in between, and no other host is affected.

First done for lite-dev, 11 October 2026: credential Graph: ABOS alerts (lite-dev); a forced "not polling" alert was emailed on open and on resolve (Graph 202, both received from alerts@).

If something goes wrong¶

Symptom	Cause	Fix
`alerts.notified_open` reads `not sent: Credentials not found`	No credential selected on the send node	Section 5, step 4
`not sent` mentioning a domain not allowed	Allowed HTTP Request Domains misses `graph.microsoft.com`	Section 5, step 3
`401`, `invalid_client` or `AADSTS7000215`	Secret wrong or expired	Rotate (above)
`AADSTS700016`	Wrong tenant ID in the token URL, or wrong client ID	Section 2 values
`403 ErrorAccessDenied` sending as `alerts@`	Scope not applied yet (wait two hours), or `New-ServicePrincipal` given the app registration's object ID	Section 2 warning; re-run `Test-ServiceAuthorization`
`202` sending as another mailbox	`Mail.Send` granted in Entra as well	Remove it in Entra, API permissions; re-test section 4
`400 ErrorInvalidRecipients`	`alert_recipients` empty or malformed	Section 5, step 5
Watcher log: `config ...: not started: recipients must be a list ...`	`recipients` in `host_watch.json` written as one string (`"a,b"`), which sent one "recipient" per character and got a Graph `400` (October 2026)	Make it a list, one address per item; then `--test-email`