Console phase 4: Provision, the plan

Status: plan, 15 October 2026. No code yet. Phase 4 moves ahead of phases 2 and 3 (El, 15 October 2026). Its acceptance test is tenant zero on a new production host: a new tenant, then a fresh Gmail mailbox, then the Junior sorting, all started from the Console. Phases 2 and 3 follow, apart from the minimum of phase 2 that phase 4 needs to keep the ADRs' guarantees (below).

Each step ends in something El can test on lite-dev before the next starts, as in Console first light. Only the last step touches the production host.

1. The minimum phase 4 takes from phase 2

ADR-0037 §5 asks for two things before the Console writes anything: one schema, validated on every write, and an append-only history of every change, action and job. Phase 4 is the first phase that writes, so it brings exactly these, and no more:

Phase 4 builds now	Because
The History record: append-only, every job request, approval, run and result, and every registry record a job creates (tenant, mailbox, link token issued, never its value). Who, when, before and after, main path or break-glass. The existing History page (masked views) grows to show it	ADR-0037 §5, ADR-0039 §4. Closes OI-063 for jobs and records; settings history comes with phase 2
Job definitions with an input schema, one per job in the repository, validated by the Console before approval and again by the runner, passed to Ansible as a variables file	ADR-0039 §2
Every registry write validated: the tenant and mailbox records a job creates go through the same schema check as the job inputs	ADR-0037 §5
The settings schema, defined but written by nobody: names, types, ranges and the defaults a new mailbox starts with, in one file. The playbook writes those defaults into the new tenant's n8n, checked against the schema; the Console has no settings write path yet	Lets add mailbox start a mailbox with reviewed defaults without a settings write. Phase 2 then adds the write path and the pull to the same schema (OI-062)

The wizards take no settings. Create tenant asks for a slug, a display name and a region; add mailbox asks for a tenant and an address. A new mailbox starts with the repository's defaults. Changing a setting stays a hand edit in the tenant's n8n, as today, until phase 2. That is what keeps phase 4 from writing a setting before validation and history exist for settings.

Waits for phase 2: editable settings on Tenant and Mailbox, the settings pull into the local copy and settings_version (OI-095), alert responses (OI-121), and the Alerts screen's editing. Waits for phase 3: tenant jobs through the link (reset learning, run setup as a tenant job).

2. Before the first provision

The first provision is step 7 below, create tenant on lite-dev. Items marked before production can wait until step 10, but nothing can be created on the production host without them.

Item	What has to be true	When
Inventory and secrets written by a job (OI-126)	Create tenant makes an inventory entry, a vault ID and a vault file. ADR-0024 §4 keeps the inventory static in git, and ADR-0039 §7 runs the runner from a clean main, so the runner cannot write either. Today hosts.yml is not in git at all. Decided: a separate private inventory repository the runner commits to, ADR-0042, accepted 15 October 2026	Done
OI-124: per-host and per-tenant secrets	group_vars/all holds only what every host shares. Host secrets go in host_vars/<host>/vault.yml; tenant secrets (n8n encryption key, link and detail tokens, the AI key) go in tenants/<slug>/vault.yml under the tenant's own vault ID (ADR-0024, ADR-0027 §3), because several tenants share a host. lite-dev's secrets moved and the trap 15 check passing	Step 3
OI-108: vault password custody	The runner creates a new tenant's vault password. Decided: shown once to the approver after the passkey step-up, who stores it in Bitwarden; ADR-0042 §5 keeps it out of the Console in plain text. The concentration stays on OI-108, accepted for v1	Before step 7
OI-086: tailnet policy	Least-privilege grants replace allow all: admins to tag:server on 52843 and 443; the runner's machine to tag:server on 52843; the Console's host to each tenant host on 443; nothing else between servers. Tested by the policy's own tests block; drafted in infra/tailscale/policy.hujson. lite-dev's share into El's personal tailnet removed. Lachlan does not join the tailnet: his break-glass is the Vultr web console's root login, which hardening now refuses to lose	Grants before step 4; the rest before production
Break-glass listener (OI-113)	Not needed for phase 4. ADR-0040 §6 keeps it off on any host with client data until OI-086 is closed, and the production host runs without it (SSH over the tailnet stays the way in when Cloudflare is down). Build it after the acceptance test	After phase 4
OI-100: count emails, not moves	Built and live on lite-dev before tenant zero's production month starts, Console first, then the link (the Console must accept a field before the link sends it)	Before production
Where the production Console lives (OI-127, new)	ADR-0014 puts the core stack and tenants on one host to start. Decided: the production host carries the registry, the Console and cloudflared, built by hand with Ansible, takes console.octopodia.com.au, and creates tenant zero from its own Console. lite-dev's becomes the dev Console, console-dev.octopodia.com.au and link-dev.octopodia.com.au (approved 15 October 2026). Still open: migrate lite-dev's registry history or start fresh	Before production
OI-110: company Cloudflare account	The zone, the tunnel and the Access applications move off El's personal account before production traffic depends on them	Before production
Two tenants on one host (OI-128, new)	The n8n role runs one n8n per host. ADR-0024 §1 makes a tenant an inventory host whose ansible_host is the box, so a second tenant on lite-dev (and later on the production host) needs its own compose project, data folder, port and tailnet HTTPS port	Step 5
OI-114: publishing workflows from the repository	Proved on lite-dev: a workflow imported through n8n's API keeps its credentials and sub-workflow links, and can be published	Step 6
OI-112: secrets in job output	no_log on every task touching a secret, a redaction pass in the runner, and a test job with planted secrets	Step 4
Gmail sign-in for a new instance (OI-129, new)	Each n8n instance has its own OAuth redirect URL, so each new tenant needs its redirect URI added to the Google OAuth client, and the app must be published before the sign-in (a sign-in made in Testing expires after 7 days)	Step 8, and again for production
A paid AI key for real mail	The Gemini free tier may use what it is sent, so it must never see a real inbox (n8n runbook). Tenant zero in production sorts real email: it needs a paid-tier key in its vault	Before production
Backups (OI-036, OI-054)	The production host holds real email decisions and n8n credentials. An AU object storage bucket, the age keypair (private key in Bitwarden only), the backup role and a restore test, before tenant zero's data matters (provisioning build order, step 8)	Before production
The production host's alert secret (OI-105)	Its own client secret on the ABOS alerts app, in its own host_vars vault file	Before production
Who creates the tenant record	ADR-0006 says Ansible creates it at provisioning; the adding a tenant runbook and ADR-0024 §6 have the Console create it first and the playbook check it (the pre-flight gate). Proposal: the Console writes it from the validated inputs when the job is approved, recorded in History, and the playbook fails closed without it. Settle it in ADR-0042	Before step 7

3. The steps

Each step's Test is what El checks on lite-dev before the next step starts. The tests themselves, as each step is built, are in Console phase 4: testing each step. Step 1 built and tested 15 October 2026; step 2 built 15 October 2026 and tested 16 October 2026; step 3 built 16 October 2026 (the inventory repository runbook is its test).

Step	Builds	Test on lite-dev
0. Decide	ADR-0042 (inventory, tenant secrets, who writes the tenant record), OI-108's custody step, OI-127's production placement, OI-086's policy text. Documents only	The ADR and the policy read right to El. Nothing deployed
1. History and job requests	The History record (append-only, the app role may only insert). Job definitions in the repository with their input schemas; the three v1 jobs and one harmless check host job (read-only facts). The Provisioning screen: the menu, forms built from the definitions, Request. Nothing is approved or run	A request with a bad slug or an unknown region is refused, in the form and through the API. A good request is queued and shows in History with who and when. The app role cannot update or delete a History row
2. Passkeys and approval	Passkey enrolment (two per operator), the step-up at Approve bound to the job's ID and a hash of its inputs, the replacement flow (the other operator's step-up), and the passkey recovery command (ADR-0040 §4). Still nothing runs	Both operators enrol two passkeys. An approval without a step-up, with a stale one, or with one signed for another job is refused. Changing a job's inputs after the step-up is refused. A lost passkey is replaced only with the other operator's step-up
3. Inventory and secrets layout	ADR-0042 built: lite-dev's secrets moved (OI-124), the tenant layout (tenants/<slug>/) with per-tenant vault IDs, and wherever ADR-0042 puts the inventory. Hand runs only	The console and n8n playbooks run on lite-dev by hand with nothing changed. Trap 15's check passes: a second host in the inventory sees none of lite-dev's secrets
4. The runner	A runner service on octo-ctl as its own user, with its own SSH key, vault files and Console token, behind an Access service token on the link hostname (ADR-0040 §5). It claims approved jobs through app.turns.claim, in one transaction with the check that the job is still approved: a claim and a cancel insert the same unique job_turns row, so they can never both win (built in step 2). It runs them from a clean main at a recorded commit, uploads redacted output (OI-112), and honours the pause toggle. First job: check host	Approve check host: the runner runs it, the Console shows the output, the commit and the result, and History records all of it. With the runner paused it claims nothing; with its service stopped nothing runs. A test job with planted secrets shows none of them in its stored output
5. A second tenant on one host	The n8n role per tenant (OI-128) and the registry pre-flight gate (ADR-0024 §6). A throwaway tenant, sandbox, stood up by hand with Ansible beside lite-dev's own	sandbox's n8n answers on its own tailnet HTTPS port. lite-dev's own n8n, its workflows and its drafts are untouched. The playbook fails closed for a slug the registry does not hold
6. Redeploy tenant	The redeploy tenant job: imports every Junior workflow and the tenant link from main into a tenant's n8n, binds credentials by name, publishes, and ends with the sorter fixtures. Proves OI-114. Run only on sandbox until trusted: on lite-dev's own tenant it would replace the live workflows	From the Console, redeploy sandbox: every workflow is published, each draft matches its published version, the fixtures pass, and History shows the commit
7. Create tenant	The create tenant job: the Console writes the record (per ADR-0042), the runner makes the vault ID and tenant secrets, writes the inventory, issues the link token (OI-109) and the detail token, runs the tenant playbook, then redeploy tenant. The new vault password goes to the runner's own file, and the Console shows a custody task: store it in Bitwarden, done on octo-ctl by a person, ticked off in the Console and recorded. The password itself never reaches the registry (OI-108)	Create sandbox2 from the Console. Its n8n comes up, its link pushes within 5 minutes and it shows on Fleet as Not live (no mailbox yet) with its link reporting. The custody task stays open until ticked
8. Add mailbox	The add mailbox job, safe to run twice (OI-111): the first run makes the mailbox workflow from the repository's template, the credential shell and the error workflow setting, and stops at waiting for Google sign-in, showing the link to the credential in the tenant's n8n (tailnet). The link reports when the credential works; the second run (or a resume) runs setup, publishes and starts sorting	Add a fresh test Gmail to sandbox2. Sign in to Google when asked. Run it again: setup runs, the Junior sorts, Mailbox health shows it, and the first correction appears in the counts
9. Alerts and OI-100 on a provisioned tenant	The watchdog and the alert credential arrive with redeploy tenant; OI-100 (emails, not moves) built and live, Console first, then the link	A forced alert on sandbox2 emails from alerts@. Dragging one email back and forth counts as one correction, and one dragged back where it was sorted counts as none
10. Production: tenant zero	Only after the before-step-10 checklist (section 4). By hand, as break-glass: the production host in Vultr Melbourne, hardened through all three phases, on the tailnet, with backups and a restore test, and the core stack (registry, Console, cloudflared on the company Cloudflare account). A second runner identity for production. Then, from the production Console only: create tenant zero, add its fresh Gmail mailbox, the Junior sorting	The acceptance test. Both operators sign in with MFA and enrol passkeys on the production Console. Tenant zero, its mailbox and the Junior's sorting are all started from the Console, the one human step being the Google sign-in. History shows every job with its approver, step-up and commit

Two runner identities. The runner that tests on lite-dev and the one that provisions production are separate users on octo-ctl, each with its own SSH key, vault files and Console token, so nothing tested against lite-dev can reach the production host. This fits ADR-0039 §5 (each runner its own revocable identity).

Creating the host stays manual in v1. The production VPS is made in the Vultr portal and hardened by hand, because hardening needs both operators to prove their logins before public SSH closes and the key expiry to be turned off in the Tailscale admin console. No Vultr API token is needed for phase 4. A create host job would be a new kind of job and an ADR change to ADR-0039's menu.

4. What needs El (and Lachlan), and when

When	Who	What
Step 0	El	Done 15 October 2026: ADR-0042 (accepted), custody shown once to the approver, the production and dev Consoles, the tailnet policy (drafted, to review)
Step 2	El	Enrol your YubiKey and a device passkey on lite-dev's Console. Until Lachlan enrols, a lost passkey of yours is replaced through the passkey recovery runbook, since there is no second operator to approve it
Step 3	El	New vault passwords for the layout, into Bitwarden; lite-dev's secrets moved by the runbook
Step 4	El	The runner's Access service token in Cloudflare; the tailnet grant for octo-ctl; one hand run of the hardening playbook to add the runner's SSH key to lite-dev
Step 5	El	Possibly resize lite-dev in the Vultr portal: two n8n instances, the registry and the Console on a 4 GB host is tight. I'll size it in step 5 first
Step 7	El	Store sandbox2's vault password in Bitwarden (the custody task)
Step 8	El	A fresh test Gmail account; its redirect URI added to the Google OAuth client; the Google sign-in when the job asks
Before production	El	Move Cloudflare to the company account (OI-110). The production host's alert secret on ABOS alerts in Entra (OI-105). A paid AI key. The backup bucket, and the age keypair with Lachlan (OI-036, OI-054). Decide what happens to lite-dev's Console
Before step 10	Lachlan	Checklist, must be done: his own YubiKey and a device passkey enrolled on the Console; his own Vultr login with MFA and console access; a root login on lite-dev's Vultr web console tested with the password from Bitwarden. He is already a Bitwarden Owner and does not join the tailnet (OI-086)
Step 10	El, Lachlan	Create the VPS in the Vultr portal (root password into Bitwarden); El logs in through hardening phases 1 and 2 and Lachlan proves his web console login before phase 3; turn off its Tailscale key expiry; enrol passkeys on the production Console; the fresh Gmail account for tenant zero, its redirect URI, and the Google sign-in

Not needed for phase 4: a Vultr API token (host creation stays manual), Entra changes beyond the alert secret (sign-in to the Console is already Entra through Access), and Entra P1 (OI-117 matters only if the passkey step-up cannot be built).

5. Does the new order break ADR-0037 to ADR-0040?

No, as long as phase 4 keeps the first section above.

• ADR-0037 §5 (validation and history before any write): phase 4 brings both for everything it writes (job inputs, job records, tenant and mailbox records) and writes no settings. The settings write path comes later with its schema and history already in place. Kept.
• ADR-0038 (the tenant link pulls settings and jobs): phase 4 needs only the link's push and a link token issued at create tenant; settings stay in the tenant's n8n, edited by hand, until phase 2's pull. Nothing in ADR-0038 assumes the pull exists before provisioning. Kept.
• ADR-0039 (runner, fixed menu, nothing destructive, from main): kept, with one finding that is not about the order. Create tenant has to write an inventory entry and a vault file, while ADR-0039 §7 runs from a clean main and ADR-0024 §4 keeps the inventory in git. That needs ADR-0042 before step 3 (OI-126). Add mailbox runs twice around the Google sign-in, which is still one menu entry.
• ADR-0040 (passkey step-up before the first approval; break-glass after OI-086): the step-up lands in step 2, before the runner exists in step 4, so no job can ever run unapproved. Break-glass stays off until OI-086 and OI-113 are done, which ADR-0040 §6 already allows. Kept.
• ADR-0041 §7 says the masked view ships in phase 2. It is already built and live, so the order does not affect it, and the ADR stays as written.

Outside ADR-0037 to 0040, two existing decisions need settling in ADR-0042, not because of the order but because phase 4 is the first to exercise them: ADR-0024 §4 (static inventory in git, today not true of hosts.yml) and ADR-0006's "Ansible creates the tenant record" against the runbook's "the Console creates it and the playbook checks it".