ADR-0039: Console runner and secrets custody¶
- Status: Accepted. §7 superseded in part by ADR-0042 (the runner also checks out the inventory repository, at a recorded commit)
- Date: 2026-10-11
- Deciders: El
- Supersedes: ADR-0027 in part: vault passwords may now be held unattended, by a dedicated runner user on the control machine (§5 below). ADR-0027 §5 stands: nothing is stored on a provisioned host.
- Relates to: ADR-0024, ADR-0031, ADR-0037, ADR-0040
Context¶
Provisioning is Ansible run from octo-ctl by a person who knows the playbook, the limit, the vault IDs and the order. ADR-0037 makes the Console the place routine work starts, which needs something to run playbooks on an operator's behalf.
The control machine is already where key material lives (ADR-0027 §5). What changes is who holds it: today a vault password is supplied by a person at run time; a runner must hold it without anyone present.
Decision¶
flowchart LR
OP(["Operator"]) -->|"requests, then approves<br/>passkey step-up"| CON["Console"]
CON --> Q[("Job queue<br/>in the registry")]
R["Runner<br/>octo-ctl, own user"] -->|"pulls approved jobs"| Q
R -->|"git: main, exact commit"| GIT[("Repository")]
R -->|"ansible-playbook<br/>own SSH key, own vault files"| H["Hosts, over the tailnet"]
R -->|"output, commit, result"| Q1. A runner on octo-ctl pulls approved jobs¶
The runner is a small service on the control machine. It calls out to the Console API for approved jobs; the Console never connects to it (the same direction as ADR-0038 §1). It runs each job's playbook, then reports back.
Always unlocked, with guardrails. There is no separate arming step; the guardrails below are what make that safe.
2. A fixed menu of named jobs¶
- Named jobs only. Never free text, never an arbitrary playbook, never a shell command.
- Each job is a definition in this repository: name, playbook, and an input schema. The Console builds its forms from the same definitions, so adding a job is a reviewed commit, merged to
main. - Inputs are validated against the schema (a tenant slug is lowercase alphanumeric, a region is one of the allowed values) before approval and again by the runner, and are passed to Ansible as a variables file, never interpolated into a command line.
The v1 menu:
| Job | Does |
|---|---|
| Create tenant | Tenant record, vault ID, DNS, stack, n8n, tenant link token |
| Add mailbox | A Junior mailbox for a tenant: workflow from the repository, credential shell, setup run. The Google sign-in itself stays a human step (OI-111) |
| Redeploy tenant | Re-runs the tenant's playbook from main: workflows, versions, configuration |
The menu grows by one rule (ADR-0037 §1): anything done by hand on octo-ctl more than once becomes a job.
3. Nothing destructive in v1¶
Delete tenant, rotate secrets and rebuild host stay manual, as break-glass runbooks. A destructive job needs a stronger gate than one operator's click, such as a typed confirmation and a second approver. How to add one later is OI-107.
4. Every job is recorded¶
| Recorded | |
|---|---|
| Requester and approver | From the Access identity, with the approver's passkey step-up bound to this job (ADR-0040 §4). The runner refuses a job whose approval lacks one |
| Times | Requested, approved, started, finished |
| Inputs | As validated |
| Playbook commit | The exact commit of main the runner checked out |
| Full output | Ansible's output, with secrets never printed (no_log) and a redaction pass before storage (OI-112) |
| Result | Succeeded, failed, or stopped |
The record is append-only, alongside the settings history (ADR-0037 §5).
5. The runner's own identity, and the custody change¶
| The runner has | Separate from El's and Lachlan's | Revoked by |
|---|---|---|
Its own user on octo-ctl |
Yes | Disabling the user |
| Its own SSH key to hosts | Yes, in all.yml as its own entry |
Removing it and running the hosts playbook |
| Its own vault password files, readable only by that user | Yes | Deleting the files; rotating the vault passwords if they may have leaked |
| Its own Console API token | Yes | Revoking it in the Console |
This is the custody change. Under ADR-0027 a vault password was supplied by a person and held in Bitwarden. Now the runner holds the vault passwords it needs on disk, unattended, on the control machine. Revoking the runner never touches an operator's access, and an operator can always work by hand if the runner is revoked.
It also concentrates risk: a runner that can redeploy any tenant holds every tenant's vault password, which ADR-0027 §3's per-tenant vault IDs were meant to avoid. Accepted for v1 because the control machine already held the means to provision everything, and recorded as OI-108, together with how a new tenant's vault password reaches Bitwarden (ADR-0027 §11) when the runner creates it.
6. Pausing¶
Two independent ways to stop it:
- A pause toggle in the Console. The runner checks it before claiming each job; a paused runner claims nothing. A job already running finishes.
- Stopping the service on
octo-ctl. Break-glass, for when the Console is the problem.
7. Playbooks from main only¶
The runner keeps its own clean checkout, fetches main, checks out the exact commit and records it. It refuses to run from a dirty tree or any other branch. Ordinary deployment stays deliberate (ADR-0029): merging to main changes what the next job runs, never what is running.
Consequences¶
Provisioning becomes a form, an approval and a record, with the commit and output kept for every run. An operator new to the estate can create a tenant without knowing the playbook layout.
The cost is a new standing credential holder: the runner can provision every tenant without a person present. The guardrails (fixed menu, validated inputs, a passkey step-up bound to each approval (ADR-0040 §4), nothing destructive, pause in two places, revocable identity) are what keep that acceptable, and OI-108 records the concentration.
Break-glass is unchanged: SSH and ansible-playbook by hand still work when the Console or runner does not.
Alternatives considered¶
- The Console runs Ansible itself is rejected: it would put SSH keys and vault passwords on the internet-facing service.
- A CI system (GitHub Actions) as the runner is rejected: vault passwords and SSH access to every host in a third party's runner.
- Free-text or any-playbook jobs is rejected: it turns the Console into a remote shell.
- Arming the runner per session is rejected for v1: friction on every job, when the per-job passkey step-up (ADR-0040 §4) already gates approval.
- Destructive jobs in v1 is rejected until a two-person gate exists (OI-107).