Console first light¶

The hand steps that bring up phase 1 of the BOS Console on lite-dev: read-only, Fleet and Mailbox health from real numbers. The design is on the Console page; the decisions are ADR-0037 to ADR-0040.

The inventory moved, 16 October 2026

The live inventory is now its own private repository, octopodia-abos-inventory, cloned beside the code (ADR-0042). Paths below written as infra/inventory/... or inventory/... are now in that repository, and secrets no longer go in group_vars/all/vault.yml: a host's go in host_vars/<host>/vault.yml (vault ID core), a tenant's in tenants/<slug>/vault.yml (the tenant's own vault ID), each value added with ansible-vault encrypt_string. See the inventory repository runbook.

flowchart LR
    OPS(["El, Lachlan<br/>laptop or phone"]) -->|"console.octopodia.com.au<br/>Access: Entra SSO + MFA"| CF["Cloudflare edge"]
    N8N["lite-dev n8n<br/>Junior: tenant link"] -->|"link.octopodia.com.au<br/>Access service token + link token"| CF
    CF -->|"tunnel, dialled out"| CFD["cloudflared"]
    CFD --> CON["Console<br/>checks the Access token"]
    CON --> REG[("registry<br/>Postgres")]

This reuses what the docs site already has: the Entra app and the Entra identity provider in Cloudflare Access. New here: the tunnel, two hostnames, two Access applications and one service token.

Step	Where	Who
1. Host memory	`octo-ctl`	El
2. Docker on `octo-ctl`	`octo-ctl`	El
3. Tunnel and hostnames	Cloudflare Zero Trust	El
4. Service token and Access applications	Cloudflare Zero Trust	El
5. MFA for both operators	Entra admin centre	El
6. Secrets in the vault	`octo-ctl`	El
7. Inventory	`octo-ctl`	El
8. Merge to `main` and run the playbook	GitHub, `octo-ctl`	El
9. The link's credential, then publish	lite-dev n8n	El
10. Done checks	A laptop and a phone	El, Lachlan

First real run, 13 October 2026

The lessons from it are written into the steps below: copying the service token's values without their labels (step 4), the AUD tag's place in the newer dashboard (step 4), Docker and the ssh-agent on octo-ctl (step 2), SSH to lite-dev on port 52843 with sudo for Docker (step 1), Entra Free's security defaults (step 5), and the curl test's read lines (step 10). The first playbook run failed at Migrate the registry with ModuleNotFoundError: No module named 'app': Alembic could not import the Console's package inside the image. Fixed in c225f07 (prepend_sys_path = . in alembic.ini, PYTHONPATH=/app in the image); the rerun passed.

Junior testing runs on this host's n8n

The playbook never touches n8n: it refuses the n8n and docker roles, and fails if n8n's container start time changes during the run. Nothing in these steps restarts n8n. Step 9 adds a credential and publishes one new workflow; it changes no Junior workflow.

1. Host memory¶

SSH to lite-dev is on port 52843 over the tailnet, not 22, and Docker on the host needs sudo for the el user. The commands in this runbook assume an ~/.ssh/config entry on octo-ctl:

Host lite-dev
  HostName lite-dev.tail34193f.ts.net
  Port 52843
  User el

ssh lite-dev 'free -h; nproc'

The registry, the Console and cloudflared add about 640 MB at their limits (256 + 256 + 128) next to n8n. The playbook refuses a host under 3,200 MB (Vultr's 4 GB plan reports 3,392 MB total in free -m). On a 2 GB plan, resize to 4 GB first, outside a test round.

2. Docker on `octo-ctl`¶

The image is built on octo-ctl, never on a tenant host.

docker version && docker run --rm hello-world

If Docker is missing, on Ubuntu:

sudo apt install -y docker.io docker-buildx
sudo usermod -aG docker $USER

Log out and back in, so the group applies, and run the check again.

Then reload the ssh-agent. Logging out ends the agent that held your key, and the playbook fails with Permission denied (publickey):

eval "$(ssh-agent -s)" && ssh-add

To stop this happening at every login, use keychain: sudo apt install -y keychain, then add eval "$(keychain --eval --quiet ~/.ssh/id_ed25519)" to ~/.bashrc (with your key's file name).

3. Tunnel and hostnames¶

Cloudflare dashboard, Zero Trust, Networks, Tunnels, Create a tunnel:

Field	Value
Connector	Cloudflared
Name	`abos-lite-dev`

On the install page, do not run the command it shows. Copy only the token: the long string after --token. Put it in Bitwarden now as "cloudflared tunnel token: lite-dev". It goes in the vault in step 6.

Next, add two public hostnames (the dashboard may call them published application routes):

Subdomain	Domain	Path	Service
`console`	`octopodia.com.au`	(empty)	`HTTP` `console:8000`
`link`	`octopodia.com.au`	(empty)	`HTTP` `console:8000`

console is the container's name on the network it shares with cloudflared. Both hostnames go to the same Console: it tells operators from tenant links by the Access audience in the token.

The tunnel shows Inactive until step 8 starts cloudflared.

4. Service token and Access applications¶

Do these before step 8. Until the Access applications exist, the hostnames would reach the Console with no Access in front. The Console refuses every request without a valid Access token, but Access is the first gate.

Service token, for the tenant link¶

Access, Service auth, Service tokens, Create service token:

Field	Value
Name	`link-lite-dev`
Duration	1 year

Copy the Client ID and Client Secret to Bitwarden as "Console link service token: lite-dev". The secret is shown once. Put a calendar reminder 11 months out to rotate it.

Copy the values only

Cloudflare shows each value as a header line, CF-Access-Client-Id: <value> and CF-Access-Client-Secret: <value>. Copy only the part after the colon and space, never the label. A label pasted into the credential in step 9 sends a header Access does not recognise, and every push is refused with 403. That happened on the first run.

Application 1: the Console¶

Access, Applications, Add an application, Self-hosted:

Field	Value
Name	`BOS Console`
Session duration	8 hours
Domain	`console.octopodia.com.au`
Identity providers	Only the Entra provider the docs site uses. Untick Accept all available identity providers; tick Instant Auth

Policy:

Field	Value
Name	`Operators`
Action	Allow
Include	Emails: `el@octopodia.com.au`, `lachlan@octopodia.com.au`

Save. Open the application again and copy its Application Audience (AUD) Tag. In the newer dashboard it is under the application's Application settings.

Application 2: the tenant link¶

Add an application, Self-hosted:

Field	Value
Name	`BOS Console link`
Domain	`link.octopodia.com.au`
Identity providers	None needed

Policy:

Field	Value
Name	`Tenant links`
Action	Service Auth
Include	Service Token: `link-lite-dev`

Save, and copy this application's AUD tag too (Application settings again). It must differ from the first.

Team domain¶

Settings, Custom pages (or the top of Settings): the team domain, <team>.cloudflareaccess.com.

You now have: the tunnel token, the service token's ID and secret, two AUD tags, the team domain.

5. MFA for both operators¶

Access trusts Entra for MFA, so check Entra enforces it for both of us on this app.

On Entra ID P1 or P2: Protection, Conditional Access. Confirm a policy that requires multifactor authentication includes El and Lachlan and covers the Cloudflare Access enterprise application (or All cloud apps), with no exclusion for either of us.

On Entra Free (our tenant, 13 October 2026), Conditional Access is not available. Check Security defaults instead: Entra ID, Overview, Properties, Manage security defaults. It must be Enabled. Security defaults always asks Global Administrators for MFA, but other users only when Microsoft judges it needed, so it does not guarantee MFA for Lachlan on every sign-in. The sign-in log below is the real check.

Then, after step 10's first sign-ins: Identity, Monitoring, Sign-in logs, filter on the Cloudflare application. Each operator's sign-in entry must show Multifactor authentication under Authentication requirement. Single-factor for either of us: stop and fix it (per-user MFA, or Conditional Access on a paid plan) before using the Console.

6. Secrets in the vault¶

On octo-ctl, in ~/octopodia-abos/infra:

openssl rand -base64 33 | tr -d '/+='    # registry owner password
openssl rand -base64 33 | tr -d '/+='    # registry app password (run again: they must differ)
openssl rand -hex 32                     # the link token for lite-dev
ansible-vault edit inventory/group_vars/all/vault.yml

Add:

vault_registry_owner_password: "<first value>"
vault_registry_app_password: "<second value>"
vault_console_link_token_lite_dev: "<64 hex characters>"
vault_cloudflared_tunnel_token: "<the tunnel token from step 3>"

Store the link token in Bitwarden as "Console link token: lite-dev" (step 9 needs it). Then clear.

These are lite-dev's own secrets. group_vars/all is right only while lite-dev is the only host; on a second host they go in inventory/host_vars/<host>/vault.yml (known trap 15).

7. Inventory¶

In inventory/hosts.yml (not in git), add a console group inside all:, level with hosts:, next to n8n::

all:
  hosts:
    lite-dev:
      # ... unchanged
  children:
    n8n:
      hosts:
        lite-dev:
    console:
      hosts:
        lite-dev:
          console_access_team_domain: "<team>.cloudflareaccess.com"
          console_access_aud_console: "<AUD tag of BOS Console>"
          console_access_aud_link: "<AUD tag of BOS Console link>"
          console_tenants:
            - slug: lite-dev
              name: "lite-dev (tenant zero)"
              region: au-mel
              link_token: "{{ vault_console_link_token_lite_dev }}"
              # The Mailbox page's "See the emails in n8n". Open sort_log in n8n and copy the address.
              sort_log_url: "https://lite-dev.tail34193f.ts.net/projects/5ScJFQaqGARP9VUv/datatables/hc2KWp2dPmhnP3XQ"
              mailboxes:
                - { address: admin.octopodia@gmail.com, product: junior }

Check: ansible-inventory --graph shows @console: with lite-dev under it.

8. Merge to `main` and run the playbook¶

The console role builds only from a clean checkout of main. Open a pull request from dev to main, merge it, then on octo-ctl:

cd ~/octopodia-abos && git checkout main && git pull
cd infra && ansible-playbook playbooks/console.yml --limit lite-dev

What it does, in order: refuses a small host, records n8n's start time, starts the registry, builds the Console image here and copies it to the host, migrates and seeds the registry, starts the Console on the host's loopback, checks /health answers and /api/fleet refuses a request without a token, starts cloudflared and waits for the tunnel to register, then checks n8n's start time is unchanged.

The tunnel now shows Healthy in Zero Trust.

On the host, the memory in use:

ssh lite-dev 'sudo docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}"; free -h'

9. The link's credential, then publish¶

In lite-dev's n8n, Credentials, Create credential, Custom Auth:

Field	Value
Name	`Console link: lite-dev`
JSON	below

{"headers": {
  "CF-Access-Client-Id": "<service token Client ID>",
  "CF-Access-Client-Secret": "<service token Client Secret>",
  "Authorization": "Bearer <Console link token: lite-dev>"
}}

Open Junior: tenant link, select this credential on Get cursor, Push health and Push hourly, save, and click Execute workflow once:

Get cursor returns {"cursor": {"admin.octopodia@gmail.com": null}}.
Push health returns accepted.
Push hourly sends the last 7 days, 168 hours, and returns the new cursor.

Then publish it. From now on it runs every 5 minutes. It only reads the Junior's tables, so if the Console is down, sorting is unaffected.

10. Done checks¶

El signs in from a laptop at https://console.octopodia.com.au, through Entra with an MFA prompt, and sees Fleet.
El signs in from a phone and Fleet is readable without zooming.
Lachlan signs in the same way.
Anyone else is refused. A Microsoft account that is neither of us gets Access's not authorised page.
MFA in the sign-in logs for both operators (step 5).
No token, no data. From any machine: curl -sI https://console.octopodia.com.au/api/fleet returns a redirect to the Access login, never JSON.
The numbers match. On Fleet and on admin.octopodia's Mailbox page, today's totals by bucket and corrections match today's rows in sort_log and corrections.
A bad payload is refused whole. On octo-ctl, read the three values from Bitwarden into the shell. Run each read on its own, one line at a time, and paste the value when it waits: pasting all the lines together feeds the following lines into the first read as its value.
```
read -rs CF_ID
```
```
read -rs CF_SECRET
```
```
read -rs LINK_TOKEN
```
Then:
```
curl -s -o /dev/null -w '%{http_code}\n' https://link.octopodia.com.au/link/v1/health \
  -H "CF-Access-Client-Id: $CF_ID" -H "CF-Access-Client-Secret: $CF_SECRET" \
  -H "Authorization: Bearer $LINK_TOKEN" -H 'Content-Type: application/json' \
  -d '{"catalogue":"junior.v1","observed_at":"'"$(date -u +%FT%TZ)"'","mailboxes":[{
       "mailbox":"admin.octopodia@gmail.com","last_poll_ok_at":null,"last_polled_at":null,
       "history_reset_at":null,"poll_minutes":2,"pending_count":0,"oldest_pending_minutes":0,"today":{},
       "subject":"x"}]}'
```
The payload is valid apart from subject, a field outside the catalogue. It must print 422 and nothing is stored. Do not send it without subject: it would store a snapshot of zeros. Then unset CF_ID CF_SECRET LINK_TOKEN and clear. - [ ] Link quiet shows. Unpublish Junior: tenant link. Within 16 minutes Fleet shows admin.octopodia under Needs you as link quiet. Publish it again; within 5 minutes it clears. - [ ] n8n was not restarted. The playbook's last play passed (n8n's start time is unchanged).

11. The masked view of corrections (ADR-0041)¶

Each count in a mailbox's Corrections section opens the corrections behind it, masked: row number, sorted and corrected times, rule, from and to bucket. The Console asks the tenant's Junior: detail webhook over the tailnet, with a token per tenant. Until both are in place the counts stay plain numbers.

Token, on octo-ctl:
```
openssl rand -hex 32
ansible-vault edit inventory/group_vars/all/vault.yml
```
```
vault_console_detail_token_lite_dev: "<64 hex characters>"
```
Store it in Bitwarden as "Console detail token: lite-dev", then clear. (group_vars/all while lite-dev is the only host; known trap 15.)

Inventory, in lite-dev's entry under console_tenants:

detail_url: "https://lite-dev.tail34193f.ts.net/webhook/junior-detail"
detail_token: "{{ vault_console_detail_token_lite_dev }}"

n8n credential, on lite-dev's n8n: Credentials, Add, Header Auth, named Console detail: lite-dev, name Authorization, value Bearer <the token>. Select it on the Webhook node of Junior: detail, save, and publish Junior: detail. Never publish it without the credential selected.
Merge to main and run the console playbook. It migrates the registry (the view record), writes the token file and seeds detail_url. It also reads the detail_url's host from this host's tailscale status and writes it, with its tailnet IP, into the Console container's /etc/hosts: the container uses Docker's DNS, which cannot resolve tailnet names (known trap 16). If that name is not a node this host can see, the playbook stops and says so.
Check the path from the Console's container to n8n over the tailnet, on lite-dev:

First from the host itself, which proves the webhook and tailscale serve without Docker. The host does not take MagicDNS (accept-dns=false), so give curl the address:
```
curl -sS -o /dev/null -w '%{http_code}\n' --resolve "lite-dev.tail34193f.ts.net:443:$(tailscale ip -4 | head -1)" -X POST -H 'Content-Type: application/json' -d '{}' https://lite-dev.tail34193f.ts.net/webhook/junior-detail
```
It must print 403. Then from the Console's container:
```
sudo docker exec console grep ts.net /etc/hosts
sudo docker exec console python -c '
import urllib.request as u, urllib.error as e
r = u.Request("https://lite-dev.tail34193f.ts.net/webhook/junior-detail", data=b"{}", headers={"Content-Type": "application/json"})
try: print(u.urlopen(r, timeout=10).status)
except e.HTTPError as x: print(x.code)
except Exception as x: print("FAIL", x)
'
```
The first line shows the name with lite-dev's tailnet IP. The second must print 403 (the webhook refusing a call without the token). 404 means Junior: detail is not published. FAIL ... Name or service not known means the name is not in the container's /etc/hosts (rerun the playbook); FAIL ... timed out or refused means the name resolves but the tailnet path does not answer: fix that before going on.
Test on the Console: open the mailbox, select a correction count. The rows appear with Open row N in n8n, and History (in the top bar) shows the look, with who and when. No sender or subject appears anywhere on the Console.

To take it off: unpublish Junior: detail. The counts then answer "Could not reach this tenant's n8n", and History records each try.

If it has to come off¶

On the host. The registry's data stays in /opt/registry/data.

cd /opt/cloudflared && sudo docker compose down
cd /opt/console && sudo docker compose down
cd /opt/registry && sudo docker compose down

Unpublish Junior: tenant link first, so it stops trying.