Skip to content

ADR-0027: Secrets and credential custody

  • Status: Accepted
  • Date: 2026-08-17
  • Deciders: El, Lachlan
  • Supersedes: ADR-0014 §9; extends ADR-0010 (the three-tier model stands unchanged)

Context

ADR-0010 chose Ansible Vault and placed the vault key in "the founders' password manager." Two gaps have since become material: there is a single vault key for the whole estate, and "the founders' password manager" is not a place — at two people with different personal managers, it is an assumption that drifts silently as tenants are added.

AGENTS.md also carries "Ansible Vault / SOPS", which reopens a question ADR-0010 already closed. Two encryption mechanisms for one class of value is not flexibility; it means a secret eventually lands in the wrong store and a rotation covers half the estate.

Decision

  1. Ansible Vault remains the secrets tool for ABOS (ADR-0010 stands). SOPS is not used in this repository. AGENTS.md is corrected accordingly.
  2. The portal repository continues to use SOPS + age. This divergence is deliberate: the portal is a single self-deploying box on Railway (pull model, where a host decrypting its own secrets is the point); ABOS is push-provisioned multi-tenant infrastructure. Neither repo is "corrected" to match the other.
  3. A separate --vault-id per tenant, plus one for core. One vault password no longer opens every client.
  4. Values encrypted inline with ansible-vault encrypt_string, so variable names stay readable in git and a secrets change is reviewable in a PR without being decrypted.
  5. No vault password or private key is stored on a provisioned host. Key material stays on the control machine. This is the property that makes Vault the right fit for a push model.
  6. Bitwarden Teams (cloud-hosted) is the shared credential store. One jointly owned collection; both partners admins; it survives either individual account. Personal password managers remain a free choice for personal credentials and are not used for Octopodia infrastructure. MFA on both accounts, with Bitwarden recovery codes stored outside Bitwarden.

Not self-hosted, and not Vaultwarden: hosting the credential store on the infrastructure whose credentials it holds is circular — when the box is down, the vault needed to fix it is unreachable.

Not shared out of one partner's personal vault: that makes credentials owned by one person and borrowed by the other, with asymmetric revocation. 7. The boundary between stores is human access vs machine access, not ours vs the client's:

Store Example
Human — a person logging in to a console Bitwarden Cloudflare login, Microsoft 365 global admin, Vultr portal
Machine — a token a playbook or container decrypts at runtime Ansible Vault Cloudflare API token used by the dns role, client Xero/AroFlo keys

The same provider therefore appears in both stores. Machine tokens are scoped to the minimum required — for Cloudflare, DNS edit on the specific zone, never the global key (ADR-0022 §9). 8. Per-client OAuth tokens are unchanged from ADR-0010: obtained at runtime through a consent flow, stored encrypted in n8n's own database, protected by an n8n encryption key that is itself Vault-managed. Vault cannot manage dynamically-refreshing tokens. 9. SSH private keys are never shared (ADR-0021 §4). 10. Adding a tenant includes confirming its vault password is in the shared collection before the tenant is considered provisioned (ADR-0024 §11). A canonical list of credential names — never values — is maintained in the repository so the collection can be audited without opening it. 11. Microsoft 365 break-glass account: deferred, deliberately. Two global admins exist on Octopodia accounts, but both are working identities under the same conditional access policies, so a single bad policy could lock out both. Mitigation, in force now: both admins' MFA recovery codes in Bitwarden, at least two MFA methods registered per admin, and no tightening of conditional access policies until the break-glass account exists — that is the mechanism that would cause the lockout. Trigger to add it: the first paying client.

Consequences

Either partner can operate the estate alone, and credentials survive the loss of either personal account. A leaked vault password is bounded to one client rather than all of them. Nothing on a provisioned host can decrypt the secrets store.

Two credential stores means remembering which holds what, and a single provider appearing in both looks like duplication but is not — rotating Cloudflare access means considering the login and the scoped token separately. The human/machine split has a useful property: anything a playbook needs is by definition in Ansible Vault, so a deploy never waits on a person opening Bitwarden.

The dependency on a third-party cloud credential service is accepted deliberately: its availability is independent of our infrastructure, which is the entire point.

Alternatives considered

  • SOPS + age for ABOS — rejected here. Its principal advantage is a host decrypting its own secrets unattended, which suits pull-based deploys; ABOS deploys by push, so that advantage is unrealised while its cost — a private key at rest on a multi-tenant host — is paid in full. Its secondary advantage, reviewable diffs, is matched by encrypt_string.
  • Both tools, split by scope — rejected: the boundary is precisely where a value gets filed into the wrong store.
  • Self-hosted Bitwarden / Vaultwarden — rejected: circular dependency, plus a service to patch and back up with total blast radius if breached.
  • Sharing from a personal vault — rejected: single ownership, asymmetric revocation, dies with that account or subscription.
  • A hosted secrets manager for machine secrets — deferred: revisit if the operator count grows or an audit requires access logging.