Skip to content

ADR-0019: SkillOpt is an optional, deferred optimiser for the learned layer — not a core capability, not a differentiator

  • Status: Accepted
  • Date: 2026-07-31
  • Deciders: El
  • Related: ADR-0008 (residency), ADR-0015 (staging/prod), ADR-0017 (principals; rule precedence; learned layer), ADR-0018 (Hermes not adopted — controlled vs runtime self-evolution), the llm-client boundary; rule-curator / eval-harness roadmap items

Context

SkillOpt (Microsoft Research, MIT-licensed, ~May 2026, on PyPI) treats a skill .md as a trainable artifact: an optimizer model proposes bounded add/delete/replace edits from scored rollouts and keeps a candidate only if a held-out validation set improves. It is offline/training-time; the output is a static best_skill.md deployed like code — no model weights touched, no GPU, negligible extra inference cost. It is the disciplined form of the self-evolution we rejected in Hermes (ADR-0018): improvement proposed offline and reviewed, versus rewritten live in production.

Decision

Adopt SkillOpt as an optional module, later — not a core platform capability and not a standalone differentiator.

  1. Placement. It automates the "optimise" step of the learned rule layer (ADR-0017). It fits the triage / classification / extraction side (Capture engine), where correctness is scorable. It is a poor fit for the drafting / voice side, which is subjective — that stays retrieval-of-approved-examples, per the two-feedback-channel split (ADR-0017). This is the seam we already built.
  2. Compatible with "No review, no go" only with our gate on top. SkillOpt's validation gate catches performance regression, not safety/policy violations. A human approval gate must sit between "SkillOpt proposes an optimised skill" and "deploy to a client"; our guardrail layer remains mandatory. A LearnedRuleCandidate is promoted to an active Rule only on human approval (ADR-0017).
  3. The prerequisite is a scored rollout / eval harness — the ability to auto-score task performance, plus validation data per skill. This is the same prerequisite as our own rule curator. Build the eval harness first; SkillOpt is inert without it. With ~1 client there is no trajectory volume and no skill sprawl to optimise — it solves a scale problem we do not yet have.
  4. Not a moat. MIT-licensed Microsoft tooling anyone can run. The differentiator is our per-client data, the eval harness and the governed pipeline — not the optimiser. Same lesson as Hermes (ADR-0018).

Do now (cheap, high-value): design the learned-layer skill artifacts and the eval-harness seam so SkillOpt — or DSPy, or our own build — can slot in when volume justifies it. "Bake the metadata in now, build later."

Consequences

When adopted, per-client skills (Finance, Sales, HR, Customer Service, Invoice Processing) become versioned artifacts in the tenant's learned layer, promoted staging → prod (ADR-0015) with rollback. When a learned skill validated for one client is abstracted into a shared vertical template, the promoted artifact must be treated as tenant-scoped and scrubbed of client-specific data — it must not leak client-derived content into shared IP. Explainability is a strength: output is a readable .md diff, not opaque weights — good for the audit story.

Alternatives considered

  • SkillOpt as a core platform capability now — rejected: premature (no eval harness, no volume), and it is not a moat.
  • Fine-tuning per client — rejected (standing decision): retrieval over fine-tuning; SkillOpt is consistent with that (no weight changes).
  • Loose runtime self-evolution (e.g. Hermes-style) — rejected (ADR-0018): SkillOpt is the controlled, offline, validation-gated alternative.
  • Using SkillOpt on the voice/drafting layer — rejected: subjective quality is not reliably scorable; community results on creative/copy tasks are mixed.

Risks

Overfitting on small per-client validation sets; skill drift across successive offline runs (mitigate: bounded edits, version-pin, rollback); the safety-vs-performance gap in its validation gate (mitigate: our human gate + guardrails on top); the optimiser loop feeds client trajectories to an optimizer LLM — client data crossing the boundary — so it must route through llm-client and honour AU residency; per-client skills must be tenant-isolated like all tenant content.

Acceptance criteria (2026-07-31): residency is a definition-of-done, not a note

SkillOpt's optimiser loop feeds client trajectories to an optimiser LLM — client data crossing the model boundary. When SkillOpt (or any equivalent optimiser) is adopted, it is NOT complete until: 1. All optimiser LLM calls route through packages/llm-client — no direct SDK or third-party path. 2. AU data residency (ADR-0008) and provider strategy (ADR-0009) are honoured on those calls. 3. Per-client skills/trajectories are tenant-isolated like all tenant content. This travels with the "adopt SkillOpt later" task; do not treat it as advisory prose.