Email triage: eval¶
Measuring whether classification is any good, and whether a change made it worse.
Without this, "the brain works" means "it returned valid JSON". Assist autonomy is the only defensible setting until something measures decision quality (OI-060).
Two different jobs¶
| Regression | Improvement | |
|---|---|---|
| Question | Did this change break anything? | Is version B better than A? |
| Run on | Every prompt or model change | When trying something |
| Needs | A fixed set and a pass threshold | Two runs, compared |
| Cost | Seconds | Can be slower |
Regression first. It is cheap, it runs on every change, and it catches the failure that actually happens: a prompt tweak that fixes one case and breaks three others.
Improvement measurement matters later, and needs a larger set to say anything meaningful.
What can be scored¶
| Scoreable without a human? | How | |
|---|---|---|
| Facts | Yes | A fixture email produces a known set, every time |
| Category | Against labels someone wrote | Needs the walkthrough set |
| Draft quality | No | Subjective. Measured indirectly, via edit rate |
Facts are nearly free¶
Deterministic extraction means the same input always produces the same output, so a fixture and its expected facts is an ordinary unit test. No model call, no cost, no flakiness.
This catches most regressions. A change to the job-number regex, sender resolution or the date parser breaks a test immediately, and those are the things that silently degrade everything downstream.
Run them on every commit.
Category needs labelled examples¶
The set comes from the twenty-email walkthrough during the Process Map, categorised with the person who actually does the triage.
That exercise does three jobs at once, which is why it is worth protecting in the schedule:
| Output | Feeds |
|---|---|
| Validated taxonomy | The category list |
| Labelled examples | The Classify prompt |
| A labelled set | This |
Prompt examples and eval cases must not be the same emails
An email used as a few-shot example in the prompt will be classified correctly, because the answer is sitting in the prompt. Scoring against it measures nothing.
Split the twenty: roughly eight become prompt examples, twelve become the eval set. The capture form must mark which is which.
Draft quality is not scored¶
There is no way to score prose automatically that is worth the effort at this scale. Use the proxies already in reporting:
| Signal | Means |
|---|---|
| Approval rate | Drafts approved without edit |
| Edit rate | Approved but reworded. The better signal |
| Rejection rate | Discarded entirely |
A high approval rate with a high edit rate means people are fixing the output rather than trusting it, which reads as success and is not.
How many cases¶
The count is emails, not categories.
| Stage | Emails | Per category | Enough for |
|---|---|---|---|
| Launch | ~12, from the walkthrough | Mostly one, some none | Catching gross breakage only |
| Working target | 40 to 55 | 3 to 5 | Noticing a specific category degraded |
| Mature | 100+ | 8 to 10 | Aggregate scores that mean something |
The per-category floor is the real rule. With one job_change case you cannot tell whether job_change classification got worse. You need several before a failure is signal rather than noise.
But a small set is still useful, because of the case-level pass rule below: any previously passing case that now fails means the run fails, whatever the aggregate says. That works at twelve. What does not work at twelve is aggregate scoring, since one failure moves the number by eight points and tells you nothing.
Coverage beats count
Forty well-chosen cases including every adjacent pair, several unclear, and
some multi-category will catch more than a hundred easy ones.
Padding with obvious cases inflates the score and measures nothing. If the set is growing but the failures are not getting more interesting, it is getting worse, not better.
The gap between twelve and forty closes on its own, so it is not extra labelling work, just time.
The set grows for free¶
flowchart LR
W["Walkthrough<br/>~12 labelled"] --> S[("Eval set")]
C["Client corrects<br/>a category"] --> S
U["An unclear_category<br/>a human then labels"] --> S
R["A rejected draft's<br/>work item"] --> SEvery human correction is a labelled example. When a client says "that is not a complaint", someone who knows has just labelled it. That is exactly what the set needs, and it arrives as a by-product of running the product (rules: two guards).
Same for unclear_category escalations. A human decided what it actually was, and that decision is a label.
Which turns client dissatisfaction into the training data you would otherwise have to manufacture.
What a case holds¶
| Field | Why |
|---|---|
| The raw message | The input |
| Deterministic facts | So the case runs without a client system |
| Expected category | The label |
Expected categories_present |
For multi-category cases |
| Why, where it was contested | The important one |
| Source | Walkthrough, client correction, or escalation |
| Date labelled | Labels age as a taxonomy shifts |
The "why" field is what makes a disputed case reusable. A year later, "we called this job_status because asking when something finishes is not asking to change it" is the reasoning. Without it, someone re-litigates the same edge case from scratch.
Passing¶
| Rule | |
|---|---|
| Fact extraction | 100%. These are deterministic. A failure is a bug, not a regression in judgement |
| Category | A threshold, not perfection. Some cases are genuinely arguable |
| Any previously passing case now failing | Fails the run, whatever the aggregate says |
That last rule matters most. An aggregate score hides the pattern that actually happens: a change fixes two cases and breaks three, leaving the total flat while the behaviour got worse.
Where it runs¶
| Fact tests | Every commit. No model call, so effectively free |
| Category eval | Before any prompt or model change ships, and on a schedule |
| Cost | Around a dozen cheap model calls per run |
Schedule it as well as running it on change. Providers update models under a stable name, so behaviour can drift with nothing on your side having changed. A weekly run against a fixed set is how you find out.
What this does not do¶
It does not license more autonomy on its own. A passing eval says a change did not break the cases you have, not that the system is good enough to act unsupervised. That decision needs volume, time and a client who wants it.
It does not measure the cases you never labelled. A set built from one client's inbox reflects that inbox. Client two needs its own.