Email triage: reporting¶
What to measure, what each number means, and what you do about it. A metric with no decision attached is decoration.
Three audiences, and they want different things.
| Audience | Question | Section |
|---|---|---|
| Us, operating it | Is it working, and where is it weak? | Health |
| The client | What did we get for the retainer? | Value |
| Us, commercially | Is this profitable and repeatable? | Commercial |
Definitions are locked before the first run
Redefining a metric in month four silently invalidates months one to three,
and nobody notices until a trend looks wrong (OI-008).
Work items pin a config_version precisely so a change is visible rather
than retrospective.
Health (us)¶
Run weekly. Every one of these has a different fix.
Outcome mix¶
| Metric | Rising means | Do this |
|---|---|---|
sent_by_us share |
Working as intended | Nothing. This is the number you want going up |
| "Worked correctly" flags | More complex work arriving, or rules too broad | Check which reasons. If high_value dominates, the threshold may be too low |
| "Fell short" flags | The system is degrading | Break down by reason. Each one has its own fix, below |
closed_manual share |
Too slow to be useful | Look at time-to-draft, not accuracy. A different problem entirely |
closed_by_human share |
Our drafts are not being used | Sample them. Either the draft was wrong, or the flag made someone distrust it |
closed_no_action share |
Lots of noise arriving, or over-eager closing | Sample them. Closing something that needed a reply is the worst failure here |
The two kinds of flag must never be summed. One is the product working, the other is a gap. A combined "flagged rate" is uninterpretable.
And a flag is not an outcome. An item flagged legal_risk that was drafted, approved and sent is sent_by_us. Counting it as a failure would understate what the product did.
"Fell short" breakdown¶
The single most useful report. Each reason points at a different part of the system.
| Reason | Means | Do this |
|---|---|---|
unclear_category |
The taxonomy no longer fits the inbox | Sample 20. If they cluster, that is a missing category. If scattered, the prompt is weak |
parse_failure |
Prompt or model regression | Check whether it started on a deploy or a model version change |
ambiguous_job |
Resolution needs work | Is it always the same customer? They may need job references in their signature |
unresolved_reference |
Pattern matches something real that is not a job | Check the tenant's job-number regex. May be catching invoice or PO numbers |
multi_category |
Customers batching requests | Often falls as response time improves. If it does not, consider resolve_then_decide |
action_not_permitted |
Rules and the model disagreed | Normal in small numbers. Rising means the rules are fighting the prompt |
no_permitted_action |
Rules contradict each other | A rule bug. Should be near zero |
action_not_enabled |
The brain wants an action the tenant lacks | Either enable it, or the rules are pointing somewhere wrong |
call_ceiling |
Something is looping | Investigate individually. This should be rare, and each one is a bug |
Quality¶
| Metric | Means | Do this |
|---|---|---|
| Approval rate | Drafts approved without edit | The headline quality number. Falling means voice or content is drifting |
| Edit rate | Approved but reworded | Sample the diffs. Consistent rewording is a prompt fix, and the diffs are the training signal |
| Rejection rate | Approved by nobody, discarded | Each one is worth reading. This is the model being wrong, not imprecise |
| Category accuracy | Against a labelled set | Needs the eval harness (OI-060) |
| Fact extraction accuracy | Against fixtures | Objectively scorable, no human needed. Most of your regression protection |
Edit rate matters more than approval rate. A high approval rate with a high edit rate means people are fixing it rather than trusting it, which reads as success and is not.
Timeliness¶
| Metric | Means | Do this |
|---|---|---|
| Time to draft | Arrival to draft ready | Should be seconds. Minutes means a queue or a slow lookup |
| Time to approval | Draft ready to human decision | Their responsiveness, not ours. Drives closed_manual |
| Time to send | Arrival to sent | The number the client actually feels |
| Approval expiry rate | Items timing out | The approver is overloaded or disengaged. A relationship problem, not a technical one |
Cost¶
| Metric | Means | Do this |
|---|---|---|
| Model calls per work item | The real cost driver | Above ~3 average, something is looping or the loop is too chatty |
| Tokens per item, in and out | Rising with no volume change means prompt bloat | |
| Cost per item | Total spend divided by work items | The number that decides whether the pricing works |
| Cost per email actually sent | Spend divided by sent_by_us outcomes |
The honest one. Escalated and closed items cost model calls and produce no automation |
Value (the client)¶
Monthly. Fact and model are presented separately, and never blended.
Facts¶
Countable, defensible, no assumptions.
| Metric | Presented as |
|---|---|
| Emails handled | Total work items |
| Drafts prepared | |
| Approved and sent | |
| Correctly flagged for a human | Framed as caught, not as failed |
| Documents retrieved | Where document_request was answered |
| Fastest and median response time | Against their baseline |
Model¶
Estimates, labelled as estimates.
| Metric | Built from | Guard |
|---|---|---|
| Hours returned | Items handled × the agreed minutes-per-item | Their figure, agreed at Blueprint (OI-009) |
| Dollar value | Hours × their loaded hourly cost | Their figure too |
| Response time improvement | Now versus baseline | Only valid if the baseline was captured |
Never blend fact and model in one number
"412 invoices processed" and "38 hours returned" are different kinds of claim. The first is countable. The second rests on a multiplier they agreed to, and when it is challenged the answer must be their own sign-off rather than our assumption.
What not to report¶
| Avoid | Because |
|---|---|
| A combined flagged rate | Sums a success and a failure |
confidence scores |
Weakly calibrated, and invites false precision |
| Accuracy without a labelled set | Unsupportable |
| Anything not traceable to the capture form | Cannot be defended when questioned |
Commercial (us)¶
Monthly, internal.
| Metric | Means | Do this |
|---|---|---|
| Cost per item vs price per item | The margin | If cost is a large share, either the price or the loop is wrong |
| Items per tenant per month | Volume, and whether the pricing tier fits | |
| Bespoke cards or rules per tenant | Drift | Rising means "config, not code" is slipping (ADR-0003) |
| Time spent on this tenant | Support load | Falling should follow the first month. If not, something is wrong |
| Category set size vs the template | Divergence | Well above 15 means the taxonomy is fragmenting |
Cost per item versus price is the one to watch first. Model costs are the only line that scales directly with client volume, and a chatty loop turns a profitable client into a break-even one without anything appearing broken.
Throughout this page, sent means an email that went out to the customer (
sent_by_us), never a request sent to the model. Model usage is counted as model calls and tokens.
Rules health¶
Run monthly, per tenant. Rules are config, so they drift quietly and nobody notices until behaviour is odd.
| Metric | Rising means | Do this |
|---|---|---|
| Guardrail fire rate | More genuinely serious mail, or a word list that is too broad | Sample them. A guardrail firing on routine mail is worse than one that never fires, because people start ignoring it |
| Guardrails that never fire | The word list misses real cases, or the risk is not present | Test against known examples. A guardrail nobody has ever seen fire is unproven, not safe |
set_category override count |
Classification quality being papered over | More than a handful for one tenant is a model problem, not a config problem (rules) |
set_attention_reason override count |
Someone is suppressing a flag | Read every one. Suppressing sentiment for a person creates a permanent blind spot |
| Tenant rule count | Config drift, or a genuinely unusual client | Well above the vertical template means "config, not code" is straining |
no_permitted_action flags |
Rules contradicting each other | Each one is a rule bug. Should be near zero |
| Rules that never fire | Dead config | Remove them. Dead rules make the set harder to reason about |
The two override counts are the important ones. They are the only place where a client's dissatisfaction converts silently into a permanent behaviour change, and they are also your best signal about classification quality.
Every override is a labelled example. A human said "this is not a complaint", which is exactly what the eval set needs (OI-060). Client complaints become training data.
Where these come from¶
| Source | Feeds |
|---|---|
work_item.status |
Outcome mix |
work_item.attention_reason |
Both flag breakdowns |
approval records |
Approval, edit and rejection rates |
decision token counts |
All cost metrics |
decision.rules_fired |
Rules health |
| Tenant rule config | Rule counts and dead rules |
| Timestamps across both | Timeliness |
| The capture form | Baseline comparisons |
Every decision records which rules fired. Without that, rule health cannot be measured and a misbehaving rule cannot be traced back from a bad outcome.
Aggregates leave the tenant and rows never do (ADR-0032), so cross-tenant reporting queries the registry rather than N databases.
Correction alerts¶
Human corrections are the clearest signal the Senior is drifting, and they reach us as alerts in the ADR-0034 model. Same design as the Email Assistant, Junior:
| Alert | Fires when | Kind |
|---|---|---|
| Corrections creeping up | More than a set share of items corrected over the last 7 days | Advisory. The categories or prompt no longer fit this tenant. A conversation |
| Corrections spiking | A sudden jump in a day, especially just after a prompt or model change | Incident. Something we changed broke it |
Thresholds are a count or a percentage, configurable per tenant, and corrections are counted by type, since they mean different things:
| Type | Signals |
|---|---|
corrected_category set |
The read was wrong. Classify needs work |
regenerated |
The words were wrong. Draft needs work |
handled_by_human |
Possibly neither. A workflow preference unless it clusters |
They also run globally. A rise across every tenant at once points at a model change on the provider's side, invisible tenant by tenant.