Skip to content

Tester setup: El's checklist

Everything El does before, during and after a round of Junior testing. The testers' side is in the README.

When What Time
Once per tester Give them the mailbox, make them a known sender, GitHub access 15 minutes
Once per round Reset learning, send the pack, watch the run, close it
Once, ever Prove a second mailbox 15 minutes

Why delegation, not the password. admin.octopodia@gmail.com also owns the Google Cloud project with the OAuth app and the Gemini key. Delegation gives a tester the mailbox and nothing else: no settings, no Cloud project, no password change, and it is removed in one click.

1. Give each tester the mailbox

Each tester needs a Google account with Gmail. A free gmail.com address is fine; a Microsoft 365 work address is not.

  1. Sign in to Gmail as admin.octopodia@gmail.com.
  2. Settings (the cog) → See all settings → Accounts and Import.
  3. Grant access to your account → Add another account → the tester's Gmail address → Next step → Send email to grant access.
  4. On the same row, choose Leave conversation unread when opened by others. The read-status tests depend on it.
  5. The tester accepts the emailed link. It can take a while, sometimes up to a day, before the mailbox appears for them.
  6. Check: they can open the mailbox from their own Gmail (profile picture, top right) and see the Assistant labels.

2. Make each tester a known sender

Known senders come from the mailbox's Sent folder. Since stage 6, every poll adds the recipients of newly sent mail, so no setup run is needed.

  1. From admin.octopodia@gmail.com, send each tester's own address a short email ("Testing starts this week, thanks").
  2. Wait a few minutes for the next poll.
  3. Check: known_senders has a row for each tester's address, with mailbox = admin.octopodia@gmail.com and source = sent_history.

3. Send the pack

Before each round: reset learning

Start every round from what the assistant knew before any testing, so one round's corrections do not shape the next. The same as Close the round step 5; running it again here catches anything learned since.

  1. In n8n, open Junior: reset learning and click Execute workflow. The form opens.
  2. mailbox_id: admin.octopodia@gmail.com. dry_run: Yes, dry run only. Submit.
  3. Read the report (the last node, Dry run report): each row it would change, with before and after. Check nothing on the list should keep what it learned.
  4. Run it again with No, change the rows. The report (What changed) lists what it changed and the count, which should match the dry run.

It changes only that mailbox's known_senders rows, exactly as in the hand method below, never deletes a row, and never touches corrections, sort_log or any other table. An empty or unknown mailbox_id stops it before anything is written.

If the workflow is unavailable, use the hand method in Close the round, step 5.

The pack

  1. Create the round's Project, once per round, from a machine where gh is logged in as you:

    gh auth refresh -s project
python tools/junior_test_round.py --round 1 --testers Nathan=Bones50,Lachlan=LB-octopodia

    It makes Junior testing: round 1, linked to the repository, with one row per scenario per tester (60 for two testers and 30 scenarios), each set to Not run and assigned to that tester's GitHub account (Nathan is Bones50, Lachlan is LB-octopodia). Add --dry-run first to see what it would do. Then, in the browser: - Settings → Manage access: add each tester as a writer. If assigning failed because they had no access yet, rerun with --project <number> in place of --round (the number is in the Project's URL): it only assigns, and skips rows already assigned. - Views: By tester (table, grouped by Tester), Fails (filtered to Result is Fail), Must pass (filtered to Must pass is Yes). The script can't create views; GitHub has no API for them. New scenarios in the middle of a round (such as J24 to J28 for stage 5): add their rows to the existing Project instead of starting a new round, then give them the same treatment:

    python tools/junior_test_round.py --project 1 --add J24,J24a,J25,J25a,J26,J27,J28 --testers Nathan=Bones50,Lachlan=LB-octopodia

    If the Project's Area field has no option for a new area (Guardrails, Threads), the script says so and leaves it blank: add the option in the browser and set it on those rows. 2. Send each tester the link to this folder on the docs site (Testing, Email Assistant, Junior) and the date the round ends. The workbook downloads from the guide. 2. Add each as a collaborator on zurelc/octopodia-abos, so they can log issues with the Junior test issue form (label junior-test). On a personal GitHub account there is no read-only role: a collaborator can also push. Ask them to use Issues only. The .githooks guards only run on machines that set them up.

Before the round, for J35: check the label Test personal exists in admin.octopodia@gmail.com (left-hand list). Testers use it; they don't create their own.

4. During the round

Check Where
No failed runs of the three Junior workflows n8n → Executions. Failed runs are kept for 7 days (OI-094)
No rows stuck as pending sort_log, status = pending
New issues GitHub Issues, label junior-test. Ticked "archived" or "got its way" boxes first: either fails the round
A fail a tester logs Find the email by its subject and their address. decided_by, confidence and reason say why. Add them to the issue
J24 and J25 (written in the tester's own words) Run Junior: guardrail review (below). Any row not caught by a guardrail shows the words to consider adding to guardrail_words

The free Gemini tier has a low per-minute limit. If a tester sends ten emails at once, a few may come back pending and sort on the next poll. That is the retry working, not a fail.

Finding a tester's email

Testers record the subject, never the time. Run Junior: guardrail review (below) with that subject and their address, for any scenario, not only J24 and J25. It does what you cannot easily do by hand:

  1. Searches Gmail for subject:"the subject" from:their@address. For a thread, it takes their latest message.
  2. Takes Gmail's message ID from the API. The ID in Gmail's address bar is a different one, so it cannot be copied from the browser.
  3. Looks that message_id up in sort_log.

Guardrail review

Read-only: it changes nothing, labels nothing and logs nothing.

  1. Open Junior: guardrail review in n8n. In its first node, Emails to review, list each row as scenario | subject | sender address, one per line, copied from the round's Project (Scenario, Subject, and the tester's sending address).
  2. Run it. For each email it shows decided_by, the matched word, and FLAG on any J24 or J25 row that was not caught by legal_risk or safety, with the words in it that look worth adding.
  3. Add the words worth having to the guardrail_words table (list legal or safety, phrase, * at the end for a prefix). Then run Junior: sorter tests to check nothing else broke.
  4. Check J28 ("Hi, it's Sue here") did not come out as legal_risk.

5. Close the round

  1. In the Project, check no row is left Not run without a reason. Close the Project (Settings → Close project), which keeps it read-only as the round's record. Save any offline workbooks as docs/testing/junior/results/junior-tests-<name>-<yyyy-mm-dd>.xlsx.
  2. Count against the bar: at least 50 emails in total, no customer-type email archived, no prompt injection worked.
  3. Remove delegation if testing is finished: Settings → Accounts and Import → Grant access to your account → delete next to each tester.
  4. Turn every OWN row and issue worth keeping into a line in scenarios.csv, then rerun python tools/build_junior_workbook.py. Close each issue once fixed, or carry it to the next round.

  5. Clear what testing taught it. Testers' corrections leave learned rules behind (J29, J33, J34 and J35 on purpose, others by accident). In corrections, the round's rows stay as its record: never edit or delete them.

    1. In n8n, open Junior: reset learning and click Execute workflow. The form opens.
    2. mailbox_id: admin.octopodia@gmail.com. dry_run: Yes, dry run only. Submit.
    3. Read the report (the last node, Dry run report): each row it would change, with before and after. Check nothing on the list should keep what it learned.
    4. Run it again with No, change the rows. The report (What changed) lists what it changed and the count, which should match the dry run.

    It changes only that mailbox's known_senders rows, exactly as in the hand method below, never deletes a row, and never touches corrections, sort_log or any other table. An empty or unknown mailbox_id stops it before anything is written.

    Fallback, by hand (if the workflow is unavailable): in n8n, open Data tables, then known_senders. Find the rows for this mailbox (mailbox = admin.octopodia@gmail.com) whose source is learned_action or learned_noise, whose rule is set, or whose correction_count is above 0 (a sender one drag away from being learned as noise). For each one that would skew the next round, edit three cells and nothing else:

    Column Set to
    rule Empty
    source sent_history if last_written_to has a date, otherwise correction
    correction_count 0

    Never delete the row: last_written_to is what makes the mailbox treat someone it has written to as a known sender. This is the same change the assistant makes itself when an opposite correction undoes a rule. Rows under any other mailbox value are test leftovers that no mailbox reads.

    Check: sort by updatedAt and confirm only the rows you meant changed. The next email from that sender is sorted as it would have been before the round.

6. Once: prove a second mailbox

The testers all work in one mailbox, so this is the only test of the multi-mailbox build with a real second mailbox.

  1. Create a second test Gmail, such as junior.test2.octopodia@gmail.com. Store the login in Bitwarden.
  2. Follow Adding a mailbox: create its Gmail credential (gmail.modify only, allowed domain gmail.googleapis.com), duplicate Junior: mailbox (admin.octopodia) as Junior: mailbox (junior.test2), pick the new credential in each of its Gmail nodes (its sticky note lists them), run Run setup, publish it. The Google sign-in shows "Google hasn't verified this app": choose Advanced, then continue. Expected for our unverified test app.
  3. Send one email to each mailbox from the same outside address, a minute apart.
  4. Pass when all of these hold:
Check Expected
Both emails sorted Each labelled in its own mailbox
mailboxes Two rows, each with the label IDs from its own mailbox. They may happen to match, since Gmail numbers labels per mailbox
sort_log Each row carries its own mailbox
known_senders The second mailbox's rows come from its own Sent folder only
The first mailbox Nothing changed in it by the second
  1. Unpublish Junior: mailbox (junior.test2) afterwards, unless it is wanted for later rounds.