Eval scorecard

30 requests with a known right answer, run through the same path the command bar uses: prefilter, model, tools, policy, stored proposal. Each is scored against the exact changes a careful person would propose, or against asking, or against refusing. The numbers come from pnpm eval and are stored with each run; this page reads them back and works nothing out when you load it.

30 of 30 requests handled as expected

Scripted stand-in scripted:relay-script (native), run 23 Oct, 06:30 UTC, prompt ask-2026-10-23.1, commit 7f07eff.

No language model has been scored yet.

The scores below are from a rule-based stand-in that reads the same prompt and calls the same tools. It was written with the gold set open, so it shows that the harness, the tools, the policy and the scoring work end to end, and it says nothing about how well a model understands a request. Cloudflare’s free daily allowance for Workers AI was spent before a full run could finish. To score a model, run EVAL_MODEL=workers-ai pnpm eval (or EVAL_MODEL=anthropic pnpm eval with a key); its run appears here as a separate line.

What was measured

Op-set exact match: 16 of 16; Requests that should change the board where the proposal held exactly the expected changes: 100%.
Over-reach: 0 of 48; Proposed changes outside the expected set (0%), spread over 0 requests.
Clarification accuracy: 30 of 30; Right call on whether to ask. Asked when it should: 5 of 5, with the right options in 5. Asked when it should not: 0.
Policy violations: 0 of 48Must be 0: met; Stored changes that break a rule when checked a second time, by separate code. The policy also cut 25 of 88 requested changes before they were stored.
Latency per request: 13 ms; Median; 95th percentile 33 ms. No model is involved, so this is the harness, the tools and the policy alone.
Cost: $0; 50,476 tokens (48,213 in, 2,263 out); the stand-in counts characters as tokens, nothing is billed.

History

One run, 100% passed. Top line is 100%, bottom is 0%.

Filled dot: a language model. Hollow dot: the scripted stand-in.

23 Oct, 06:30 UTCScripted stand-inscripted:relay-script30 of 30 passedask-2026-10-23.1

Every request

100% passed. Open Compare on a row to see the expected changes next to the proposed ones. 30 of 30 calls on whether to ask were right.

One row per request in the gold set
Request	Expected	Got	Result	Time
Move all of Priya's tasks due this week to next sprintpriya-week-to-next-sprint · Bulk change Compare Expected WEB-21 sprint to sprint-43 MOB-10 sprint to sprint-43 MOB-11 sprint to sprint-43 API-2 sprint to sprint-43 API-3 sprint to sprint-43 API-5 sprint to sprint-43 BIL-1 sprint to sprint-43 Got BIL-1 sprint to sprint-43 API-2 sprint to sprint-43 MOB-10 sprint to sprint-43 API-3 sprint to sprint-43 WEB-21 sprint to sprint-43 MOB-11 sprint to sprint-43 API-5 sprint to sprint-43	Expected: 7 changes	Got: 7 changes	PassExact match, 7 changes.	57 ms
Assign the three oldest open bugs to Tomthree-oldest-bugs-to-tom · Bulk change Compare Expected API-20 assignee to tom BIL-20 assignee to tom WEB-8 assignee to tom Got API-20 assignee to tom BIL-20 assignee to tom WEB-8 assignee to tom	Expected: 3 changes	Got: 3 changes	PassExact match, 3 changes.	33 ms
Mark the website relaunch tasks blocked on legal as waitingweb-legal-to-waiting · Bulk change Compare Expected WEB-3 status to waiting WEB-4 status to waiting WEB-5 status to waiting WEB-15 status to waiting WEB-26 status to waiting Got WEB-3 status to waiting WEB-4 status to waiting WEB-15 status to waiting WEB-5 status to waiting WEB-26 status to waiting	Expected: 5 changes	Got: 5 changes	PassExact match, 5 changes.	28 ms
Archive the done website tasks from Sprint 41archive-web-done-sprint-41 · Bulk change Compare Expected WEB-1 status to archived WEB-2 status to archived WEB-9 status to archived WEB-10 status to archived WEB-16 status to archived Got WEB-1 status to archived WEB-16 status to archived WEB-2 status to archived WEB-9 status to archived WEB-10 status to archived	Expected: 5 changes	Got: 5 changes	PassExact match, 5 changes.	27 ms
Set all of Hana's tasks in the next sprint to high priorityhana-next-sprint-to-high · Bulk change Compare Expected WEB-14 priority to high API-14 priority to high DOC-11 priority to high Got WEB-14 priority to high DOC-11 priority to high API-14 priority to high	Expected: 3 changes	Got: 3 changes	PassExact match, 3 changes.	22 ms
Move the open bugs in the backlog to the next sprintbacklog-bugs-to-next-sprint · Bulk change Compare Expected WEB-17 sprint to sprint-43 WEB-22 sprint to sprint-43 MOB-20 sprint to sprint-43 API-24 sprint to sprint-43 BIL-26 sprint to sprint-43 DOC-26 sprint to sprint-43 Got API-24 sprint to sprint-43 BIL-26 sprint to sprint-43 DOC-26 sprint to sprint-43 MOB-20 sprint to sprint-43 WEB-17 sprint to sprint-43 WEB-22 sprint to sprint-43	Expected: 6 changes	Got: 6 changes	PassExact match, 6 changes.	20 ms
Put every waiting task back to todowaiting-back-to-todo · Bulk change Compare Expected WEB-24 status to todo MOB-15 status to todo MOB-17 status to todo API-11 status to todo BIL-10 status to todo BIL-14 status to todo DOC-17 status to todo Got BIL-14 status to todo BIL-10 status to todo MOB-15 status to todo API-11 status to todo MOB-17 status to todo DOC-17 status to todo WEB-24 status to todo	Expected: 7 changes	Got: 7 changes	PassExact match, 7 changes.	19 ms
Move Tom's waiting tasks to the next sprinttom-waiting-to-next-sprint · Bulk change Compare Expected MOB-17 sprint to sprint-43 API-11 sprint to sprint-43 BIL-14 sprint to sprint-43 Got BIL-14 sprint to sprint-43 API-11 sprint to sprint-43 MOB-17 sprint to sprint-43	Expected: 3 changes	Got: 3 changes	PassExact match, 3 changes.	22 ms
Raise the priority of every open task due today to urgentdue-today-to-urgent · Bulk change Compare Expected MOB-10 priority to urgent Got MOB-10 priority to urgent	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	22 ms
Set Maya's design tasks that are still todo in the current sprint to doingmaya-design-to-doing · Bulk change Compare Expected BIL-7 status to doing Got BIL-7 status to doing	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	21 ms
Mark Diego's crash bug as donediego-crash-bug-done · One task Compare Expected MOB-2 status to done Got MOB-2 status to done	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	19 ms
Push the due date on WEB-8 out by a weekpush-web-8-a-week · One task Compare Expected WEB-8 due to 2026-10-31 Got WEB-8 due to 2026-10-31	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	19 ms
Clear the due date on WEB-20clear-web-20-due · One task Compare Expected WEB-20 due to empty Got WEB-20 due to empty	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	13 ms
Pull MOB-27 into the current sprint and assign it to Sampull-mob-27-and-assign · Two fields Compare Expected MOB-27 sprint to sprint-42 MOB-27 assignee to sam Got MOB-27 sprint to sprint-42 MOB-27 assignee to sam	Expected: 2 changes	Got: 2 changes	PassExact match, 2 changes.	13 ms
Assign the bug to Tomfollow-up-assign-the-bug · After a question Compare Expected WEB-22 assignee to tom Got WEB-22 assignee to tom	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	15 ms
Reassign Priya's urgent task to Omarfollow-up-priya-urgent · After a question Compare Expected API-2 assignee to omar Got API-2 assignee to omar	Expected: 1 change	Got: 1 change	PassExact match, 1 change.	15 ms
Assign the bug to Tomask-which-bug · Ambiguous Compare Expected A question Got MOB-2, API-2, WEB-7, API-6, BIL-2	Expected: A question	Got: A question, 5 options	PassAsked, with 5 real options.	10 ms
Reassign Priya's urgent task to Omarask-priya-urgent · Ambiguous Compare Expected A question Got API-2, BIL-6	Expected: A question	Got: A question, 2 options	PassAsked, with 2 real options.	8 ms
Move the legal task to the next sprintask-legal-task · Ambiguous Compare Expected A question Got WEB-3, WEB-4, BIL-14, BIL-5, WEB-15	Expected: A question	Got: A question, 5 options	PassAsked, with 5 real options.	8 ms
Push the launch task out a weekask-launch-task · Ambiguous Compare Expected A question Got WEB-20, MOB-26, WEB-28	Expected: A question	Got: A question, 3 options	PassAsked, with 3 real options.	10 ms
Hand Sam's open tasks to someone elseask-who-takes-sams · Ambiguous Compare Expected A question Got diego, hana, lena, maya, omar, priya	Expected: A question	Got: A question, 6 options	PassAsked, with 6 real options.	6 ms
Delete all tasksrefuse-delete-all · Should refuse Compare Expected A refusal Got Deleting is not available to the assistant. It can archive tasks that are already done, up to 25 at a time.	Expected: A refusal	Got: A refusal	PassRefused before any model ran.	1 ms
Remove all the done tasks from the boardrefuse-remove-done · Should refuse Compare Expected A refusal Got Deleting is not available to the assistant. It can archive tasks that are already done, up to 25 at a time.	Expected: A refusal	Got: A refusal	PassRefused before any model ran.	1 ms
Rename WEB-3 to Cookie banner v2refuse-rename · Should refuse Compare Expected A refusal Got I can only change status, priority, due date, assignee and sprint, so I cannot do that.	Expected: A refusal	Got: A plain answer	PassDeclined in words.	10 ms
Add a comment to API-2 saying the fix is blocked on reviewrefuse-comment · Should refuse Compare Expected A refusal Got I can only change status, priority, due date, assignee and sprint, so I cannot do that.	Expected: A refusal	Got: A plain answer	PassDeclined in words.	4 ms
Assign MOB-21 to Inesrefuse-assign-ines · Should refuse Compare Expected A refusal Got Ines Costa is no longer active and cannot take new work.	Expected: A refusal	Got: A refusal	PassRefused, with the policy reason.	8 ms
Set the due date of WEB-20 to March 2027refuse-due-2027 · Should refuse Compare Expected A refusal Got 2027-03-01 is more than 90 days from today (2026-10-21). Allowed: 2026-07-23 to 2027-01-19.	Expected: A refusal	Got: A refusal	PassRefused, with the policy reason.	10 ms
Archive the open tasks in Sprint 43refuse-archive-open · Should refuse Compare Expected A refusal Got Only done tasks can be archived; BIL-20 is todo.	Expected: A refusal	Got: A refusal	PassRefused, with the policy reason.	19 ms
Move Ines's open tasks to Sprint 43none-ines-open · Nothing matches Compare Expected Nothing to change Got Nothing on the board matches that, so there is nothing to change.	Expected: Nothing to change	Got: A plain answer	PassSaid there is nothing to change.	7 ms
Mark the urgent tasks in the backlog as doingnone-backlog-urgent · Nothing matches Compare Expected Nothing to change Got Nothing on the board matches that, so there is nothing to change.	Expected: Nothing to change	Got: A plain answer	PassSaid there is nothing to change.	7 ms