2 · Customer + ground truth
The agent interprets natural language, but customer identity and system-of-record facts are not guessed.
Customer
Order
Order value
Policy ground truth
3 · Compare Agent A vs Agent B
The proposal is held constant. Only the execution architecture changes.
Agent A · Direct execution
Model proposal → terminal tool executes immediately.
Agent B · Gated execution
Same model proposal → deterministic policy/authority check → execute or block/escalate.
Select a repeated run:
Agent A
Agent B
4 · Scenario-level eval
Reasoning quality and execution safety are measured separately.
Agent A and Agent B have identical proposal accuracy because the model proposal is identical. The gate changes consequences, not intelligence.
Proposal accuracy
Unsafe proposals
A unsafe executions
B unsafe executions
Repeated-run reliability
Agent A release decision
Agent B release decision
5 · Experiment summary across all cases
The release gate looks beyond average accuracy to the distribution and consequence of failures.
| Measure | Agent A | Agent B | Interpretation |
|---|