Portfolio experiment · Agent reliability

Would you ship this AI support agent?

Compare two execution architectures on the same customer cases and the same repeated model proposals. Inspect tool trajectories, failure severity, and whether a deterministic authority boundary changes what is safe to release.
Current mode: illustrative replay. The customer cases, policies and order data are synthetic. The proposal traces on this page are sample runs built to demonstrate the eval product. In the next phase, this same UI will consume recorded outputs from real LLM executions.

2 · Customer + ground truth

The agent interprets natural language, but customer identity and system-of-record facts are not guessed.
Customer
Order
Order value
Policy ground truth

3 · Compare Agent A vs Agent B

The proposal is held constant. Only the execution architecture changes.
Agent A · Direct execution
Model proposal → terminal tool executes immediately.
Agent B · Gated execution
Same model proposal → deterministic policy/authority check → execute or block/escalate.
Select a repeated run:
Agent A
Agent B

4 · Scenario-level eval

Reasoning quality and execution safety are measured separately.
Agent A and Agent B have identical proposal accuracy because the model proposal is identical. The gate changes consequences, not intelligence.
Proposal accuracy
Unsafe proposals
A unsafe executions
B unsafe executions
Repeated-run reliability
Agent A release decision
Agent B release decision

5 · Experiment summary across all cases

The release gate looks beyond average accuracy to the distribution and consequence of failures.
MeasureAgent AAgent BInterpretation