Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation

Abstract

LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust agent judges to sufficiently cover and reliably score the failure modes of conversational AI systems? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce score differences from human raters that fall within the range of human–human disagreement. We then identify a score–coverage dissociation: scoring reliability improves logarithmically with panel size, while unique issue discoveries follow a sublinear power law—both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A full-scale control with 32 identical agents without persona conditioning reproduces both scaling shapes, indicating that they are general to stochastic multi-session evaluation; in a separate controlled ablation, structured persona conditioning reduces score SD by 46% and produces 47% more insights per session.