What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks

Abstract

Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt the lenses of convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, which we use to analyze 56 capability and safety benchmarks using 53 models. We assign benchmarks with substantively similar \textit{purported concepts} to a shared \textit{assigned concept}, and ask whether model rankings on benchmarks with the same assigned concept correlate strongly with one another, and whether model rankings on benchmarks with different assigned concepts correlate less strongly. We ask analogous questions using model scores at the item-level, drawing on item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting that these assigned concepts may be conceptualized inconsistently across benchmarks. For benchmarks with assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting that different assigned capability concepts may not discriminate that well from one another. In some cases, benchmarks that share benchmark design elements (e.g., task structure, score format) correlate more strongly with one another than benchmarks with the same assigned concept. Finally, model rankings on some individual benchmarks correlate more strongly with model rankings on benchmarks with a different assigned concept than with model rankings on benchmarks sharing their own assigned concept, suggesting that these benchmarks may measure a different concept than they purport to. For example, model rankings on BBQ-accuracy correlate more strongly with model rankings on benchmarks assigned with reasoning than with model rankings on benchmarks that share its assigned concept, bias. To support future empirical work on the validity of benchmarks, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.