Construct validity

A new review of 445 LLM benchmarks raises concerns about their validity and the reliance of enterprises on potentially misleading data for AI investment decisions. The study highlights weaknesses in benchmark design, including vague definitions, lack of statistical rigor, data contamination, and unrepresentative datasets. It urges businesses to prioritize internal, domain-specific evaluations over public benchmarks, focusing on custom metrics, thorough error analysis, and clear definitions relevant to their unique needs to mitigate financial and reputational risks.