FrontierML

One desirable property of human-run frontier AI labs is that they can consistently improve their models with increased access to capital. As agents begin doing machine learning work, the same question applies: given more money, can an agent build a better model?

FrontierML is a benchmark that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks. Our benchmark makes several key advancements in capturing the real-world complexity of modern machine learning:

• Scaling spending budgets: Existing benchmarks scale how much agents can think. We scale how much they can spend. We give agents $10, $50, and $200 per task to spend on training compute and data, and evaluate whether agents can use the additional capital productively.
• Open-ended, end-to-end tasks: FrontierML is the most open-ended benchmark for agentic machine learning. In each task, agents can choose architectures, source data, implement training, provision compute, evaluate checkpoints, and optimize inference. Each task is scored on a continuous scale based on the final performance of the submitted model.
• Robust verification: Agents submit an inference server that serves their model on an air-gapped H100 machine. Submissions are evaluated on held-out out-of-distribution tests, and a separate reviewer agent examines each submission and its trajectory for violations of task rules.
• Diverse and real-world focus: FrontierML contains 21 long-horizon tasks inspired by real work at frontier labs, neolabs, and academic institutions.