Automated Research

Program and algorithm discovery

  • AlphaTensor - Reinforcement learning searches tensor decompositions for matrix multiplication. Finds a 4-by-4 algorithm using 47 rather than 49 multiplications over the two-element field; this result does not apply to arbitrary real-valued matrices. Fawzi et al. 2022.
  • AlphaDev - Reinforcement learning builds assembly programs using correctness and latency rewards. Discovered small sorting routines were translated into C++ and adopted in LLVM libc++; the search relies on supplied tests, instructions and hardware objectives. Mankowitz et al. 2023.
  • FunSearch - Uses a frozen LLM to evolve functions inside a supplied program skeleton using executable scores and separate populations. In direct cap-set search, 4 of 140 runs found a size-512 set in eight dimensions; this establishes a lower bound, not optimality. Romera-Paredes et al. 2024.
  • AlphaEvolve - Extends FunSearch to multi-function program evolution with supplied evaluators. Reusing earlier programs beat restarting from the initial program on matrix-decomposition and kissing-number tasks, averaged over three runs. Novikov et al. 2025.
  • Learning-algorithm search - AutoML-Zero evolves learning programs from basic operations. It supplies a non-LLM connection to algorithm discovery; pipeline and hyperparameter search remain on AutoML.

Experiment automation

  • Adam robot scientist - Combines logical hypotheses about yeast enzymes with robotic growth experiments. Humans supplied models and lab infrastructure; subsequent protein assays confirmed three predicted gene-enzyme links. Some rediscoveries came from an incomplete database. King et al. 2009.
  • The AI Scientist - Proposes ideas, edits code, runs experiments and writes papers from an experiment template. Authors report incorrect implementations and changed metrics compared against unchanged baselines, despite automated review. Lu et al. 2024; code.
  • Recursive automated research - Industrial case study of parallel experiment threads for model training and GPU kernels. Released scripts and selected outputs allow inspection, but the complete research controller is not released. Recursive 2026; code.
  • Pipeline automation - MLZero uses data inspection, library knowledge and execution history to revise ML pipelines. Predictive pipeline performance and production of new research knowledge are different evaluation targets.
  • AIRA2 - Combines asynchronous experiment workers with fixed hidden search data and a separate final-selection split. Its ablation removes late-search degradation caused by inconsistent evaluation. Eight-GPU results are not compute-matched to single-GPU baselines. Hambardzumyan et al. 2026.
  • Heuresis - Among six search strategies, Greedy wins on nanoGPT and unlearning; MAP-Elites wins on Breakout. Islands transfers best to held-out Asterix; Greedy’s five elites all fail or time out. Search-task scores do not identify the best transfer strategy. Antoniades et al. 2026.

Improving the researcher

  • Fisher-R1 - On 222 hard P-Bench tasks, SFT plus RL raises Qwen2.5-Coder-14B strict pass@1 from 23.7% to 33.0%, versus GPT-5.4’s 30.5%. Strict scores require a correct decision and near-reference p-value; three evaluation runs. Miao et al. 2026.

Research benchmarks and evidence

  • Human evaluation of research ideas - In a blinded study with over 100 NLP researchers contributing ideas or reviews, LLM ideas received higher novelty ratings but slightly lower feasibility ratings. These were judgments of proposals before execution. Si et al. 2024.
  • MLAgentBench - Provides 13 ML experimentation tasks with file editing, execution and output inspection. Evaluates baseline improvement and time/token costs. Its ReAct-based agent tracks research plans, status and recent actions. Huang et al. 2024; code.
  • Ideation-execution gap - 43 experts executed NLP research ideas. AI ideas lost more review score after execution; with projects as independent units, post-execution human-AI differences were not significant. Proposal novelty did not establish better research outcomes. Si et al. 2025.
  • RE-Bench - Seven ML research-engineering tasks compare agents with 61 experts. Best agents lead at two total hours; humans lead at eight and 32. Larger-budget estimates use best-of-k attempts, including four eight-hour human runs at 32 hours. Wijk et al. 2025.
  • CORE-Bench - 270 tasks from 90 papers separate reading supplied outputs, executing code in a supplied environment, and setting up that environment. Tests reproducibility using authors’ code and data, unlike PaperBench’s implementation-from-paper task. Siegel et al. 2025.
  • PaperBench - Tests replication of 20 ICML papers using author-developed rubrics, fresh code execution and an LLM judge. Scores measure weighted rubric completion. For o1, code-only scores correlate weakly with full replication scores. Starace et al. 2025.
  • ML engineering benchmarks - MLE-bench tests agents on ML competitions. Competition scores measure task performance; PaperBench instead checks replication of specified research results. Neither alone establishes novelty.
  • MLS-Bench - Tests 140 tasks across 12 domains with fixed editable components, capacity limits and multiple evaluation settings. Agents usually trail reproduced human methods even with their code in context; removing capacity limits lets some scores rise through larger models. Lyu et al. 2026.
  • Unsatz - Agents buy experiments and infer hidden variables and equations in 80 simulated systems. A supplied fitter estimates coefficients. Nine LLMs solve 21-60% on unseen interventions, longer horizons and conditions outside the explored range. Ji et al. 2026.

Software

  • autoresearch - Agents edit train.py and retain changes by validation bits per byte. prepare.py fixes data/evaluation; humans edit program.md. Five minutes is a per-run training budget, excluding startup and compilation; total search costs accumulate across runs. code.
  • FunSearch implementation - Releases discovered programs, evaluation notebooks and a single-threaded search pipeline. Users must supply the language model and execution sandbox; the original distributed search infrastructure is not included. code.