Model Evaluation and Comparison

Evaluation targets

  • Fitted model versus learning procedure - Performance of one trained model differs from average performance over new training sets. Bates et al. establish this distinction for cross-validation in ordinary least squares. Bates et al. 2023.
  • Selection versus assessment - Optimizing a noisy validation criterion can overfit it; evaluating the selected winner on that same criterion is optimistic. Cawley and Talbot find selection effects comparable to algorithm differences. Cawley and Talbot 2010.

Splitting and resampling

  • .632+ bootstrap - Combines training and out-of-bag error with a weight adjusted for overfitting. Efron and Tibshirani study classification error estimation; this is an error estimator, not a confidence interval. Efron and Tibshirani 1997.
  • Rolling-origin evaluation - Repeatedly train on earlier observations and test later horizons. Match forecast horizons to deployment; random folds can expose information unavailable at prediction time. Tashman 2000, Hyndman and Athanasopoulos 2021.
  • Group-aware validation - GroupKFold and leave-one-group-out prevent the same subject, device or group from appearing on both sides of a split. Group choice defines whether evaluation tests new observations or generalization to new groups. docs.
    • Clonal pseudoreplication - Shared clones make null perturbations look reproducible in random cell halves. In one T-cell donor, STATE’s effect correlation is 0.26 versus a 0.08 baseline, then below baseline across transductions. Clone effects are smaller in tested cell lines. Huang 2026.
  • Cross-validation - Holdout, k-fold and leave-one-out trade training-set size, computation and dependence among error estimates. Arlot and Celisse distinguish risk estimation from selecting a model; the best procedure depends on the objective. Arlot and Celisse 2010.
  • Stratification - Preserves approximate class proportions across folds. Cross-validation documentation warns that this engineering safeguard can shrink observed fold variability; it does not remove sampling uncertainty. docs.
  • Leakage-safe preprocessing - Fit imputation, scaling, selection and learned representations within training splits. Pipelines help enforce this; preprocessing or selecting a pipeline using test outcomes still leaks information. docs.
  • Nested cross-validation for selection - Tune preprocessing, features and hyperparameters inside each outer training fold; assess the resulting procedure on its outer test fold. Outer results must not drive further selection. Cawley and Talbot 2010.
  • Data thinning - Under convolution-closed models, splits each observation into independent parts for clustering/PCA validation without holding out rows. Gaussian thinning needs the correct variance/covariance; misspecification can induce dependence. Neufeld et al. 2024.

Uncertainty in reported performance

  • Dependent folds - Bengio and Grandvalet prove no universally unbiased estimator of k-fold CV variance exists. Ignoring error correlations can underestimate uncertainty; extra folds are not independent replications. Bengio and Grandvalet 2004.
  • Small-sample evaluation - Varoquaux shows wide prediction-accuracy uncertainty in small neuroimaging samples. Standard errors across folds can understate it; resampling cannot substitute for collecting independent observations. Varoquaux 2018.
  • Training and search variability - Bouthillier et al. find data sampling, initialization and hyperparameter choices materially affect comparisons on five tasks. Repeating one fitted model’s test evaluation misses these sources. Bouthillier et al. 2021.
  • Stratified bootstrap intervals - Resample runs within benchmark tasks to quantify uncertainty in aggregate scores. Specify whether tasks themselves are fixed or sampled; the resampling unit determines the uncertainty represented. Agarwal et al. 2021.
  • Confidence sequences - Builds intervals valid simultaneously over time, allowing data-dependent stopping. Martingale concentration bounds require stated tail or boundedness assumptions; repeated monitoring costs wider intervals. Howard et al. 2021; code.
  • Nested CV for uncertainty - Bates et al. estimate prediction-error variance using nested resampling; reported interval coverage improves where naive CV intervals fail. This objective differs from nesting for hyperparameter selection. Bates et al. 2023.
  • Post-collection selective inference - Conditions on both adaptive query choices and a later chosen contrast. Gives selective p-values under Gaussian noise with known variance; the implemented search requires tractable selection events, illustrated with TPE and GP-UCB. Nishino et al. 2026.

Statistical inference using predictions

  • Prediction-powered inference - Uses labeled data to correct prediction-based population estimates and intervals. The basic setup needs i.i.d. samples and an independent predictor; the paper’s basic intervals are asymptotic, and weak predictions can widen them. Angelopoulos et al. 2023; code.
    • PPI++ - Tunes the weight of prediction-based corrections, with zero weight recovering labeled-only inference. Optimizes asymptotic variance under regularity assumptions; this is not a finite-sample improvement guarantee. Angelopoulos et al. 2026.
    • Finite-sample prediction assistance - In mean estimation, learning the PPI++ correction weight adds variance. Weakly correlated predictions can worsen error versus labeled-only estimation, even with arbitrarily many unlabeled samples; the threshold depends on label count. Mani et al. 2026.
    • Calibrated prediction-powered inference - Calibrates model scores on labeled data before pooling predictions for mean estimation. Linear calibration matches PPI++ at first order; isotonic calibration optimizes within monotone score transformations under regularity assumptions. van der Laan and van der Laan 2026.

Metrics and scoring rules

  • Thresholded classification - Precision, recall and F-scores evaluate decisions at a chosen threshold; macro and pooled averaging weight classes differently. Report the threshold-selection procedure. docs.
  • ROC versus precision-recall - Davis and Goadrich relate both curve spaces. ROC-AUC optimization need not optimize PR-AUC; precision depends on class prevalence, so comparisons need the class mix. Davis and Goadrich 2006.
  • Point-prediction losses - Squared error targets the conditional mean; absolute error targets a conditional median. Choosing a loss specifies the prediction being evaluated, not only its units. docs.
  • Proper scoring rules - Brier and logarithmic scores assess predictive probabilities; CRPS assesses predictive distributions. Propriety rewards truthful distributions in expectation. Gneiting and Raftery 2007.
  • Interval scores - Combine interval width with penalties for observations outside the interval. Coverage alone rewards arbitrarily wide intervals and misses useful sharpness. Gneiting and Raftery 2007.
  • Calibration-error estimation - Binning, sample size and estimator bias affect reported calibration. Kumar et al. develop more accurate estimators and scaling-binning calibration; smooth reliability plots alone are insufficient evidence. Kumar et al. 2019.

  • Reject option and selective prediction - Chow derives a Bayes-optimal error-reject trade-off from true class posteriors. Geifman-El-Yaniv bounds selective risk under IID calibration. Coverage is the acceptance rate; risk is error among accepted predictions. Chow 1957, Chow 1970, Geifman and El-Yaniv 2017.

Predictive calibration

  • Multiclass probability calibration - Transforms classifier scores into multiclass probability estimates. Zadrozny and Elkan 2002.
  • Predicting good probabilities - Compares distortions in classifier probabilities and correction by Platt scaling or isotonic regression; calibration method and calibration-set size affect results. Niculescu-Mizil and Caruana 2005.
  • Nearly-isotonic regression - Relaxes monotonicity through a convex penalty and computes a path of fits; intermediate solutions help examine whether strict monotonicity is appropriate. Tibshirani et al. 2011.
    • Ensembles of calibration models - Studies ensembles of near-isotonic regression for binary probability calibration and compares them with linear-trend calibration; relaxes strict isotonicity. Pakdaman Naeini and Cooper 2017.
  • Beta calibration - Introduces a parametric calibration family for binary classifiers that includes the identity map and accommodates skewed score distributions. Kull et al. 2017.
  • Temperature scaling - Evaluates modern neural-network calibration and a single-parameter post-processing baseline; effectiveness on the evaluated datasets does not guarantee calibration after distribution shift. Guo et al. 2017.
  • CRUDE - Calibrates regression uncertainty using empirical errors; assumes a common residual-distribution shape after shifting by predicted means and scaling by predicted standard deviations. Zelikman et al. 2020.
  • Label-free calibration transfer - Projects predictions toward compatibility with a calibrated reference using unlabeled paired outputs. Proper-loss improvement requires a calibrated reference and failure of mutual calibration; reference errors can offset the gain. Kong et al. 2026.

Prediction-set coverage and risk control

  • Conformal prediction under covariate shift - Weights calibration scores by target-to-training covariate density ratios while holding the conditional response law fixed. Exact known-weight guarantees do not automatically carry over to inaccurately estimated ratios. Tibshirani et al. 2019.
  • Split conformal prediction - Calibrates a fitted predictor’s scores on held-out data to form prediction sets. Exchangeability supports finite-sample marginal coverage; report set size and subgroup coverage; the guarantee is not conditional on every input. Angelopoulos and Bates 2023.
  • Conformal risk control - Calibrates a threshold to bound expected loss over calibration and test draws. Requires exchangeable, bounded, non-increasing, right-continuous losses and an achievable target; the guarantee is not per input. Angelopoulos et al. 2024; code.
    • Anytime-valid conformal risk control - Bounds calibration-conditional risk with high probability simultaneously as IID calibration data accumulate. Requires a fixed nested set family and bounded, monotone, right-continuous loss; does not cover arbitrary online model updates. Hultberg et al. 2026.

Statistical comparisons

  • Comparisons across datasets - Wilcoxon signed-rank compares two methods; Friedman with post-hoc tests compares several. Datasets are the comparison units; correlated CV folds are not interchangeable with independent datasets. Demšar 2006.
  • Permutation tests - Shuffle labels to test class structure, or shuffle features within classes to test exploited dependencies. The permitted shuffles define the tested null hypothesis. Ojala and Garriga 2010.
  • Bayesian comparisons - Estimate probabilities of superiority, practical equivalence and inferiority under a statistical model. Specify the equivalence region and dependence model; these differ from null-hypothesis p-values. Benavoli et al. 2017.

Group fairness criteria

  • Equalized odds and opportunity - Equalized odds matches error rates across groups conditional on the outcome; equal opportunity requires equal true-positive rates. Hardt et al. construct post-processing rules; the criteria use observed labels and group membership. Hardt et al. 2016.
  • Conflicting fairness criteria - Chouldechova shows that predictive-value parity and equal false-positive/negative rates conflict at unequal base rates for nondegenerate imperfect classifiers. A fairness target must specify which errors and populations matter. Chouldechova 2017.

Benchmarks, reuse and distribution shift

  • Adaptive holdout reuse - Repeated feedback can overfit a test set. Dwork et al. use controlled, randomized access to limit this effect; guarantees belong to the access procedure, not unrestricted repeated leaderboard inspection. Dwork et al. 2015.
  • Uncertainty under shift - Ovadia et al. find in-distribution post-hoc calibration insufficient under tested shifts; some model-averaging methods fare better. Evaluate predictive accuracy and calibration separately. Ovadia et al. 2019.
  • Robust benchmark summaries - Interquartile mean and performance profiles expose information hidden by one mean or median. Agarwal et al. show uncertainty-aware analyses change conclusions on studied few-run RL benchmarks. Agarwal et al. 2021.
  • Natural distribution shifts - WILDS benchmarks hospital, location, time and other shifts. Standard training and tested shift-handling methods retain substantial out-of-distribution gaps; the split defines the generalization claim. Koh et al. 2021.
  • Learning curves - Compare held-out performance against training-set size. Viering and Loog document diverse shapes, including nonmonotonic curves; extrapolating a fitted power law is an additional modeling assumption. Viering and Loog 2023.
  • LLM-as-judge evaluation - MT-Bench studies agreement with human preferences alongside position, verbosity and self-preference biases. A model judge approximates judgments in a tested setting; it is not a factual oracle or a universal replacement for human assessment. Zheng et al. 2023.
  • AgentAbstain - 263 paired tasks test both acting and abstaining. Best paired accuracy is 59.5% across 17 agents. One rollout per task; failed runs are excluded, and GPT-5.4 judges verbal responses. The score does not assess full task success. Liu et al. 2026.
  • Leaderboard manipulation - Offline attacks on LMArena data change Bradley-Terry rankings through strategic votes, added model copies and prompt selection. Unchanged models can move when competitors change; these are simulated attacks, not demonstrated live compromise. Fleisig et al. 2026.
  • Select-LLM - Chooses queries to reduce uncertainty about the best model. Across 23 datasets, needs fewer reference labels than adapted active-selection baselines. Uses precomputed model outputs and reliable references; savings exclude model-inference cost. Durmazkeser et al. 2026.

  • Verbalized evaluation awareness - Across eight open models and four benchmarks, inserting awareness sentences mid-trace or removing them changes answers little. Removal excludes unsuccessful edits; this tests verbalized awareness, not hidden evaluation detection or gaming. Knecht et al. 2026.

Software

  • scikit-learn - Splitters, scoring functions and pipelines support evaluation workflows. The chosen split, search and scorer still determine what is estimated. docs, docs, docs.
  • rliable - Implementation of interval estimates, interquartile means, probability of improvement and performance profiles for benchmark score arrays. Run/task structure and score normalization remain explicit inputs to the analysis. code.
  • baycomp - Bayesian comparison tools for classifiers on one or multiple datasets; returns probabilities for superiority, practical equivalence and inferiority. The caller supplies the equivalence threshold. docs.