Model Evaluation and Comparison
Evaluation targets
- Fitted model versus learning procedure - Performance of one trained model differs from average performance over new training sets. Bates et al. establish this distinction for cross-validation in ordinary least squares. Bates et al. 2023.
- Selection versus assessment - Optimizing a noisy validation criterion can overfit it; evaluating the selected winner on that same criterion is optimistic. Cawley and Talbot find selection effects comparable to algorithm differences. Cawley and Talbot 2010.
Splitting and resampling
- Cross-validation - Holdout, k-fold and leave-one-out trade training-set size, computation and dependence among error estimates. Arlot and Celisse distinguish risk estimation from selecting a model; the best procedure depends on the objective. Arlot and Celisse 2010.
- Nested cross-validation for selection - Tune preprocessing, features and hyperparameters inside each outer training fold; assess the resulting procedure on its outer test fold. Outer results must not drive further selection. Cawley and Talbot 2010.
- Group-aware validation - GroupKFold and leave-one-group-out prevent the same subject, device or group from appearing on both sides of a split. Group choice defines whether evaluation tests new observations or generalization to new groups. docs.
- Rolling-origin evaluation - Repeatedly train on earlier observations and test later horizons. Match forecast horizons to deployment; random folds can expose information unavailable at prediction time. Tashman 2000, Hyndman and Athanasopoulos 2021.
- Stratification - Preserves approximate class proportions across folds. Cross-validation documentation warns that this engineering safeguard can shrink observed fold variability; it does not remove sampling uncertainty. docs.
- Leakage-safe preprocessing - Fit imputation, scaling, selection and learned representations within training splits. Pipelines help enforce this; preprocessing or selecting a pipeline using test outcomes still leaks information. docs.
- .632+ bootstrap - Combines training and out-of-bag error with a weight adjusted for overfitting. Efron and Tibshirani study classification error estimation; this is an error estimator, not a confidence interval. Efron and Tibshirani 1997.
- Data thinning - Under convolution-closed models, splits each observation into independent parts for clustering/PCA validation without holding out rows. Gaussian thinning needs the correct variance/covariance; misspecification can induce dependence. Neufeld et al. 2024.
- Dependent folds - Bengio and Grandvalet prove no universally unbiased estimator of k-fold CV variance exists. Ignoring error correlations can underestimate uncertainty; extra folds are not independent replications. Bengio and Grandvalet 2004.
- Nested CV for uncertainty - Bates et al. estimate prediction-error variance using nested resampling; reported interval coverage improves where naive CV intervals fail. This objective differs from nesting for hyperparameter selection. Bates et al. 2023.
- Small-sample evaluation - Varoquaux shows wide prediction-accuracy uncertainty in small neuroimaging samples. Standard errors across folds can understate it; resampling cannot substitute for collecting independent observations. Varoquaux 2017.
- Training and search variability - Bouthillier et al. find data sampling, initialization and hyperparameter choices materially affect comparisons on five tasks. Repeating one fitted model’s test evaluation misses these sources. Bouthillier et al. 2021.
- Stratified bootstrap intervals - Resample runs within benchmark tasks to quantify uncertainty in aggregate scores. Specify whether tasks themselves are fixed or sampled; the resampling unit determines the uncertainty represented. Agarwal et al. 2021.
- Confidence sequences - Builds intervals valid simultaneously over time, allowing data-dependent stopping. Martingale concentration bounds require stated tail or boundedness assumptions; repeated monitoring costs wider intervals. Howard et al. 2021; code.
Statistical inference using predictions
- Prediction-powered inference - Uses labeled data to correct prediction-based population estimates and intervals. The basic setup needs i.i.d. samples and an independent predictor; the paper’s basic intervals are asymptotic, and weak predictions can widen them. Angelopoulos et al. 2023; code.
Metrics and scoring rules
- Thresholded classification - Precision, recall and F-scores evaluate decisions at a chosen threshold; macro and pooled averaging weight classes differently. Report the threshold-selection procedure. docs.
- ROC versus precision-recall - Davis and Goadrich relate both curve spaces. ROC-AUC optimization need not optimize PR-AUC; precision depends on class prevalence, so comparisons need the class mix. Davis and Goadrich 2006.
- Point-prediction losses - Squared error targets the conditional mean; absolute error targets a conditional median. Choosing a loss specifies the prediction being evaluated, not only its units. docs.
- Proper scoring rules - Brier and logarithmic scores assess predictive probabilities; CRPS assesses predictive distributions. Propriety rewards truthful distributions in expectation. Gneiting and Raftery 2007.
- Interval scores - Combine interval width with penalties for observations outside the interval. Coverage alone rewards arbitrarily wide intervals and misses useful sharpness. Gneiting and Raftery 2007.
- Calibration-error estimation - Binning, sample size and estimator bias affect reported calibration. Kumar et al. develop more accurate estimators and scaling-binning calibration; smooth reliability plots alone are insufficient evidence. Kumar et al. 2019.
Prediction-set coverage and risk control
- Split conformal prediction - Calibrates a fitted predictor’s scores on held-out data to form prediction sets. Exchangeability supports finite-sample marginal coverage; report set size and subgroup coverage; the guarantee is not conditional on every input. Angelopoulos and Bates 2021.
- Conformal risk control - Calibrates a threshold to bound expected loss over calibration and test draws. Requires exchangeable, bounded, non-increasing, right-continuous losses and an achievable target; the guarantee is not per input. Angelopoulos et al. 2024; code.
- Conformal prediction under covariate shift - Weights calibration scores by target-to-training covariate density ratios while holding the conditional response law fixed. Exact known-weight guarantees do not automatically carry over to inaccurately estimated ratios. Tibshirani et al. 2019.
Statistical comparisons
- Permutation tests - Shuffle labels to test class structure, or shuffle features within classes to test exploited dependencies. The permitted shuffles define the tested null hypothesis. Ojala and Garriga 2010.
- Comparisons across datasets - Wilcoxon signed-rank compares two methods; Friedman with post-hoc tests compares several. Datasets are the comparison units; correlated CV folds are not interchangeable with independent datasets. Demšar 2006.
- Bayesian comparisons - Estimate probabilities of superiority, practical equivalence and inferiority under a statistical model. Specify the equivalence region and dependence model; these differ from null-hypothesis p-values. Benavoli et al. 2017.
Group fairness criteria
- Equalized odds and opportunity - Equalized odds matches error rates across groups conditional on the outcome; equal opportunity requires equal true-positive rates. Hardt et al. construct post-processing rules; the criteria use observed labels and group membership. Hardt et al. 2016.
- Conflicting fairness criteria - Chouldechova shows that predictive-value parity and equal false-positive/negative rates conflict at unequal base rates for nondegenerate imperfect classifiers. A fairness target must specify which errors and populations matter. Chouldechova 2016.
Benchmarks, reuse and distribution shift
- Robust benchmark summaries - Interquartile mean and performance profiles expose information hidden by one mean or median. Agarwal et al. show uncertainty-aware analyses change conclusions on studied few-run RL benchmarks. Agarwal et al. 2021.
- Adaptive holdout reuse - Repeated feedback can overfit a test set. Dwork et al. use controlled, randomized access to limit this effect; guarantees belong to the access procedure, not unrestricted repeated leaderboard inspection. Dwork et al. 2015.
- Learning curves - Compare held-out performance against training-set size. Viering and Loog document diverse shapes, including nonmonotonic curves; extrapolating a fitted power law is an additional modeling assumption. Viering and Loog 2021.
- LLM-as-judge evaluation - MT-Bench studies agreement with human preferences alongside position, verbosity and self-preference biases. A model judge approximates judgments in a tested setting; it is not a factual oracle or a universal replacement for human assessment. Zheng et al. 2023.
- Natural distribution shifts - WILDS benchmarks hospital, location, time and other shifts. Standard training and tested shift-handling methods retain substantial out-of-distribution gaps; the split defines the generalization claim. Koh et al. 2021.
- Uncertainty under shift - Ovadia et al. find in-distribution post-hoc calibration insufficient under tested shifts; some model-averaging methods fare better. Evaluate predictive accuracy and calibration separately. Ovadia et al. 2019.
Software
- scikit-learn - Splitters, scoring functions and pipelines support evaluation workflows. The chosen split, search and scorer still determine what is estimated. docs, docs, docs.
- rliable - Implementation of interval estimates, interquartile means, probability of improvement and performance profiles for benchmark score arrays. Run/task structure and score normalization remain explicit inputs to the analysis. code.
- baycomp - Bayesian comparison tools for classifiers on one or multiple datasets; returns probabilities for superiority, practical equivalence and inferiority. The caller supplies the equivalence threshold. docs.
Star
Issue