Modeling

Different models can fit the same observations yet imply different mechanisms or predictions. This review examines how observation assumptions, imposed structure and learned components affect what a model can explain or predict. Forecasting, ensembles and causal effects have separate reviews.

🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked.

Modeling goals

  • The two cultures - Breiman contrasts assumed stochastic data models with algorithms assessed by predictive accuracy. He argues for testing predictions and using a wider range of models instead of relying on goodness of fit within an assumed family. Breiman 2001.
  • Explanation and prediction - Shmueli shows how explanatory and predictive goals change variable selection, fitting and validation. An underspecified model can predict better than a correctly specified one. Predictive accuracy does not establish a causal explanation. Shmueli 2010.

Observation models

Linear and generalized linear models

  • Least squares and ridge - Least squares minimizes squared prediction errors; ridge adds a squared-coefficient penalty to stabilize fits with correlated predictors. docs.
  • Generalized linear models - Connects an exponential-family response mean to a linear predictor through a link function. Unifies models fitted by iteratively weighted linear regression. Nelder and Wedderburn 1972.
  • Generalized linear mixed models - Adds random effects to GLMs for correlated observations and overdispersion. Breslow-Clayton develops approximate inference using penalized quasi-likelihood. Breslow and Clayton 1993.
  • Lasso - Constrains the sum of absolute regression coefficients, shrinking some exactly to zero. Combines estimation and variable selection. Tibshirani 1996.
  • Logistic regression - Models binary class probability through a logistic link applied to a linear score. The documented implementation also supports multiclass classification and regularization. docs.
  • Elastic net - Combines L1 and L2 penalties, allowing sparse coefficients while retaining ridge-style shrinkage. A useful alternative to lasso for correlated predictors. Zou and Hastie 2005; docs.
  • Tests as linear models - Worked connections between regression, t-tests, correlation and ANOVA. Rank-based analogues are also discussed; some equivalences are approximate. Lindeløv 2019.

Survival analysis

  • Kaplan-Meier - Estimates a survival curve from incomplete lifetime observations using a product-limit estimator. Independent censoring is a substantive assumption, explicitly discussed in the paper. Kaplan and Meier 1958.
  • Cox regression - Models covariate effects as multipliers of an unspecified baseline hazard. Handles censored event times without choosing a parametric baseline distribution. Cox 1972.
    • DeepSurv - Uses a neural network for the risk function in a Cox proportional-hazards model, allowing nonlinear covariate interactions in survival prediction. Katzman et al. 2018.
  • Distributional ensembles - See Ensemble Learning for quantile regression forests and NGBoost.
  • Treatment effects - See Causal Inference for heterogeneous effects and meta-learners.

Flexible predictors

Additive models and adaptive splines

  • Generalized additive models - Replaces a GLM’s linear predictor terms with smooth functions of individual inputs. Local scoring extends additive regression to different response distributions. Hastie and Tibshirani 1986.
  • MARS - Builds regression models from products of spline basis functions, selecting their number, variables and knot locations from data. Separates additive contributions from interactions. Friedman 1991.
  • Intelligible classification and regression - Studies additive models whose individual feature contributions can be plotted, comparing fitting methods and the accuracy cost of restricting interactions. Lou et al. 2012.
  • GAMs with autoregressive terms - Combines nonlinear covariate terms with autoregression for serially correlated outcomes. Demonstrated on temperature and daily mortality. Yang et al. 2012.
  • GA2M - Extends additive models with selected pairwise interactions. FAST ranks candidate feature pairs, keeping explanations as one- and two-dimensional functions. Lou et al. 2013.
  • Intelligible healthcare models - Pneumonia and readmission case studies use additive models with pairwise interactions to expose unexpected learned associations and revise models. Caruana et al. 2015.
  • Neural Additive Models - Jointly trains a separate neural network for each input feature, then combines their outputs linearly. Learns flexible univariate shapes while preserving additive structure. Agarwal et al. 2021.
  • How trustworthy are GAMs? - Shows that similarly accurate GAMs can learn conflicting feature shapes. Compares inductive biases, sparsity and behavior on rare subpopulations; accuracy alone does not validate an explanation. Chang et al. 2021.

Trees and tabular prediction

  • CART - Recursively partitions feature space into regions with simple predictions. Depth and pruning control complexity; individual trees can be unstable under changes to training data. docs.
  • Bayesian additive regression trees - Represents a response as a sum of trees with a regularization prior that keeps individual trees weak. Bayesian backfitting samples the model and its predictive uncertainty. Chipman et al. 2010.
  • TabPFN: original classifier - Pretrains a transformer on synthetic classification tasks. At inference, labeled training rows provide context for predictions without updating the model weights; the original scope is small tabular datasets. Hollmann et al. 2023.
    • TabPFN: classification and regression - Extends prior-data-fitted tabular prediction to regression, categorical inputs and missing values. A pretrained transformer performs in-context prediction from labeled examples. Hollmann et al. 2025.
  • TabICL - Combines representations of table rows with a transformer for in-context classification. Synthetic-task pretraining supports prediction on new tables without task-specific weight updates. Qu et al. 2025.
  • Repeated measurements - At low noise, repeats can harm new-input prediction. IID priors linking inputs and responses count repeats as fresh input evidence even with exact inference; TabPFN-v3’s row-count-dependent attention adds harm. Averaging often helps with many repeats. Liu et al. 2026.
  • Tabular-model compression - Fits affine replacements to target-table activations. Across eight models and 171 classification datasets, halving depth costs at most 0.002 median AUROC per model; single-block regression deteriorates more sharply. Cannistraci and Vogt 2026.
  • Tabular ensembles - See Ensemble Learning for random forests, XGBoost, CatBoost and TabM.

Local and conditional-independence classifiers

  • Nearest-neighbor classification - Predicts from nearby labeled examples. Cover and Hart bound asymptotic one-neighbor error by twice Bayes error under their assumptions; this is neither a finite-sample guarantee nor a guarantee for an arbitrary learned distance. Cover and Hart 1967.
  • Naive Bayes - Multiplies class-conditional feature probabilities under an independence model. Domingos and Pazzani show that correct class decisions can survive some dependence violations even when estimated probabilities are inaccurate. Domingos and Pazzani 1997.

Support-vector methods

  • Support-vector networks - Constructs a linear decision surface in a nonlinear feature space and extends the approach to nonseparable training data. A foundation for kernel-based classification. Cortes and Vapnik 1995.
  • SVM implementations - Linear and kernel classification, regression and outlier detection. Decision scores are not probabilities; probability estimation adds a separate fitting step. docs.

State-space and sequence models

  • Kalman filtering - Formulates linear filtering through state transitions and recursive error-covariance updates. Treats stationary and nonstationary prediction problems in a common framework. Kalman 1960.
  • Dynamic time warping - Aligns sequences by optimizing a nonlinear time-warping path with dynamic programming. Sakoe-Chiba studies path constraints for spoken-word matching; this is an alignment method, not a forecasting model. Sakoe and Chiba 1978.
  • Hidden Markov models - Rabiner reviews Markov models with hidden states and probabilistic observations, with applications to speech recognition. A reference for the model’s statistical structure and inference problems. Rabiner 1989.
    • Infinite HMM - Uses Dirichlet processes to extend HMMs to countably many possible hidden states, integrating out transition parameters rather than fixing a finite state count in advance. Beal et al. 2001.

Mechanistic and learned components

Dynamical and mechanistic models

  • SIR epidemic dynamics - Models infection and removal in a susceptible population under simplifying assumptions about contact and immunity. Epidemics can end while susceptible individuals remain. Kermack and McKendrick 1927.
  • Predator-prey inference - Fits Lotka-Volterra differential equations with a statistical observation model to lynx-hare pelt counts. Demonstrates Bayesian parameter inference and posterior predictive checks. docs.
  • Lorenz dynamics - Studies a deterministic nonlinear convection model whose nearby initial states can diverge. Connects nonperiodic dynamics to limits of long-range prediction. Lorenz 1963.
  • Bass diffusion - Models initial product adoption using innovative and imitative behavior: purchase timing depends on previous buyers. The original study tests the model on consumer durables. Bass 1969.
    • Bass model retrospective - Reviews the origins, applications and extensions of the product-diffusion model, identifying subsequent research directions. Bass 2004.
  • Simple discrete dynamics - Shows how simple first-order nonlinear difference equations can produce stable equilibria, cycles or irregular fluctuations. A foundation for studying deterministic complexity. May 1976 🔒.
  • DICE - Describes an integrated climate-economy model for dynamic optimization of climate policy. Documents the structure and assumptions behind its cost-benefit analysis. Nordhaus 1992.

Calibration and model discrepancy

  • Bayesian calibration - Fits simulator parameters to physical observations while accounting for their remaining uncertainty. A discrepancy term represents systematic differences between the simulator and reality that adjusting parameters alone cannot remove. Kennedy and O’Hagan 2001.
    • Model discrepancy and identifiability - Ignoring simulator error can bias parameters and overstate certainty. Adding a discrepancy function can improve predictions without identifying physical parameters. Recovering them requires informative assumptions about the discrepancy. Brynjarsdóttir and O’Hagan 2014 🔒.
  • Hybrid scientific models - See Scientific ML for learned components constrained by physical models, and Symbolic Regression for discovering dynamical equations.

Software

  • rpart - R implementation of recursive partitioning for classification, regression and survival trees, based on CART. code.
  • party - R toolbox for conditional-inference trees, forests and model-based recursive partitioning. Supports response types including censored and multivariate outcomes. code.
  • mgcv - R implementation of generalized additive and additive mixed models, with smoothing-parameter estimation and several response-distribution families. code.