Bayesian Inference

Remember that using Bayes’ Theorem doesn’t make you a Bayesian. Quantifying uncertainty with probability makes you a Bayesian. (Michael Betancourt)

Overview

• Books
  • Free book on Bayesian Inference written in Jupyter Notebooks - Bayesian Methods for Hackers Cam Davidson-Pilon (the main author)
  • Bayesian Data Analysis (2020) Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, Donald B. Rubin
• Bayesian Workflow (2020) Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, Martin Modrák

Markov chain Monte Carlo (MCMC)

• Introduction to Markov Chain Monte Carlo (2011) Charles Geyer
• MH Metropolis–Hastings algorithm
  • Equation of State Calculations by Fast Computing Machines (1953) N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, E. Teller
  • Monte Carlo Sampling Methods Using Markov Chains and Their Applications (1970) W. K. Hastings
  • Understanding the Metropolis-Hastings Algorithm (1995) S. Chib, E. Greenberg
• HMC Hamiltonian Monte Carlo
  • MCMC Using Hamiltonian Dynamics (2011) Radford M. Neal
  • The Geometric Foundations of Hamiltonian Monte Carlo (2014) Michael Betancourt, Simon Byrne, Sam Livingstone, MarkGirolami
  • A Conceptual Introduction to Hamiltonian Monte Carlo (2017) Michael Betancourt
• NUTS
  • The No-U-Turn Sampler: Adaptively Setting Path Lengthsin Hamiltonian Monte Carlo (2014) Matthew D. Hoffman, Andrew Gelman

Approximate Bayesian Computation (ABC)

• Approximate Bayesian Computation in Population Genetics (2002) Mark A. Beaumont, Wenyang Zhang†and, David J. Balding
• Markov chain Monte Carlo without likelihoods (2003) Paul Marjoram, John Molitor, Vincent Plagnol, Simon Tavare
• Sequential Monte Carlo without likelihoods (2007) S. A. Sisson, Y. Fan†, Mark M. Tanak
• Non-linear regression models for Approximate Bayesian Computation (2009) Michael G.B. Blum, Olivier François
• Likelihood-free Markov chain Monte Carlo (2010) Scott A. Sisson, Yanan Fan
• Approximate Bayesian Computation(ABC) in practice (2010) Katalin Csillery, Michael G.B. Blum, Oscar E. Gaggiotti, Olivier Francois
• Hamiltonian ABC
• Reliable ABC model choice via random forests (2016) Pierre Pudlo, Jean-Michel Marin, Arnaud Estoup, Jean-Marie Cornuet, Mathieu Gautier, Christian P. Robert

Variational Inference (VI)

• Bayesian parameter estimation viavariational methods (1999) T.S. Jaakkola, M.I. Jordan
• The Variational Gaussian Approximation Revisited (2009) Manfred Opper, Cedric Archambeau
• Doubly Stochastic Variational Bayes for non-Conjugate Inference (2014) Michalis K. Titsias, Miguel Lazaro-Gredilla
• Variational Inference: A Review for Statisticians (2018) David M. Blei, Alp Kucukelbir, Jon D. McAuliffe
• SVGD Stein Variational Gradient Descent
  • Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm (2019) Qiang Liu, Dilin Wang

Gaussian processes

• Gaussian Processes for Machine Learning (2006) Carl E. Rasmussen, Christopher K. I. Williams

Uncertainty calibration

• The Well-Calibrated Bayesian (1982) A.P. Dawid
• Transforming Classifier Scores into Accurate Multiclass Probability Estimates (2002) Bianca Zadrozny, Charles Elkan
• Predicting Good Probabilities With Supervised Learning (2005) Alexandru Niculescu-Mizil, Rich Caruana
• Nearly-Isotonic Regression (2011) Ryan J. Tibshirani, Holger Hoefling, Robert Tibshirani
• Binary classifier calibration using an ensemble of piecewise linear regression models (2012) Mahdi Pakdaman Naeini, Gregory F. Cooper
• Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers (2017) Meelis Kull, Telmo de Menezes e Silva Filho, Peter Flach
• Verified Uncertainty Calibration (2019) Ananya Kumar, Percy Liang, Tengyu Ma
• Improving Regression Uncertainty Estimates with an Empirical Prior (2020) Eric Zelikman, Christopher Healy

Software

• Stan
• BUGS
• JAGS
• R
  • LaplacesDemon
  • Greta
  • Rstanarm
• Python
  • PyMC3, PyMC4
  • Tensorflow Probability
  • Edward - now part of Tensorflow Probability
  • Zhusuan
  • Pyro
  • NumPyro - Pyro on JAX
  • Brancher
  • Bambi - PyMC and Stan interface
  • BRMP - Pyro interface
• Julia
  • BAT.jl
  • Gen
  • Turing.jl
  • Soss.jl
  • DynamicPPL
• Javascript
  • WebPPL
• Web
  • StatSim
• Installable
  • JASP

Star Issue