Reinforcement Learning

Temporal differences and action values

  • Temporal-difference learning - Learns predictions from differences between successive predictions, rather than waiting only for final outcomes. Establishes convergence and optimality results for specified cases. Sutton 1988.
  • Q-learning - Incrementally estimates action values from experience. Proves convergence under the paper’s tabular representation, sampling, and convergence assumptions; this is not a deep-network convergence guarantee. Watkins and Dayan 1992.
  • DQN - Combines convolutional visual representations with Q-learning to estimate future reward from game pixels. The original preprint studies seven Atari games; later DQN evaluations use different protocols and game sets. Mnih et al. 2013.
    • Double DQN - Separates action selection from action evaluation in the Q-learning target. Reduces overestimation arising from using the same estimates to select and evaluate an action. van Hasselt et al. 2016.
    • Dueling networks - Splits the action-value network into state-value and action-advantage estimators. Shares learning across actions without requiring a different underlying reinforcement-learning algorithm. Wang et al. 2016.
    • Rainbow - Combines six DQN extensions and measures their contributions through ablations. Demonstrates that improvements need to be tested jointly; isolated gains need not simply add together. Hessel et al. 2018.
  • Prioritized experience replay - Samples selected stored transitions more often instead of replaying all experience uniformly. Changes which past interactions drive learning, rather than how the agent collects new data. Schaul et al. 2016.
  • Replay volume and recency - In continuous-control off-policy RL, recency-biased replay helps most when few transitions are replayed per environment step. Equal-recency comparisons also favor higher-entropy sampling; the study does not cover Atari. Korniak et al. 2026.

Return distributions

  • Distributional reinforcement learning - Models the distribution of random returns instead of only their expectation. Develops distributional Bellman analysis and a practical approximate-distribution algorithm, while identifying instability in the control setting. Bellemare et al. 2017.
  • Quantile regression DQN - Represents the return distribution through quantiles and learns them with quantile regression. Connects approximate distribution learning to a distributional Bellman formulation. Dabney et al. 2018.

Policy gradients and actor–critic methods

  • REINFORCE - Derives stochastic-network updates aligned with the gradient of expected reinforcement. Covers immediate rewards and specified delayed-reward settings, and connects the updates with backpropagation. Williams 1992.
  • TRPO - Optimizes a policy surrogate with a constraint on policy divergence. The practical algorithm approximates a theoretically justified update; monotonic improvement is not an unconditional implementation guarantee. Schulman et al. 2015.
  • Generalized advantage estimation - Uses an exponentially weighted advantage estimator to reduce policy-gradient variance at the cost of bias. Connects advantage estimation to temporal-difference learning with eligibility traces. Schulman et al. 2016.
  • A3C - Runs parallel actor-learners with asynchronous parameter updates. Studies how diverse concurrent experience stabilizes deep value and actor-critic learning. Mnih et al. 2016.
    • UNREAL - Adds pixel control, reward prediction and value replay to A3C. On 13 Labyrinth tasks, the authors report a 10x mean reduction in steps needed to reach A3C’s final performance. Jaderberg et al. 2017.
  • DDPG - Uses a deterministic policy gradient within an actor-critic architecture for continuous actions. Extends deep Q-learning ideas to settings where enumerating all actions is impractical. Lillicrap et al. 2016.
  • PPO - Alternates interaction with minibatch optimization of a policy surrogate. Its clipped-ratio variant limits incentives for large updates; clipping is not a hard bound on policy divergence. Schulman et al. 2017.
  • TD3 - Uses the minimum of two critic estimates and delays policy updates to reduce effects of approximation error. Addresses overestimated values in continuous-action actor-critic learning. Fujimoto et al. 2018.
  • Soft Actor-Critic - Optimizes reward together with policy entropy using an off-policy stochastic actor-critic. The entropy term encourages varied actions while the critic estimates task value. Haarnoja et al. 2018.
    • SAC temperature adaptation - Extends Soft Actor-Critic with a constrained formulation that tunes the entropy temperature automatically. Studies sensitivity to settings and applications in simulated and physical control. Haarnoja et al. 2018.
  • Control as inference - Connects maximum-entropy control with probabilistic inference. Distinguishes exact inference under deterministic dynamics from a variational treatment under stochastic dynamics. Levine 2018.

Exploration and goal relabeling

  • Density-based pseudo-counts - Derives visitation-like counts from density models to extend count-based exploration beyond tabular states. Converts estimated novelty into intrinsic rewards. Bellemare et al. 2016.
  • Intrinsic curiosity module - Uses prediction error in an inverse-dynamics-trained feature space as an intrinsic reward. Encourages exploration through failures to predict action consequences rather than raw pixel reconstruction alone. Pathak et al. 2017.
  • Hindsight experience replay - Reuses unsuccessful goal-conditioned episodes by relabeling them with goals actually achieved. Recomputes the associated rewards to learn from sparse binary feedback. Andrychowicz et al. 2017.
  • NoisyNet - Adds parameterized noise to network weights and learns its parameters with the task objective. Produces exploratory policy variation through the network rather than only random action replacement. Fortunato et al. 2018.
  • Random network distillation - Rewards prediction error against features from a fixed random network. Introduces a novelty signal and a mechanism for combining intrinsic and task rewards. Burda et al. 2019.

Goal-conditioned representations and planning

  • Search on the Replay Buffer - Builds a graph of visited states with goal-value-based distance estimates, then searches for subgoals a local policy can reach. Planning depends on replay coverage and distance accuracy; evaluated on image navigation. Eysenbach et al. 2019.
  • Contrastive RL - Contrasts future states with random states for each state-action pair. The optimal critic encodes discounted future-state occupancy ratios; maximizing its score trains a goal-reaching policy. Eysenbach et al. 2022.
  • HIQL - Learns an offline goal-value function, a high-level policy predicting latent subgoals and a low-level controller. Extends IQL to hierarchical goal reaching; value learning can use action-free trajectories, while controller extraction needs actions. Park et al. 2023.
  • Inference via Interpolation - Its temporal contrastive parameterization yields a Gauss-Markov chain under Gaussian-marginal and optimal-critic assumptions. Infers latent waypoints from offline trajectories; no online control-policy learning. Eysenbach et al. 2024.
  • C-Slerp - Spherical interpolation of contrastive embeddings supplies adaptive waypoints, improving success in 8-tower and Ant U5 Maze simulations. Its derivation assumes a Bayes-optimal critic, uniform spherical marginals and a small-variance approximation. Park et al. 2026; code.
  • Skill objectives and goal criteria - Repeated occupancy, exact-time arrival and arrival within a window can require different policies. Finite-state theory links skill-information objectives to matching goal criteria; no one diversity objective optimizes all of them. Modirshanechi et al. 2026.

Memory and temporal abstraction

  • Options and Option-Critic - Options specify initiation, an internal policy and termination. Option-Critic learns option policies and termination functions by gradient methods, enabling temporal abstraction. Sutton et al. 1999, Bacon et al. 2017.
  • Deep recurrent Q-networks - Adds an LSTM to DQN to integrate observations over time. Studies partial observability and changing observation quality; recurrence does not show a systematic advantage on every fully observed task. Hausknecht and Stone 2015.
  • Successor features - Separates expected future feature occupancy from reward weights, enabling policy transfer across tasks with shared dynamics and different rewards; paired with generalized policy improvement. Barreto et al. 2017.
  • HIRO - Learns high-level goals and lower-level goal-conditioned control using off-policy data. Corrects higher-level replay for changes in the lower-level behavior that alter what a goal accomplishes. Nachum et al. 2018.
  • DIAYN - Learns diverse skills without task rewards using an information-theoretic objective and maximum-entropy policies. Studies skill pretraining and hierarchical composition for later tasks. Eysenbach et al. 2019.

Search and self-play

  • AlphaZero - Combines neural policy/value prediction, Monte Carlo tree search, and self-play for board games. Uses the game rules for planning; learning without human games does not mean learning without a simulator. Silver et al. 2017.

Learned dynamics and planning

Learning from imagined experience

  • Linear Dyna and prioritized sweeping - Learns from imagined transitions generated by a world model. Extends Dyna planning and prioritized sweeping to linear function approximation, with convergence results under stated conditions. Sutton et al. 2008.
  • Model-Based Policy Optimization - Uses short model-generated rollouts branching from real observations. Limits reliance on long simulated trajectories, where accumulated model error can undermine policy optimization. Janner et al. 2019.
  • Dreamer - Learns behavior by propagating value gradients through imagined trajectories in a compact latent world model. Separates learning environmental dynamics from collecting every policy update through real interaction. Hafner et al. 2020.
    • DreamerV2 - Uses discrete latent world-model representations and learns behavior from model predictions. Trains the world model separately from the policy and studies both Atari and continuous control. Hafner et al. 2021.
    • DreamerV3 - Combines latent imagination with normalization, balancing, and transformation choices intended to stabilize learning across domains. Evaluates a shared configuration across diverse tasks. Hafner et al. 2025.

Task-relevant models for planning

  • Value Prediction Network - Predicts rewards, discounts and values along abstract rollouts without reconstructing observations. In stochastic Collect gridworlds, it outperforms the tested observation-prediction model. Oh et al. 2017.
  • MuZero - Learns latent dynamics from reward, value and search-policy targets for tree search. AlphaZero instead uses known game rules. MuZero requires neither pixel reconstruction nor matching the encoding of a future observation. Schrittwieser et al. 2020.
    • EfficientZero - Adds a SimSiam-style loss matching predicted latents to encodings of future observations. Removing it cuts median human-normalized score from 1.16 to 0.34 across 26 Atari 100k games. Ye et al. 2021.
  • Value equivalence - Models can produce identical Bellman updates for specified policies and candidate value functions despite different transitions. Expanding these sets reduces which dynamics the model can ignore. Grimm et al. 2020.
    • Proper value equivalence - In discounted MDPs, models matching every deterministic policy’s value at every state preserve optimal planning despite different dynamics. MuZero’s current-policy loss alone does not ensure this. Grimm et al. 2021.
  • TD-MPC / TD-MPC2 - Continuous-control planning with latent models and terminal value estimates. TD-MPC couples reward and value prediction to matching future encodings.
  • Uncertainty-aware receding-horizon control - See MPC for PETS and probabilistic dynamics ensembles.

Learning from fixed datasets

  • Batch-Constrained Q-learning - Restricts candidate actions to remain close to those supported by a fixed dataset. Addresses extrapolation errors that arise when offline policy learning evaluates poorly supported actions. Fujimoto et al. 2019.
  • Behavior-regularized actor-critic - Organizes offline RL around regularizing policy behavior relative to logged data. Uses shared comparisons and ablations to examine which algorithmic choices matter. Wu et al. 2019.
  • Conservative Q-learning - Adds a value regularizer to Bellman-error training to learn conservative estimates from offline data. Lower-bound results have assumptions; conservative training does not guarantee deployment performance. Kumar et al. 2020.
    • Cal-QL - Calibrates conservative offline value estimates against a reference policy to improve initialization for subsequent online learning; addresses the offline-to-online transition. Nakamoto et al. 2023.
  • Implicit Q-learning - Fits an expectile state-value estimate, backs it into a Q-function, and extracts a policy through advantage-weighted cloning. Avoids querying unseen actions during value learning. Kostrikov et al. 2021.
  • Decision Transformer - Predicts actions autoregressively from desired return, past states, and actions. Recasts offline policy learning as conditional sequence modeling; requesting a return does not guarantee achieving it. Chen et al. 2021.
  • Diffuser - Plans by iteratively denoising trajectories, combining learned generation with conditioning on goals or rewards. Moves part of trajectory optimization into the learned model. Janner et al. 2022.

Off-policy evaluation

  • Sequential doubly robust evaluation - Combines importance weighting with value-model corrections to estimate a target policy’s return from logged behavior. Jiang and Li reduce variance in studied tasks; requires policy probabilities and action support. Jiang and Li 2016.
  • Fitted Q Evaluation - Repeatedly regresses Bellman targets for a fixed policy on logged transitions, then estimates its value. Le et al.’s error bounds require coverage and function-class conditions; fitting low regression error alone does not certify a new policy. Le et al. 2019.

Demonstrations and preference feedback

  • DAgger - Aggregates expert labels on observations encountered by the learner’s policy. Addresses distribution shift between expert demonstrations and the states visited after the learner makes mistakes. Ross et al. 2011.
  • GAIL - Learns a policy through adversarial comparison with expert behavior, without a supplied task reward. The original method still collects policy rollouts; it is not purely offline behavioral cloning. Ho and Ermon 2016.
  • Human preference rewards - Learns task objectives from human comparisons of trajectory segments and trains policies using that feedback. Replaces hand-written task rewards with learned preferences, not with direct access to human intent. Christiano et al. 2017.
  • Learning with demonstrations and long horizons - Combines a transformed Bellman operator, temporal consistency, and human demonstrations for Atari. Separates reward-scale handling, long-horizon learning, and exploration contributions. Pohlen et al. 2018.

Robotic imitation

  • ACT - A conditional VAE predicts action chunks and averages overlapping predictions. Without this averaging, increasing chunks from 1 to 100 steps raises mean success from 1% to 44% across two simulated bimanual tasks, then tapers. Chunking also helps tested baselines. Zhao et al. 2023; code.
  • Diffusion Policy - Denoises action sequences and executes a prefix before replanning. The better of its CNN and Transformer variants matches or beats tested baselines on eight simulated tasks. On BlockPush, the CNN completes both blocks in 11% of trials versus BET’s 71%. Chi et al. 2023; code.
  • OpenVLA - Fine-tunes a 7B vision-language model on 970k robot episodes to emit discretized action tokens. Adapted to new tasks, it beats Diffusion Policy on diverse language-conditioned tasks, while Diffusion Policy remains competitive on narrower ones. Kim et al. 2025; code.
    • OpenVLA-OFT - On LIBERO with OpenVLA’s inputs, parallel decoding of 8-step chunks raises success from 76.5% to 90.2%. Continuous L1 regression reaches 95.3%, versus 95.4% for 50-step diffusion. Demonstrations use a consistent strategy per task. Kim et al. 2025; code.
  • Embodied reasoning and control - In robot-policy ablations, movement and trajectory guidance help more than high-level reasoning alone. Scaling noisy embodied CoT pretraining can reduce success; ERVLA combines grounding, reasoning dropout and an action policy. Sun et al. 2026.

Multiple agents and constrained control

  • MADDPG - Extends actor-critic learning to account for other agents’ actions during training. Studies cooperative and competitive interaction, where other learning agents make the environment nonstationary. Lowe et al. 2017.
  • Constrained Policy Optimization - Optimizes reward subject to expected-cost constraints through a policy-search update. Theoretical near-constraint satisfaction depends on assumptions and does not certify physical safety of a learned controller. Achiam et al. 2017.
  • Evolution strategies for control - Optimizes policy parameters through black-box evaluations and perturbations. Provides a control-learning baseline that does not require value functions or policy-gradient credit assignment through actions. Salimans et al. 2017.
  • QMIX - Combines per-agent values with a monotonic mixing network for centralized training and decentralized action choice. The monotonicity restriction makes local greedy choices consistent with the joint value model. Rashid et al. 2018.
  • Multi-agent RL review - Organizes challenges including nonstationarity, partial observability, communication, and centralized versus decentralized training. Surveys approaches and their application settings. Nguyen et al. 2020.

Evaluation and datasets

  • Reproducibility in deep RL - Shows how nondeterminism, hyperparameters, and reporting choices complicate algorithm comparisons. Calls for standardized protocols and uncertainty reporting rather than isolated best-run scores. Henderson et al. 2018.
  • Arcade Learning Environment protocols - Reviews Atari evaluation practices and introduces sticky actions as environmental stochasticity. Makes protocol differences explicit when comparing game-playing agents. Machado et al. 2018.
  • Off-policy evaluation benchmark - Stress-tests methods that estimate a policy’s performance from data collected by other policies. Studies how experimental conditions affect estimator reliability and provides the COBS benchmark. Voloshin et al. 2019.
  • D4RL - Benchmarks offline learning with human, controller-generated, multitask, and mixed-policy datasets. Tests limitations hidden by datasets collected only from partially trained RL agents. Fu et al. 2020.
  • Statistical uncertainty in RL benchmarks - Shows that few-run point estimates can change conclusions about algorithm rankings. Recommends interval estimates, performance profiles, and robust aggregates such as the interquartile mean. Agarwal et al. 2021.

Software

  • Stable-Baselines3 - Provides Python implementations with consistent interfaces, documentation, and comparisons to reference codebases. A practical baseline implementation collection; the cited paper describes its tested release. Raffin et al. 2021.
  • CleanRL - Provides self-contained algorithm implementations in single files, including training details. Supports inspection and modification of an individual algorithm without tracing a large shared framework. Huang et al. 2022.