Neural Networks and Deep Learning

Overviews and approximation

  • Perceptron - Rosenblatt’s probabilistic model of recognition, learning, and memory. Studies statistical separability and the capabilities and limitations of early neural systems. Rosenblatt 1958.
  • Universal approximation - Establishes approximation capacity for sufficiently wide feedforward networks with squashing activations. An existence result, not a guarantee of efficient training or generalization. Hornik et al. 1989.
  • Large multilayer perceptrons - Studies deep, wide MLPs trained by online backpropagation on augmented handwritten digits using GPUs. Results concern MNIST and the specified training procedure. Ciresan et al. 2010.
  • Historical overview - Surveys deep supervised, unsupervised, reinforcement, and evolutionary learning, organized around credit-assignment paths and their history. Schmidhuber 2015.
  • Deep Learning - Textbook covering feedforward, convolutional, recurrent, and generative networks; optimization, regularization, representation learning, and approximate inference. Goodfellow et al. 2016.

Feedforward and convolutional architectures

  • GMDH polynomial networks - Builds multilayer polynomial models, selecting useful combinations before constructing more complex ones. Uses an external criterion to control model complexity. Ivakhnenko 1971.
    • GMDH algorithm families - Reviews polynomial and nonparametric GMDH algorithms for forecasting, clustering, and system identification. Ivakhnenko and Ivakhnenko 1995.
  • Neocognitron - A self-organizing visual hierarchy of simple- and complex-cell-like layers, designed for pattern recognition tolerant of position changes. Fukushima 1980.
  • Backpropagation for digit recognition - Encodes task constraints in network architecture and learns the mapping from normalized handwritten digit images to classifications. An early end-to-end recognition system. LeCun et al. 1989.
  • GPU-trained CNNs - Presents a configurable GPU implementation of CNNs with supervised feature learning. Evaluates architecture variants on digit and object classification. Cireşan et al. 2011.
  • MLP-Mixer - Alternates MLP operations across patches and across channels. Demonstrates an image-classification architecture built without convolution or attention. Tolstikhin et al. 2021.
  • CNN survey - Reviews convolutional architectures, component choices, and applications across one-, two-, and higher-dimensional inputs. Li et al. 2022.
  • Kolmogorov-Arnold networks - Learn spline functions on edges. At matched parameters or FLOPs, Yu et al. find MLPs generally better outside formula fitting; B-spline MLPs match or beat KANs on tested formulas. Liu et al. 2025, Yu et al. 2024.

Residual connections

  • ResNet - Learns residual functions relative to layer inputs. Studies how this formulation enables optimization of substantially deeper image-recognition networks. He et al. 2016.
    • Hyper-Connections - Replaces a single residual stream with several streams and learned mixing between them. With four streams, an OLMo-7B comparison lowers validation loss and raises average task accuracy from 70.1% to 71.0% at nearly unchanged parameter count. Zhu et al. 2025.

Sequence models, attention, and implicit depth

Recurrent sequence models

  • LSTM - Uses memory cells and multiplicative gates to preserve error flow over long delays. The original design targets vanishing gradients in recurrent sequence learning. Hochreiter and Schmidhuber 1997.
    • Bidirectional LSTM - Combines forward and backward sequence context for framewise phoneme classification. Access to later inputs must match the deployment setting. Graves and Schmidhuber 2005.
    • xLSTM - Adds exponential gates and scalar or matrix memories to LSTM. Matrix memory supports parallel training. At 1.3B parameters on SlimPajama, extending 2K training contexts to 16K yields lower perplexity than tested Mamba and RWKV-4 baselines. Beck et al. 2024.
  • S4 - Uses a structured state-space parameterization to model long sequences efficiently, reducing its core computation to a Cauchy kernel. Gu et al. 2022.
  • Mamba - Makes state-space parameters depend on the input, allowing selective retention and forgetting. Uses a hardware-aware recurrent computation with linear scaling in sequence length. Gu and Dao 2023.
    • Mamba-2 - Connects structured state-space models and attention variants through semiseparable matrices. Uses this duality to derive a revised selective state-space layer. Dao and Gu 2024.
  • RWKV - Pairs parallelizable training with recurrent decoding whose state size is independent of context length. Across 12 NLP tasks, models up to 14B parameters achieve average accuracy comparable to Pythia, OPT and BLOOM at similar training compute. Peng et al. 2023.
  • DEER - Evaluates nonlinear recurrences with parallel Newton solves over the sequence. Dense state Jacobians cost quadratic memory and cubic work in state width; speedups depend on sequence length, width and convergence. Lim et al. 2024.
    • Quasi-DEER and ELK - Quasi-Newton approximations reduce DEER’s state-width cost; ELK uses a trust-region/Kalman-smoothing connection to stabilize iterations. Tested chaotic dynamics can still favor sequential evaluation. Gonzalez et al. 2024.
  • Zoology - In matched Pile runs, a repeated-bigram slice accounts for 82% of the average log-loss gap between attention and H3, Hyena or RWKV-v4. Adding sparse, input-dependent attention closes most of the gap in tested convolutional hybrids. Arora et al. 2024.
  • BASED - Combines linear attention, local attention and short convolutions. At about 1.3B parameters and 50B Pile tokens, it has similar perplexity to Mamba but improves document extraction and question answering. Full attention still leads on these tasks. Arora et al. 2024.

Fast-weight and delta-rule memory

  • Delta-rule memory - Connects linear attention to fast-weight networks. Replaces additive key-value storage with corrections to the value already retrieved for a key, reducing interference in finite memory. Tests include associative recall, translation and language modeling. Schlag et al. 2021.
    • Parallel DeltaNet - Parallelizes delta-rule updates using products of Householder matrices. At 1.3B parameters and 100B tokens, it improves perplexity and average downstream scores over tested Mamba and GLA models. Attention hybrids further improve recall. Yang et al. 2024.
    • Gated DeltaNet - Combines scalar memory decay with delta-rule corrections, allowing rapid erasure and targeted updates. At 1.3B parameters and 100B tokens, it improves language modeling and retrieval over tested Mamba2 and DeltaNet models. Yang et al. 2025.
    • Kimi Delta Attention - Extends the gated delta rule with channel-wise memory decay. In 16-layer ablations at matched training FLOPs, three KDA layers per latent-attention layer give lower validation perplexity than all-attention, 1:1, 7:1 or 15:1 mixtures. Kimi Team 2025.

Attention

  • Transformer - Builds sequence transduction around attention, removing recurrence and convolution from the architecture. The original encoder-decoder is evaluated on translation and parsing. Vaswani et al. 2017.
  • Multi-query attention - Shares one key and value head across query heads, reducing cache size and memory traffic during autoregressive decoding. Translation and language-model experiments find faster decoding with a small quality loss. Shazeer 2019.
    • Grouped-query attention - Shares keys and values within groups of query heads. Converting T5 checkpoints and training for 5% of the original pretraining compute recovers quality near multi-head attention with decoding speed near multi-query attention. Ainslie et al. 2023.
  • Gated attention - Output gates let heads suppress residual updates without extreme softmax scores. Training with gates reduces 8-bit quantization loss in BERT, OPT and ViT. For OPT-1.3B, quantized perplexity remains 29.95 versus 12.21 at full precision. Bondarenko et al. 2023.
    • Gating at language-model scale - Per-dimension sigmoid gates add nonlinearity and query-dependent suppression between value and output projections. After YaRN extension, RULER-128K rises from 31.65 to 58.82. Native-32K scores are 79.50 and 79.77. Qiu et al. 2025.
  • Rotary position embeddings - Encodes position through rotations of query and key vectors, making their dot products depend on relative position offsets; used inside attention rather than as added position vectors. Su et al. 2024.
    • p-RoPE - RoPE need not make attention decay with distance. Gemma uses high frequencies for positional heads. At 8K context, Gemma-2B’s perplexity slightly improves with the lowest quarter unrotated but worsens with three quarters unrotated. Barbero et al. 2025.
  • Multi-head latent attention - Compresses keys and values into a shared latent vector, with separate positional keys. DeepSeek-V2 ablations reduce cached elements while retaining comparable or better scores than multi-head attention on the four tested benchmarks. DeepSeek-AI et al. 2024.

Recurrent depth and adaptive computation

  • Adaptive Computation Time (ACT) - Learns how many recurrent updates to perform before each output, with a penalty for extra computation. Improves tested algorithmic tasks, while character-level Wikipedia modelling shows little gain. Graves 2016.
  • Universal Transformers - Apply shared attention and transition layers repeatedly to all positions. Improve over Transformers on tested algorithmic and language tasks. Optional per-position halting helps structured tasks but slightly worsens translation. Dehghani et al. 2019.
    • MoEUT - Combines shared layers with sparse experts. Tested 44M-1B language models reduce perplexity with fewer arithmetic operations than dense baselines. The reported implementation trains 1.5-2x slower than the dense FlashAttention baseline. Csordás et al. 2024.
  • Huginn - Repeats a shared Transformer core before each token. A 3.5B model trained on 800B tokens improves math and coding scores with more iterations. The 180B-token comparison matches parameters and data, not training compute. Geiping et al. 2025.
  • Recurrence and compute - In 116 FineWeb-Edu runs at 20 effective layers, shared loops save parameters but increase validation loss at matched training compute. The trade-off changes with gradient truncation and hyperconnections. Schwethelm et al. 2026.
  • Loop placement - In dense models up to 1B scale, full-stack loops generally give lower language-model loss at fixed parameters and data. Middle-stack loops make better use of tight training-FLOP budgets. Gains saturate or reverse as more loops are added. Ekbote et al. 2026.

Continuous and implicit depth

  • Neural ODEs - Parameterizes hidden-state derivatives with a neural network and computes outputs with an ODE solver. Numerical tolerances and solver behavior become part of model execution. Chen et al. 2018.
  • Deep equilibrium models - Finds a weight-tied network’s equilibrium by root finding and differentiates it implicitly. Replaces an explicit stack of layers with a fixed-point computation. Bai et al. 2019.

Architecture comparisons

  • Architecture modification transfer - Many published Transformer changes fail to improve a shared T5 setup. GLU activations and RMSNorm transfer more consistently across pretraining, fine-tuning and translation. Most comparisons hold hyperparameters fixed. Narang et al. 2021.
  • bux architecture comparisons - In small code/web models, fine-grained MoE improves loss per active parameter. Sharing attention keys and values trades some quality for cheaper serving. Equal-serving-cost rankings extrapolate from small models to much larger ones. Pavlichenko 2026; code.

Reservoir computing

  • Echo state networks - Drives a fixed recurrent reservoir with inputs and fits a linear readout. The echo-state property makes states determined by input history as initialization effects fade; readout fitting avoids training recurrent weights. Jaeger 2001.
  • Liquid state machines - Uses transient responses of a recurrent spiking circuit as temporal features for a trained readout. Its approximation results require both separation by the liquid and sufficient readout capacity. Maass et al. 2002.
  • Delay-based reservoirs - Time-multiplexes inputs through one nonlinear node with delayed feedback, creating virtual reservoir nodes. Reduces physical component count while processing those virtual nodes serially. Appeltant et al. 2011.
  • Information-processing capacity - Measures linear-readout reconstruction of orthogonal functions of input history. With i.i.d. inputs and finite fourth moments, the infinite-data sum of capacities is bounded by the number of observed state variables. Dambre et al. 2012.
  • Next-generation reservoir computing - Fits ridge regression on delayed observations and polynomial features, replacing a random recurrent reservoir. Demonstrates forecasting and attractor reconstruction on Lorenz63 and simulated double-scroll dynamics. Gauthier et al. 2021.

Learning operators between functions

  • Operator learning - See Scientific ML for DeepONet, Fourier neural operators and their numerical assumptions.

Conditional computation

  • Sparsely gated mixture of experts - A learned gate selects a sparse combination of feedforward experts per example. Separates total parameter capacity from the subset activated for each input. Shazeer et al. 2017.
  • Switch Transformer - Simplifies mixture-of-experts routing and addresses communication costs and training instability. Studies sparse conditional computation in language models. Fedus et al. 2022.
  • DeepSeekMoE - Splits experts into smaller units and keeps shared experts active for every token. At matched total and active parameters, 2B-scale ablations find finer routing and shared experts improve most tested benchmarks over coarse top-2 routing. Dai et al. 2024.
  • Mixture-of-experts scaling - See NLP for dense/MoE comparisons at matched parameter and compute budgets, including the trade-off between memory and training tokens.

Unsupervised and generative networks

  • Representation learning - See Feature Extraction for competitive learning, autoencoders and self-supervised objectives.
  • Generative networks - See Generative Models for Boltzmann machines, latent-variable models, autoregressive networks and GANs.
  • Sequence generation and forecasting - TimeGAN generates sequences in Generative Models. ForGAN predicts future values in Forecasting.

Bayesian neural networks and uncertainty

  • Bayesian neural inference - See Bayesian Inference for neural priors, approximate posteriors and predictive uncertainty.

Activations and normalization

  • Rectified units in RBMs - Uses noisy rectified linear units to approximate replicated binary units in restricted Boltzmann machines, preserving information about relative input intensity. Nair and Hinton 2010.
  • Deep rectifier networks - Studies rectifying activations that produce exact zeros, including supervised learning without mandatory unsupervised pretraining. Glorot et al. 2011.
  • Leaky rectifiers - Retains a nonzero gradient for negative inputs. Compares rectifier variants and sigmoid-based acoustic models on conversational speech recognition. Maas et al. 2013.
  • Maxout - Takes the maximum over a set of inputs to form a learnable nonlinear unit. Designed and evaluated in conjunction with dropout. Goodfellow et al. 2013.
  • PReLU and rectifier initialization - Introduces a parametric rectifier and an initialization derived for rectifier networks. Studies their use in deep image classification. He et al. 2015.
  • Randomized leaky ReLU - Randomizes the negative slope during training. Compares standard, leaky, parametric, and randomized rectifiers on image-classification tasks. Xu et al. 2015.
  • Batch normalization - Normalizes layer inputs using mini-batch statistics as part of the architecture. Studies effects on training, learning-rate choice, and regularization. Ioffe and Szegedy 2015.
    • Why batch normalization helps - Challenges internal covariate shift as the main explanation and analyzes smoother optimization behavior. An analysis of mechanism, not a universal account of every normalization setup. Santurkar et al. 2018.
  • ELU - Uses an identity branch for positive inputs and a saturating negative branch. Motivated by shifting average activations closer to zero. Clevert et al. 2016.
    • Parametric ELU - Learns the shape of the exponential-linear activation instead of fixing its parameters. Different layers can learn different nonlinear responses. Trottier et al. 2017.
  • S-shaped ReLU - Learns a three-segment piecewise-linear activation with four parameters, jointly with the network. Can represent convex and nonconvex shapes. Jin et al. 2016.
  • GELU - Weights an input by the standard Gaussian cumulative probability at that value. Provides a smooth alternative to ReLU’s sign-based gating. Hendrycks and Gimpel 2016.
  • Layer normalization - Computes normalization statistics across a layer for each example, with learned gain and bias. Avoids dependence on other examples in a mini-batch. Ba et al. 2016.
    • RMSNorm - Normalizes activations by their root mean square with learned rescaling, omitting LayerNorm’s mean subtraction; preserves rescaling invariance without recentering activations. Zhang and Sennrich 2019.
  • Swish - Searches activation functions and identifies input times sigmoid of a scaled input. The scale parameter generalizes the SiLU form. Ramachandran et al. 2017.
  • SELU - Uses scaled exponential-linear activations to obtain self-normalizing behavior under the paper’s assumptions. The result is not a guarantee for arbitrary architectures or initialization. Klambauer et al. 2017.
  • SiLU - Multiplies an input by its sigmoid. The paper studies SiLU and its derivative as activations for reinforcement-learning function approximation. Elfwing et al. 2018.
  • Mish - Uses input times tanh of softplus, producing a smooth nonmonotonic activation. Evaluated across image-classification and detection architectures. Misra 2020.
  • Gated feedforward layers - Compares GLU variants in Transformer feedforward sublayers. Combines two projections by elementwise multiplication, with a nonlinearity applied to one branch. Shazeer 2020.

Training, regularization, and generalization

  • Unsupervised pretraining - Tests how pretraining affects optimization and generalization. Experiments support a regularization interpretation through the basins reached during supervised training. Erhan et al. 2010.
  • Dropout - Randomly omits feature detectors during training to reduce co-adaptation. Trains shared weights across many perturbed network configurations. Hinton et al. 2012.
    • Adaptive dropout - Learns a binary belief network that determines which hidden activities to suppress. The dropout mechanism is trained jointly with the predictive network. Ba and Frey 2013.
    • Fast dropout - Approximates dropout’s random contributions with a Gaussian distribution, sampling or integrating that approximation instead of repeatedly sampling unit masks. Wang and Manning 2013.
    • Dropout analysis - Analyzes Bernoulli gating, ensemble averaging, and regularization in linear and nonlinear networks. Distinguishes exact identities from approximations used for nonlinear activations. Baldi and Sadowski 2014.
  • Differentially private SGD - Clips each example’s gradient, adds Gaussian noise and accounts for privacy loss over training. Abadi et al. trade accuracy against a stated privacy budget; the protected unit here is an example, not automatically an entire user. Abadi et al. 2016.
  • Adversarial training - Optimizes prediction loss against worst-case perturbations in a specified threat set. Madry et al. approximate the inner attack with projected gradients; resistance to tested attacks is not a certificate against every possible perturbation. Madry et al. 2018.
  • Neural tangent kernel - Describes function evolution during gradient training through a kernel. In the specified infinite-width limit the kernel becomes fixed; finite feature-learning networks need not follow that regime. Jacot et al. 2018.
    • Lazy training - Shows how parameter scaling can make training resemble a model’s linearization at initialization. Its kernel-like behavior is not a complete explanation of neural feature learning. Chizat et al. 2019.
  • Double descent - Documents nonmonotonic test error as model size or training duration increases. The observed interpolation regimes qualify simple claims that more capacity always helps or hurts. Nakkiran et al. 2020.
  • Language-model scaling - See NLP for model/data allocation, Chinchilla and evidence about data filtering.
  • Softmax collapse - At high confidence, rounding can erase the correct-class gradient while others remain nonzero; under Adam this can trigger Slingshot spikes. A reformulation of cross-entropy in fp32 matches standard cross-entropy in fp64 on tested modular-arithmetic tasks. Prieto et al. 2025, Liu et al. 2026, Barron et al. 2026.
  • Time-bomb loss spikes - Adam shifts logits together, making BF16 lose token-score differences. In 28M language models trained for 16B tokens at high learning rate without weight decay, FP32 output stages or tuned momentum SGD on the head prevent delayed divergence. Chen et al. 2026; code.

Credit assignment and alternatives to backpropagation

  • Backpropagation - Applies the chain rule backward through network layers to compute weight gradients. Credit assignment determines the learning signal; an optimizer such as SGD or Adam uses that signal to update weights. Rumelhart et al. 1986.

Perturbation-based learning

  • Node versus weight perturbation - Uses loss changes after perturbing activations or weights. In linear teacher-student networks, maximum stable learning rates scale inversely with noise dimension. Werfel et al. 2003.
  • MeZO - Replays SPSA-style weight perturbations from seeds, enabling fine-tuning with inference-level memory. Under a local low-effective-rank Hessian bound, its per-step slowdown depends on effective rank rather than parameter count. Malladi et al. 2023.
  • EGGROLL - Uses low-rank factors for weight-space perturbations in evolution strategies, lowering perturbation storage and evaluation cost versus full-rank search. Population averaging can produce higher-rank updates. Sarkar et al. 2025.
  • Dust - Applies node perturbation per token in transformers. Large populations give lower loss than backprop with SGD at 100k and 1M tokens; with Adam at 1M, they approach the backprop baseline. Extra draws cost more compute. Dahal et al. 2026.

Feedback signals and learned targets

  • Difference target propagation - Learns approximate inverse maps to send activation targets backward. A difference correction compensates for imperfect inverses; local fitting and the original top-layer update still use derivatives. Lee et al. 2015.
  • Feedback alignment - Sends errors through fixed random feedback weights instead of transposed forward weights, retaining local activation derivatives. Lillicrap et al. 2016.
    • Direct feedback alignment - Projects output errors directly to each hidden layer through fixed random weights, bypassing feedback alignment’s layer-by-layer error relay; updates retain local activation derivatives. Nøkland 2016.
  • Synthetic gradients - Learns predictors of downstream error gradients so modules can update without waiting for downstream computation. Gradient predictors need training signals; this decouples backpropagation rather than eliminating it. Jaderberg et al. 2017.

Local objectives, forward differentiation, and equilibrium learning

  • Equilibrium propagation - Contrasts free and weakly nudged equilibria in an energy-based network. Under smoothness and fixed-point assumptions, the update approaches the objective gradient as nudging vanishes; settling has a computation cost. Scellier and Bengio 2017.
    • Equilibrium propagation at ImageNet scale - Centered nudging with a predictive-coding VGG10 reaches 13.23% top-5 error versus 12.20% for backprop after 50 epochs. Iterative equilibration is required; the result establishes accuracy at this scale, not equal training cost. Kerjan et al. 2026.
  • Forward-Forward - Each layer learns to raise a goodness score on positive data and lower it on negative data. Replaces end-to-end error propagation with local objectives; negative-example construction is part of the method. Hinton 2022.
  • Forward gradients - Forward-mode autodiff computes a directional derivative; multiplying by its random direction gives an unbiased estimate for independent zero-mean, unit-variance tangent components. Removes reverse-mode differentiation, not derivatives. Baydin et al. 2022.
    • Local forward gradients - Ren et al. reduce estimator variance with activity perturbations and local greedy losses. Forward-mode autodiff supplies the directional derivatives; ImageNet performance improves over tested alternatives but still trails backprop. Ren et al. 2023.

Comparative evidence

  • Scaling evidence - Bartunov et al. find tested target-propagation and feedback-alignment variants far behind backprop on ImageNet. Results concern their architectures and tuning; direct feedback alignment exceeded their 16GB GPU memory. Bartunov et al. 2018.
  • Noise - A benchmark for training transformers without end-to-end backpropagation. Methods train the same 37.7M-parameter model on 10M tokens, competing on final loss or time to a target loss. The rules allow exact credit through linear maps. code.

Pruning, quantization, and efficient execution

  • Optimal Brain Damage - Uses second-derivative information to remove weights while balancing network complexity against training error. LeCun et al. 1989.
  • Knowledge distillation - Transfers a teacher model or ensemble’s predictions into a single model. Separates the cost of producing training targets from the cost of deploying the student. Hinton et al. 2014.
  • Weightless neural systems - Reviews RAM-based neurons addressed by Boolean inputs, including WiSARD and related architectures. Covers capacity analysis, extensions, and applications. França et al. 2014.
  • Prune and retrain - Trains a dense network, removes unimportant connections, then fine-tunes remaining weights. Reduced parameter counts do not alone establish hardware speedups. Han et al. 2015.
  • Binarized networks - Trains networks with binary weights and activations at runtime, replacing many arithmetic operations with bitwise ones. Actual efficiency depends on implementation and hardware. Courbariaux et al. 2016.
  • Activation checkpointing - Stores selected forward activations and recomputes intermediate values during backpropagation, trading additional computation for lower training memory. Chen et al. 2016.
  • Taylor-based filter pruning - Approximates the loss change from pruning with a Taylor criterion and interleaves removal with fine-tuning. Evaluated for adapted convolutional networks and classification. Molchanov et al. 2017.
  • Compact binary-network training - Studies low learning rates, PReLU, bipolar regularization, output scaling, and multiple activation binarizations to improve the compression-accuracy trade-off. Tang et al. 2017.
  • L0 regularization - Uses differentiable expectations over stochastic gates to encourage exact zeros during training. The hard-concrete relaxation enables gradient-based structure learning. Louizos et al. 2018.
  • Lottery ticket hypothesis - Finds sparse subnetworks whose original initializations permit effective retraining. The experiments support an initialization-sensitive hypothesis, not a universal pruning guarantee. Frankle and Carbin 2019.
  • KD-Lib - A PyTorch library collecting knowledge distillation, pruning, and quantization methods, with support for tuning and experiment logging. Shah et al. 2020.
  • Parameter-efficient adaptation - See Transfer Learning for LoRA, QLoRA and comparisons with full fine-tuning.
  • FlashAttention - Tiles exact attention to reduce traffic between GPU memory levels. Changes execution and memory use without replacing dense attention by an approximate attention rule. Dao et al. 2022.
  • GPTQ - Uses approximate second-order information for one-shot post-training weight quantization of large Transformers. Low-bit accuracy and speed depend on the model and inference implementation. Frantar et al. 2023.
  • AWQ - Uses activation statistics to identify and scale important weight channels before low-bit weight-only quantization. Avoids backpropagation during calibration. Lin et al. 2024.
  • ReplaceMe - Replaces transformer blocks with linear maps fitted to calibration activations; compatible maps merge into existing projections. Least-squares fitting is closed form; cosine-distance fitting uses optimization. Shopkhoev et al. 2025.
  • MoE-to-dense compression - Selects experts, concatenates their feedforward blocks and distills a dense student. Diversity-aware selection beats tested selection/grouping alternatives across three MoEs; benefits vary with expert pool size and distillation budget. Kim et al. 2026.

Ensembles

  • Neural ensembles - See Ensemble Learning for combination methods, disagreement and accuracy-compute comparisons.