Transfer Learning
- Transfer learning survey - Organizes transfer by differences between source and target domains, tasks, and available labels; covers instance, feature, parameter, and relational knowledge transfer. Pan and Yang 2010.
Pretraining and task adaptation
- ULMFiT - Pretrains a language model, adapts it to target text, then tunes a classifier. Layer-specific learning rates and gradual unfreezing control how pretrained weights change. Howard and Ruder 2018.
- Domain- and task-adaptive pretraining - Continues language-model pretraining on unlabeled domain text or task-specific text before supervised fine-tuning; studies both separately and together. Gururangan et al. 2020.
- Language pretraining - See NLP for BERT and text-to-text learning for T5.
- Language-defined classification - CLIP uses descriptions of candidate classes for zero-shot prediction without target-dataset fine-tuning; see Multi-View Learning.
Frozen features and fine-tuning
- DeCAF - Reuses activations from a supervised object-recognition CNN as fixed features for scene recognition, domain adaptation, and fine-grained classification. Donahue et al. 2014.
- How transferable are features? - Layer-transfer experiments separate task specialization from optimization difficulties caused by splitting co-adapted neurons; transferability depends on layer and task distance. Yosinski et al. 2014.
- L2-SP - Regularizes fine-tuning toward pretrained parameter values rather than zero, explicitly encouraging retention of the source solution. Li et al. 2018.
- Linear probing then fine-tuning - Fits a linear head before updating the whole model. Studies how fine-tuning can distort useful pretrained features and hurt out-of-distribution accuracy despite improving in-distribution accuracy. Kumar et al. 2022.
- FINO - Adapts visual representations to scientific domains using discrete or continuous metadata alongside self-supervised learning. Gardès et al. 2026.
Parameter-efficient adaptation
- Bottleneck adapters - Inserts small trainable modules into a pretrained Transformer. The original recipe also tunes layer normalization and the task head; most backbone parameters stay fixed. Houlsby et al. 2019.
- Prefix tuning - Freezes the language model and learns continuous prefix activations across its layers, providing task-specific conditioning for generation. Li and Liang 2021.
- Prompt tuning - Learns soft prompt embeddings prepended to the input while freezing the language model; the trainable prompt operates at the input rather than every layer. Lester et al. 2021.
- LoRA - Freezes pretrained weights and learns low-rank additive updates to selected matrices. Reduces trainable parameters and task-specific storage; the base weights need not be low-rank. Hu et al. 2022.
- QLoRA - Trains LoRA adapters through a frozen 4-bit base model. NF4 storage, double quantization, and paged optimizers reduce training memory; computation uses higher precision. Dettmers et al. 2023.
- DoRA - Separates pretrained weights into magnitude and direction; learns magnitude changes and uses LoRA for directional updates. Liu et al. 2024.
- LoRA Learns Less and Forgets Less - In studied code and math adaptation settings, standard low-rank LoRA learns less than full fine-tuning but retains more performance outside the target domain. Biderman et al. 2024.
- IA3 - Learns vectors that rescale attention keys, values, and intermediate feed-forward activations while freezing the backbone; introduced with the T-Few adaptation recipe. Liu et al. 2022.
- LoRA tuning evidence - With learning-rate sweeps, LoRA, PiSSA, MiLoRA, Init[AB] and DoRA reach similar peak scores on tested 0.6B-13B LLM tasks but favor different rates. Rank and batch-size sweeps cover selected settings; the study does not establish equivalence on all tasks. Lee et al. 2026.
Domain adaptation
- Covariate-shift weighting - Reweights source examples toward the target input distribution. Assumes unchanged label probabilities given inputs and source support covering the target; estimates weights discriminatively. Bickel et al. 2009.
- DANN - Uses labeled source and unlabeled target data. A gradient-reversal layer trains features to support source classification while confusing a source-versus-target discriminator. Ganin et al. 2016.
- Deep CORAL - Aligns source and target feature covariances during neural-network training, extending linear CORAL to learned nonlinear representations. Sun and Saenko 2016.
- SHOT - Adapts from a pretrained source model and unlabeled target data without revisiting source data. Freezes the classifier and updates features through information maximization and pseudo-labeling. Liang et al. 2020.
Test-time adaptation
- Test-Time Training - Jointly trains prediction and a self-supervised auxiliary task, then updates shared features on unlabeled test inputs before prediction. Includes per-sample and online variants. Sun et al. 2020.
- Tent - Minimizes prediction entropy on unlabeled test batches by updating normalization statistics and channel-wise affine parameters; requires no source data or changes to source training. Wang et al. 2021.
- CoTTA - Adapts to changing target streams using averaged teacher predictions and stochastic restoration of source weights to reduce error accumulation and forgetting. Wang et al. 2022.
Generalization to unseen domains
- Invariant Risk Minimization - Seeks a representation admitting the same optimal classifier across training environments. Counterexamples show this objective can still fail on unseen shifts. Arjovsky et al. 2019, Rosenfeld et al. 2021.
- Group DRO - Minimizes worst-group training loss using predefined groups. Experiments show regularization or early stopping matters for worst-group generalization in overparameterized networks. Sagawa et al. 2020.
Few-shot and in-context adaptation
- Learning to adapt - See Meta-Learning for Matching Networks, Prototypical Networks, MAML and their comparisons.
- In-context learning - See NLP for GPT-3 task demonstrations without gradient updates.
Learning with limited or unreliable labels
- Limited and unreliable labels - See Learning with Limited Labels for semi-supervised learning, active label acquisition, positive-unlabeled learning and weak or noisy supervision.
Model merging
- Model Soups - Averages weights of compatible models fine-tuned from a shared pretrained initialization. Produces one model for inference; weight averaging is a different operation from averaging predictions. Wortsman et al. 2022.
- Task arithmetic - Defines task vectors as fine-tuned weights minus a shared pretrained initialization. Adding, subtracting or combining these vectors edits model behavior without another gradient-training run. Ilharco et al. 2023.
- TIES-Merging - Merges fine-tuned models by trimming small parameter changes, resolving conflicting update signs and combining sign-aligned changes; targets interference between task-specific updates. Yadav et al. 2023.
- Sequential adaptation - See Continual Learning for retention, replay and interference across task sequences.
Evidence and evaluation
- Domain-adaptation theory - Relates target error to source error, domain divergence, and the existence of a hypothesis that works in both domains. Distribution alignment alone is insufficient. Ben-David et al. 2009.
- Negative transfer - Defines harmful transfer relative to the same learning algorithm without source data; analyzes source-target mismatch and target-label availability, and proposes filtering unrelated source examples. Wang et al. 2019.
- DomainBed study - Compares domain generalization under controlled architectures and model selection. Carefully tuned empirical risk minimization is competitive on the studied benchmarks. Gulrajani and Lopez-Paz 2021.
- WILDS - Benchmarks naturally occurring distribution shifts across hospitals, locations, time, and other groups, with prescribed splits and evaluation metrics. Koh et al. 2021.
Software
- PEFT - Hugging Face library for adapting pretrained models through a subset of parameters; integrates with Transformers, Diffusers, and Accelerate. docs.
- TorchVision models - Pretrained vision weights with associated preprocessing transforms. Select a specific weight version for reproducible transfer experiments. docs.
- DomainBed - Domain-generalization algorithms, datasets, training scripts, and model-selection procedures in a shared experimental framework. code.
- WILDS package - Dataset loading, predefined splits, group metadata, evaluation functions, and baseline code for distribution-shift benchmarks. code.