Learning with Limited Labels

When labels are scarce or unreliable, learning depends on which examples get labeled and what can be inferred from the rest. This review covers semi-supervised learning, active label acquisition, positive-unlabeled data and weak or noisy supervision. Adapting pretrained models belongs in Transfer Learning, and learning to adapt across tasks in Meta-Learning.

Learning from labeled and unlabeled examples

  • Harmonic label propagation - Places labeled and unlabeled examples on a similarity graph. Fixing the labeled values, the remaining predictions minimize graph energy and equal weighted averages of their neighbors. The graph encodes the assumption that nearby examples share labels. Zhu et al. 2003.
  • Mean Teacher - Averages model weights to produce teacher targets for consistency training on perturbed inputs. This uses unlabeled observations during fitting; its benefit depends on the architecture and the relation between labeled and unlabeled data. Tarvainen and Valpola 2017.
  • MixMatch - Averages predictions across augmented unlabeled images, sharpens the targets and mixes labeled and unlabeled examples with MixUp. On CIFAR-10 with 250 labels, test error is about 11% versus 38% for the reported prior best. Berthelot et al. 2019.
  • FixMatch - Uses confident predictions on weakly augmented unlabeled images as targets for strongly augmented versions. Combines pseudo-labeling with consistency; the confidence threshold and augmentation policy determine which unlabeled evidence enters training. Sohn et al. 2020.
  • Learning from multiple views - See Multi-View Learning for co-training and assumptions about complementary views.

Choosing which examples to label

  • BALD active learning - Chooses labels expected to reduce posterior uncertainty about the model, using disagreement among predictive distributions. Houlsby et al. derive it for Gaussian-process classification; querying labels differs from passive few-shot adaptation. Houlsby et al. 2011.
  • TypiClust - Selects typical examples from diverse clusters in self-supervised representations. At very small image-label budgets it beats tested uncertainty-based strategies. Experiments and a mixture-model analysis show that the preferred strategy changes with budget. Hacohen et al. 2022.

Selective, weak and noisy labels

  • Positive-unlabeled learning - Learns from known positives and an unlabeled mixture of both classes. If labeled positives are selected uniformly within their class, label probability is proportional to class probability; selective labeling breaks this correction. Elkan and Noto 2008.
  • Class-conditional loss correction - With known class-specific label-flip rates, transforms a binary loss so its expectation equals the clean-label loss. Assumes independent flips within each class and rates summing to less than one. Natarajan et al. 2013.
  • Programmatic weak supervision - Snorkel combines noisy labeling functions into probabilistic training labels by modeling their accuracies and dependencies, then trains a discriminative model. Development and test sets remain labeled. Ratner et al. 2017.
  • Confident Learning - Uses out-of-sample probabilities to estimate the joint distribution of observed and latent labels and rank suspected errors. Assumes class-conditional label noise; exact recovery also needs constraints on prediction errors. Northcutt et al. 2021; code.

Comparative evidence

  • Semi-supervised evaluation - Oliver et al. find that out-of-class unlabeled examples can degrade tested methods and that strong supervised baselines are often underreported. Match backbones and label-selection budgets; more unlabeled data need not improve performance. Oliver et al. 2018.