Multi-View Learning

Overview

  • A Survey on Multi-view Learning - Organizes learning from multiple descriptions of the same objects into co-training, multiple kernel learning, and shared subspaces; distinguishes agreement from complementary information. Xu et al. 2013.

Co-training and label acquisition

  • Co-training - Classifiers trained on separate views exchange confident labels for unlabeled examples. The original guarantee assumes each view is sufficient and the views are conditionally independent given the label. Blum and Mitchell 1998.
    • Analyzing Co-training - Examines how view independence and feature splits affect co-training; introduces co-EM, which exchanges probabilistic labels between views rather than committing to selected hard labels. Nigam and Ghani 2000.
  • Co-regularized Least Squares - Couples view-specific regressors through an agreement penalty on unlabeled examples. A semiparametric approximation reduces the cost of using a large unlabeled set. Brefeld et al. 2006.
  • Active Learning with Multiple Views - Co-Testing requests labels for examples on which view-specific classifiers disagree, combining multiple views with active label acquisition. Muslea et al. 2006.

Shared representations

Correlation-based projections

  • CCA - Finds linear projections of two paired variable sets with maximal correlation. scikit-learn provides fitting and transforms for new samples. Hotelling 1936; docs.
  • Generalized CCA - Compares five extensions of canonical analysis to three or more variable sets, including summed-correlation and maximum-variance objectives; these criteria need not yield the same solution. Kettenring 1971.
  • Kernel CCA: Convergence Analysis - Analyzes generalization and regularization for correlation maximization in kernel feature spaces, allowing nonlinear relationships between paired views. Hardoon and Shawe-Taylor 2008.
  • Deep CCA - Learns a nonlinear encoder for each of two views by maximizing correlation between their outputs. Paired training observations define the cross-view correspondence. Andrew et al. 2013.
    • Deep CCA Autoencoders - Combines cross-view correlation with reconstruction of each input view; compares correlation-based, reconstruction-based, and hybrid objectives for representations used downstream. Wang et al. 2015.
    • Deep Generalized CCA - Extends nonlinear correlation-based representation learning beyond two views through view-specific networks and a shared latent representation. Benton et al. 2019.
  • Multi-view nonlinear ICA - Gresele et al. identify independent latent sources from sufficiently different noisy nonlinear views under their corruption assumptions. Multiple views supply identifying information; maximizing correlation alone is a different objective. Gresele et al. 2020.
  • Unpaired CCA - Fits linear CCA projections after a QAP-based proxy pairing. Correlation-recovery theory assumes the true pairing maximizes correlation and a singular-value majorization condition; the practical matching solver is approximate. Ben-Ari et al. 2026.

Shared and view-specific factors

  • MOFA - Bayesian factor analysis separates variation shared across omics assays from variation specific to individual assays; supports incomplete measurements on overlapping samples. Argelaguet et al. 2018.
    • MOFA+ - Extends MOFA to multiple sample groups and modalities with flexible sparsity and scalable variational inference, including applications to single-cell data. Argelaguet et al. 2020.

Supervised projections and metrics

  • Multi-view Fisher Discriminant Analysis - Extends kernel Fisher discrimination to aligned views, with sparse and private-direction variants for supervised classification. Diethe et al. 2010.
  • Generalized Multiview Analysis - Learns a discriminative latent space for heterogeneous views using class labels; supports cross-view classification and retrieval with linear and kernel formulations. Sharma et al. 2012.
  • Multiview Metric Learning - Combines cross-view correspondence with within-view local smoothness; learns mappings into a shared space for previously unseen samples. Zhai et al. 2012.

Multiple kernel learning

  • MARK - Builds heterogeneous kernel models through boosting and kernel column generation, selecting basis functions from multiple kernels during learning. Bennett et al. 2002.
  • SimpleMKL - Learns a sparse convex combination of kernels jointly with an SVM, alternating classifier optimization and updates of kernel weights. Rakotomamonjy et al. 2008.
  • Multiple Kernel Learning Algorithms - Surveys kernel combination rules and learning objectives. Kernels can encode distinct views, feature subsets, or similarity functions over the same observations. Gönen and Alpaydin 2011.

Graph fusion and clustering

  • Co-regularized Spectral Clustering - Couples spectral embeddings of view-specific similarity graphs through agreement penalties, rather than concatenating raw features. Kumar et al. 2011.
  • Multi-view Clustering via Joint NMF - Factorizes nonnegative views while encouraging their cluster representations toward a common consensus; retains a separate basis for each view. Liu et al. 2013.
  • Similarity Network Fusion - Constructs a sample-similarity network for each data type and iteratively fuses the networks. Views share sample identities, while their feature sets can differ. Wang et al. 2014.

Missing views

  • Partial Multi-View Clustering - Learns a common subspace from two partially overlapping views, using paired examples to connect them; neither view must contain every example. Li et al. 2014.
  • Multimodal VAE - Uses a product-of-experts inference network and subsampled training to infer shared latent variables from different subsets of modalities; models joint generation and missing-modality inference. Wu and Goodman 2018.
  • Mixture-of-experts multimodal VAEs - Uses a mixture-of-experts variational posterior for multimodal generative learning, aiming to separate shared and modality-specific factors and support joint and cross-modal generation. Shi et al. 2019.
  • COMPLETER - Combines contrastive agreement with dual prediction between latent views, recovering missing representations for incomplete multi-view clustering. Lin et al. 2021.

Cross-modal alignment and fusion

  • Tensor Fusion Network - Forms an outer-product tensor from augmented language, visual and acoustic embeddings, explicitly representing unimodal, pairwise and three-way interactions. Zadeh et al. 2017.
  • Gated Multimodal Units - Learns multiplicative gates that control each modality’s contribution to a fused representation; demonstrated with text and images for movie-genre prediction. Arevalo et al. 2017.
  • Multimodal Transformer - Uses directional cross-modal attention to fuse language, audio, and visual sequences without explicit time-step alignment. Modalities still describe the same example. Tsai et al. 2019.
  • Contrastive Multiview Coding - Contrasts matching views of a scene against mismatched observations, learning representations of shared information across multiple sensory channels. Tian et al. 2020.
  • CLIP - Trains image and text encoders to identify matching pairs within a batch. Their shared space supports text-image retrieval and classification using text descriptions of classes. Radford et al. 2021; code.
  • ImageBind - Connects six modalities through image-paired training data; alignment does not require training pairs for every pair of modalities. Girdhar et al. 2023.
  • CLAP - Contrastively trains audio and text encoders on paired recordings and descriptions. Their shared space supports audio retrieval and zero-shot classification through text prompts. Elizalde et al. 2023.
  • SigLIP 2 - Combines sigmoid image-text learning with self-distillation and masked prediction for multilingual alignment and dense visual features. Tschannen et al. 2025; code.
  • Multimodal data-mixture scaling - Fits native text/image/audio MoE losses and checks three held-out training runs; 8 of 9 modality losses are within 2.6%. Transfer is asymmetric in this setup; fitted mixture coefficients depend on tokenizers, data and architecture. Khandelwal et al. 2026.
  • Qwen3-VL-Embedding - Shared vectors for text, images, document images, and video. The accompanying reranker scores pairs rather than producing independently indexable vectors. Li et al. 2026; code.

Vision-language generation

  • Flamingo - Connects frozen vision and language backbones through a visual resampler and gated cross-attention. Interleaved image/video-text prompts support few-shot generation; this differs from CLIP-style embedding alignment. Alayrac et al. 2022.
  • Visual instruction tuning - LLaVA connects a vision encoder to a language model and tunes on image-conditioned instructions generated from captions and boxes. The original open-ended comparison uses GPT-4 judgments; ScienceQA provides a separate task evaluation. Liu et al. 2023.

View selection and evaluation

  • InfoMin - Studies contrastive view selection: reduce shared information while retaining information needed for the downstream task. Removing task-relevant content violates this principle. Tian et al. 2020.
  • Gradient-Blending - Finds that joint models can underperform individual modalities on video benchmarks; weights training losses using estimates of overfitting and generalization across modalities. Wang et al. 2020.
  • MultiBench - Benchmarks multimodal prediction across domains, measuring task performance, training/inference cost, and robustness to noisy or missing modalities. Liang et al. 2021.

Software

  • mvlearn - Python implementations for multi-view decomposition, clustering, and semi-supervised learning with a scikit-learn-style API. Perry et al. 2021; code.
  • CCA-Zoo - Linear, regularized, kernel, deep, and probabilistic CCA implementations. Separate multi-view model-selection utilities preserve sample correspondence. docs, code.
  • MKLpy - Python kernel construction, normalization, combination, and evaluation; includes EasyMKL and GRAM. docs.
  • MOFA2 / mofapy2 - R and Python tools for multi-omics factor analysis on the same or overlapping sample sets, including multi-group models and analysis of learned factors. docs.
  • MultiBench code - Data loaders, fusion methods, training routines, and evaluation tools for multimodal prediction benchmarks. code.
  • multiview - Python multi-view MDS, t-SNE, and spectral clustering. Distinct from mvlearn. code, code.
  • Multiview-FDA - MATLAB routines for multi-view Fisher discrimination, citing Diethe et al.’s 2010 work on nonlinear discriminants from multiple data views. code.