Feature Extraction and Representation Learning

đź”’ Full text at this link requires payment or institutional access. Unmarked links may be unchecked.

Surveys

  • Representation Learning: A Review and New Perspectives - Reviews probabilistic models, autoencoders and manifold learning as approaches to learning representations. Bengio et al. 2013.
  • A Survey of Dimensionality Reduction Techniques - Categorizes dimensionality-reduction techniques and explains their mathematical foundations. Sorzano et al. 2014.
  • Feature Selection and Feature Extraction in Pattern Analysis - Reviews the theory, applications, and numerical implementations of feature selection and extraction methods. Ghojogh et al. 2019.

Fixed and domain-informed features

Text and image descriptors

  • Text vectorization - Sparse word/character n-grams, TF-IDF, and feature hashing. docs.
  • Image descriptors - SIFT describes local keypoints; HOG summarizes local gradient orientations. Lowe 2004, Dalal and Triggs 2005.

Signal and time-series transforms

Tabular feature construction

  • OpenFE - Expands tabular features with predefined operators, scores their incremental predictive value using feature boosting, and prunes candidates in two stages. Generation and supervised selection are coupled. Zhang et al. 2023.
  • TabPrep - Jointly tunes feature generators and downstream models, unlike OpenFE’s separate feature selection. With four model families and post-hoc ensembling, improves the best score on 19 of 51 TabArena datasets against newer model baselines. Tschalzev et al. 2026.

Linear and statistical representations

Component analysis and factorization

  • PCA (principal component analysis) - Orthogonal projection maximizing retained variance of centered data. Pearson 1901; docs.
  • Truncated SVD / latent semantic analysis - Low-rank approximation without PCA’s centering step, allowing sparse document-term inputs. Deerwester et al. 1990; docs.
  • NMF (non-negative matrix factorization) - Additive factorization of non-negative data. Lee and Seung 1999 đź”’; docs.
  • Factor analysis - A linear latent-factor model with feature-specific Gaussian noise. docs.
  • ICA (independent component analysis) - Linear separation of statistically independent sources. Hyvärinen and Oja 2000, Hyvärinen et al. 2001; docs.
  • Topic LDA (latent Dirichlet allocation) - Document representations as topic mixtures. Blei et al. 2003, Hoffman et al. 2010.
  • Robust PCA / principal component pursuit - Separates low-rank structure from sparse corruption (X = L + S), with recovery guarantees under rank, incoherence, and corruption assumptions. Candes et al. 2011.

Sparse representations

  • Sparse PCA - Sparse component loadings, selecting which input variables contribute to each component. Zou et al. 2006; docs.
  • Dictionary learning / sparse coding - Sparse sample codes over learned, possibly overcomplete dictionaries; sparsity applies to the codes rather than PCA loadings. Mairal et al. 2010; docs.

Supervised and multiview projections

  • Fisher LDA (linear discriminant analysis) - Supervised projection maximizing between-class relative to within-class scatter. Fisher 1936, Rao 1948, Martinez and Kak 2001; docs.
  • Supervised PCA - Response-based predictor screening followed by PCA. Bair et al. 2006.
  • CCA / PLS - Canonical correlation analysis maximizes correlation between projected views; partial least squares extracts cross-covariance-based components, including for regression. docs.

Kernel methods and random projections

  • Kernel PCA - Component analysis in an implicit kernel feature space. Schölkopf et al. 1998; docs.
  • Kernel Fisher discriminant analysis - Supervised class separation in a kernel feature space. Mika et al. 1999.
  • Nyström approximation - Sampled data landmarks approximate the kernel matrix; an alternative to Fourier sampling. Williams and Seeger 2000; docs.
  • Random projections - Gaussian or sparse random maps approximately preserve Euclidean distances under Johnson-Lindenstrauss conditions. Achlioptas 2003; docs.
  • Random Fourier features - Sampled Fourier maps approximate shift-invariant kernels. Unlike distance-preserving random projections, these construct nonlinear features and may expand dimension. Rahimi and Recht 2007.

Manifold learning and visualization

Distance and spectral methods

  • MDS (multidimensional scaling) - Fits coordinates to dissimilarities or their ordering. Kruskal 1964, Little et al. 2018, Little et al. 2022; docs.
  • Sammon mapping - Weighted distance reconstruction emphasizing smaller input distances. Sammon 1969.
  • SOM (self-organizing maps) - Updates grid-organized prototypes and their neighbors. Kohonen 1982 đź”’.
  • Isomap - MDS on graph-approximated geodesic distances. Tenenbaum et al. 2000; docs.
  • LLE (locally linear embedding) - Fits coordinates using fixed local linear reconstruction weights. Roweis and Saul 2000; docs.
    • HLLE (Hessian eigenmapping) - Manifold coordinates from local Hessian estimates. Supported by scikit-learn’s LLE interface. Donoho and Grimes 2003.
  • Laplacian eigenmaps / spectral embedding - Graph-local similarity represented by Laplacian eigenvectors. Belkin and Niyogi 2003.
  • Maximum variance unfolding / semidefinite embedding - Maximizes spread while preserving local distances through semidefinite optimization. Weinberger et al. 2005.
  • Diffusion maps - Spectral coordinates reflecting random-walk connectivity across scales. Coifman and Lafon 2006.

Neighborhood objectives and learned mappings

Learned representations

Reconstruction and masking

Contrastive learning

  • Contrastive Predictive Coding - Predicts future latent representations from autoregressive context using a contrastive loss and negative samples; learns sequential features without reconstructing raw observations. van den Oord et al. 2018.
  • SimCLR - Learns image representations by contrasting augmented views through a nonlinear projection head. ImageNet linear evaluation favors encoder features before the head over its output. Chen et al. 2020; code.
  • Alignment and uniformity - Relates contrastive learning to positive-pair alignment and uniform spread on the unit sphere. The analysis is asymptotic; normalization alone does not guarantee uniform learned features. Wang and Isola 2020.

Self-distillation and redundancy reduction

  • BYOL - An online network predicts a moving target network’s representation without explicit negative pairs. Grill et al. 2020.
  • DINO - Self-distillation without labels using a momentum teacher; the precursor to DINOv2/v3. Caron et al. 2021.
  • Barlow Twins - Drives cross-correlation between two augmented views toward identity, combining agreement with redundancy reduction. This objective also underlies TLDR. Zbontar et al. 2021.
  • VICReg - Separates view agreement, per-feature variance control, and covariance regularization. Unlike Barlow Twins’ cross-correlation objective, it explicitly constrains each branch’s variance. Bardes et al. 2022; code.

Prediction in representation space

  • I-JEPA - Predicts representations of masked image regions from visible context, rather than reconstructing pixels as in MAE. Assran et al. 2023.
  • JEPA learning bias - In deep linear models, JEPA’s feature-learning order depends on regression coefficients and input-target covariance; masked reconstruction is led by covariance. Theory assumes diagonal covariances, small aligned initialization and stop-gradient targets. Littwin et al. 2024.
  • LeJEPA - Combines a joint-embedding predictive objective with sketched isotropic Gaussian regularization of embeddings. Balestriero and LeCun 2025.
  • JEPA predictive closure - In CIFAR-10 sequences with three-dimensional SIGReg latents and no stop-gradient, an unpredictable cue becomes more decodable when it helps predict other features. Altering predictive links shifts which feature sets are represented. Levy et al. 2026; code.
  • Spectral Ghost - Develops a spectral account of representation sufficiency connecting component analysis and self-supervised objectives. Dai et al. 2026.

Identifiability and disentanglement

  • Limits of unsupervised disentanglement - Locatello et al. show that recovering independent factors requires inductive assumptions. Across seven datasets, stronger disentanglement does not consistently reduce downstream sample needs; factor recovery is a distinct goal. Locatello et al. 2019.
  • Identifiable VAE / nonlinear ICA - Conditions a factorized latent prior on an observed auxiliary variable. Identifiability holds up to specified transformations under variation and regularity conditions; an ordinary VAE with an unconditional prior lacks this guarantee. Khemakhem et al. 2020.

Extraction by modality

Images and video

  • DINOv2 - Scales self-supervised learning to produce reusable image-level and patch-level features. Oquab et al. 2024; code.
  • DINOv3 - Gram anchoring addresses degradation of dense features during extended self-supervised training. SimĂ©oni et al. 2026; code.
  • V-JEPA 2.1 - Dense prediction over visible and masked tokens, plus intermediate-layer self-supervision, extends JEPA features to spatially structured image/video representations. Mur-Labadia et al. 2026; code.
  • FINO - Adapts visual representations to scientific domains using discrete or continuous metadata alongside self-supervised learning. Gardès et al. 2026.

Text and retrieval

Sentence and document vectors

  • Sentence-BERT - Siamese/triplet training makes BERT sentence vectors usable with cosine similarity, avoiding joint query-document encoding. Reimers and Gurevych 2019; code.
  • E5 - Contrastive pretraining on weakly supervised text pairs; evaluated both without labeled-data fine-tuning and after supervised fine-tuning. Wang et al. 2022; code.
  • Qwen3 Embedding - Adapts language-model backbones for instruction-aware multilingual text embeddings. Zhang et al. 2025; code.

Document context and chunk pooling

  • Late chunking - Encodes the document before pooling spans into chunk vectors. Across three encoders and four BEIR datasets, mean nDCG@10 rises from 52.2 to 54.0 with 256-token chunks; individual settings include ties and regressions. GĂĽnther et al. 2024.
  • pplx-embed-context - Combines diffusion-pretrained bidirectional encoders, contrastive learning, and late chunking for document-context-aware passage vectors. Eslami et al. 2026.

Sparse and multivector scoring

  • ColBERT - Independently encodes queries and passages into token vectors; sums each query token’s maximum similarity to a passage token. Late interaction retains multiple vectors, whereas late chunking pools each chunk. Khattab and Zaharia 2020; code.
  • BGE-M3 - Jointly supports dense vectors, sparse lexical weights, and token multivectors through self-knowledge distillation. Chen et al. 2024; code.
  • Multivector expressivity - Proves a worst-case representation-size separation between Chamfer similarity and single-vector inner products; this is not an empirical retrieval ranking. Jayaram 2026.

Audio and time series

  • wav2vec 2.0 - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. Baevski et al. 2020.
  • TS2Vec - Contrastive learning of contextual timestamp features, pooled for subsequences or whole series. Yue et al. 2022; code.
  • BEATs - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. Chen et al. 2023; code.
  • MOMENT - Masked time-series pretraining supports classification, forecasting, imputation, and anomaly detection. Goswami et al. 2024; code.
  • MantisV2 - Uses synthetic pretraining, layer selection and token aggregation for frozen time-series features. UCR ablations favor intermediate layers after more updates, but the final layer with 100K samples and 100 epochs; fine-tuning favors all layers. Feofanov et al. 2026; code.

Cross-modal representations

  • CLIP features - Paired image and text encoders produce embeddings for cross-modal comparison; see Multi-View Learning for CLIP’s objective and implementation.
  • CLAP - Contrastively trains audio and text encoders on paired recordings and descriptions. Their shared space supports audio retrieval and zero-shot classification through text prompts. Elizalde et al. 2023.
  • SigLIP 2 - Combines sigmoid image-text learning with self-distillation and masked prediction for multilingual alignment and dense visual features. Tschannen et al. 2025; code.
  • Qwen3-VL-Embedding - Shared vectors for text, images, document images, and video. The accompanying reranker scores pairs rather than producing independently indexable vectors. Li et al. 2026; code.

Representation compression

  • Matryoshka representation learning - Trains nested vector prefixes at multiple dimensions. Compression is built into representation training; arbitrary truncation of ordinary embeddings is not equivalent. Kusupati et al. 2022; code.
  • TLDR - Compresses features using neighbor pairs and the Barlow Twins objective. On ROxford retrieval, the reported 128-dimensional codes gain 4.1 mAP points over whitened PCA with the tested GeM features and fitting data. Kalantidis et al. 2022; code.
  • Retrieval dimension and margin - For unit-vector retrieval supporting every k-subset of n documents, optimal-order margin requires dimension of order k log(n/k). This worst-case relevance family does not determine the dimension needed for a particular corpus. Bangachev et al. 2026.

Evaluation and comparative evidence

Benchmarks and geometry

  • Neighborhood preservation - Trustworthiness penalizes introduced neighbors; continuity penalizes lost neighbors. These measure geometry rather than held-out task performance. Venna and Kaski 2006; docs.
  • Intrinsic dimension - Global and local estimators characterize data geometry; their estimates are not task-optimized embedding dimensions. Bac et al. 2021; docs.
  • Local versus global structure - On tested 2D layouts, t-SNE/art-SNE preserve neighborhoods better, while PCA/TriMap/PaCMAP/ForceAtlas2 preserve distance ranks better on most datasets. These visualization results do not rank general predictive representations. Huang et al. 2022.
  • MTEB / MMTEB - Text-embedding evaluation across tasks, domains, and languages. Muennighoff et al. 2023, Enevoldsen et al. 2025; code.

Software

  • Python (reduction and dimension estimation) - scikit-learn, umap-learn, openTSNE, PaCMAP/LocalMAP, PHATE, scikit-dimension. docs, docs, docs, code, code, docs.
  • Python (extraction) - Sentence Transformers, timm, Kymatio, tsfresh; evaluation: MTEB. docs, docs, docs, docs, code.
  • R - dimRed, dyndimred, intrinsicDimension, Rdimtools; discriminant analysis: MASS, sda. You 2022; code, code, code, code, code, code.
  • JavaScript (extraction) - Transformers.js runs compatible pretrained models through ONNX Runtime with WASM/WebGPU. docs.
  • JavaScript (visualization) - tsne, umap-js, dimred; interactive tool: StatSim Vis. docs, docs, docs, docs.
  • C++ - Tapkee. code.

Connections

  • Sparse autoencoders for interpretability - Overcomplete sparse features from internal model activations, evaluated for reconstruction and interpretability rather than compactness alone. Gao et al. 2024.