Natural Language Processing

Statistical models and structured prediction

  • Maximum-entropy language models - Combines observed feature statistics in a probabilistic model and estimates parameters by maximum likelihood. Develops feature selection and estimation procedures for language-processing tasks. Berger et al. 1996.
  • N-gram smoothing comparisons - Compares smoothing methods across corpora, training sizes, and n-gram orders using held-out cross-entropy. Shows that model settings and data scale affect which estimator works best. Chen and Goodman 1996.
  • Conditional random fields - Models label sequences conditional on observations, allowing dependent labels and rich input features. Avoids the label-bias limitation of locally normalized maximum-entropy Markov models. Lafferty et al. 2001.
  • Neural tagging from learned features - Uses a shared neural approach for part-of-speech tagging, chunking, entity recognition, and semantic roles. Learns representations from largely unlabeled text instead of engineering separate input features for every task. Collobert et al. 2011.
  • Naive Bayes and SVM baselines - Studies unigram/bigram features and NB/SVM variants for sentiment and topic classification. Uses NB log-count ratios as SVM features and shows why a precisely specified baseline matters. Wang and Manning 2012.
  • Neural dependency parsing - Uses learned dense features to choose transitions in a greedy dependency parser. Replaces large sparse feature-template sets with compact representations of parsing configurations. Chen and Manning 2014.

Tokenization and word representations

  • word2vec - Learns word vectors through continuous bag-of-words and skip-gram prediction objectives. Produces a reusable vector per vocabulary word; the representation does not change with each sentence context. Mikolov et al. 2013.
  • GloVe - Fits a weighted log-bilinear model to global word co-occurrence statistics. Combines corpus-level counts with vector representations rather than learning only from individual context windows. Pennington et al. 2014.
  • Subword translation with BPE - Represents rare and unseen words as sequences of subword units. Adapts byte-pair encoding to vocabulary construction for translation, reducing dependence on whole-word dictionaries. Sennrich et al. 2016.
  • fastText subword vectors - Represents words by sums of character n-gram vectors within a skip-gram model. Can construct vectors for unseen words from their subword components. Bojanowski et al. 2017.
  • fastText classification - Uses a simple supervised text-classification architecture with word and n-gram features. Provides a computationally inexpensive baseline distinct from the fastText method for unsupervised word vectors. Joulin et al. 2017.
  • SentencePiece - Trains subword tokenization directly from raw sentences without requiring prior word segmentation. Provides tokenization and detokenization for languages with different writing conventions. Kudo and Richardson 2018.

Contextual pretraining

  • ELMo - Builds context-dependent word representations from internal states of a pretrained bidirectional language model. Allows downstream tasks to combine signals from different layers. Peters et al. 2018.
  • BERT - Pretrains bidirectional Transformer representations with masked-token prediction and an original next-sentence objective. Fine-tunes the encoder for tasks such as classification and extractive question answering. Devlin et al. 2019.
    • RoBERTa - Reexamines BERT training choices, data size, and optimization. Demonstrates that training conditions can explain apparent differences between pretraining methods. Liu et al. 2019.
  • XLNet - Trains autoregressive predictions across permutations of the factorization order to learn bidirectional context. Integrates Transformer-XL mechanisms without using BERT’s masked-token objective. Yang et al. 2019.
  • ELECTRA - Trains an encoder to distinguish original tokens from plausible generator replacements. Uses a discrimination target across input positions rather than predicting only the identities of masked tokens. Clark et al. 2020.
  • Multi-token prediction - Trains future-token heads on a shared trunk. At matched 13B parameter counts, four-token training raises HumanEval pass@1 from 14.1% to 15.8% and MBPP from 26.0% to 30.5%. Smaller models can regress. The heads also draft tokens for decoding. Gloeckle et al. 2024.
    • Soft-MTP - Routes existing cross-entropy gradients so future-token heads learn from the longer-context main head, while the shared output layer keeps hard-label training. At 1.2B, average zero-shot accuracy rises from 56.80% to 57.31% over standard MTP. Teoh 2026; code.
  • ModernBERT - Updates bidirectional encoder pretraining with rotary positions and alternating local/global attention; evaluated for classification and both single-vector and late-interaction retrieval. Warner et al. 2025.

Sequence generation and translation

  • Sequence-to-sequence LSTMs - Encodes an input sequence into a fixed-dimensional vector and decodes a target sequence with another LSTM. Establishes a jointly trained neural architecture for sequence transduction. Sutskever et al. 2014.
  • Neural attention for translation - Learns soft alignments so each generated word can depend on relevant source positions. Relieves the fixed-vector bottleneck of early encoder-decoder translation models. Bahdanau et al. 2015.
  • Transformer architecture - See Neural Networks for self-attention, positional representations and architectural follow-ups.
  • T5 - Expresses classification, question answering, translation, and other tasks in a common text-to-text format. Compares pretraining objectives, architectures, datasets, and transfer procedures within that framework. Raffel et al. 2020.
  • BART - Pretrains an encoder-decoder by reconstructing text corrupted with noise, including span masking and sentence reordering. Connects bidirectional encoding with left-to-right generation. Lewis et al. 2020.
  • Vision-language generation - Flamingo and LLaVA condition language generation on images; see Multi-View Learning for architectures and visual instruction tuning.

Multilingual learning

  • XLM-R - Pretrains a masked language model across 100 languages. Studies cross-lingual transfer and the tradeoff between sharing representations and dividing finite model capacity among languages. Conneau et al. 2020.
  • mT5 - Extends text-to-text pretraining to multilingual data. Studies cross-lingual task transfer and accidental translation, where generated answers switch away from the requested language. Xue et al. 2021.

Scaling and in-context learning

  • Language-model scaling laws - Fits loss against model size, data and compute. In the tested regime, compute-optimal parameter count grows faster than token count, favoring larger models stopped before convergence. Kaplan et al. 2020.
    • Compute-optimal training - Chinchilla finds roughly equal parameter/token scaling, near 20 tokens per parameter. Refitting its third procedure corrects premature fit termination and rounded coefficients, widens uncertainty, and agrees with its other two methods. Hoffmann et al. 2022, Besiroglu et al. 2024.
  • GPT-3 and few-shot prompting - Studies task demonstrations supplied in the prompt without task-specific gradient updates. Distinguishes in-context conditioning from fine-tuning. Brown et al. 2020.
  • Concentrated data repetition - Repeating a small subset can worsen test loss despite 90% unique training tokens. At the worst repetition level, an 800M model approaches a 340M model’s loss. Copying and induction-head measures also degrade. Hernandez et al. 2022.
  • Data-constrained scaling - At fixed compute, up to four epochs of repeated data yield nearly the same loss as fresh data in tested language models. A fitted law discounts repeated tokens, capturing diminishing returns but not the observed loss increases at heavy repetition. Muennighoff et al. 2023; code.
    • Repetition and overfitting - Adds an overfitting penalty to data-constrained scaling. At three tested data/compute budgets, its prescribed larger models with fewer epochs achieve lower perplexity than Chinchilla and the earlier effective-data law. Lovelace et al. 2026.
    • Compute-data scaling - Models the diminishing value of repeated or paraphrased tokens. On Dolma-3, a fit on 14M/30M models predicts 60M-600M losses with log-loss RMSE 0.079. Only learning rate and decay are tuned; larger-model advice is extrapolation. Qin et al. 2026.
  • Joint MoE scaling - At 1.1B total parameters, 2- and 4-expert MoEs reach lower loss than a dense model over a tested FLOPs range, using more tokens at matched compute. The dense model is past its compute-optimal token count. Costs use FLOPs, not measured time. Ludziejewski et al. 2025.
  • Data filtering and compute - On 670M-10B-token Common Crawl pools, sufficiently trained dense models outperform tested filtered subsets in language-modeling loss. Filtering still helps at low compute; full-crawl predictions extrapolate beyond the experiments. Mohri et al. 2026.
  • Hyperparameter transfer - See Optimization for width transfer, token-horizon effects and joint learning-rate/batch-size scaling.

Instruction and preference learning

  • Instruction tuning / FLAN - Fine-tunes on multiple tasks expressed as natural-language instructions and evaluates unseen task types. Studies how instruction format, task diversity, and model scale affect transfer. Wei et al. 2022.
  • InstructGPT - Combines supervised demonstrations with preference rankings and reinforcement learning. Studies following instructions on a specified prompt distribution; preference training does not eliminate factual errors. Ouyang et al. 2022.
  • Constitutional AI - Uses written principles to guide model critiques, revised responses, and AI-generated preference feedback. Combines supervised and reinforcement learning while retaining human choices about the principles. Bai et al. 2022.
  • Direct Preference Optimization - Reparameterizes a reward-based preference objective as direct policy training with a classification loss. Uses preference pairs without a separate reward-model fitting stage or online sampling during fine-tuning. Rafailov et al. 2023.
  • Unified training - Fits static text and reward-checked responses with one weighted likelihood loss. Mixing beats staged training at larger estimated FLOP budgets in a 124M puzzle model. In a 0.76B model, it reduces forgetting but lowers single-attempt coding accuracy. Song et al. 2026; code.

Sentence representations and retrieval

Sentence and document vectors

  • Sentence-BERT - Siamese/triplet training makes BERT sentence vectors usable with cosine similarity, avoiding joint query-document encoding. Reimers and Gurevych 2019; code.
  • SimCSE - Learns sentence embeddings contrastively using dropout views, or supervised entailment positives and contradiction negatives. Studies how these objectives shape embedding alignment and uniformity. Gao et al. 2021.
  • E5 - Contrastive pretraining on weakly supervised text pairs; evaluated both without labeled-data fine-tuning and after supervised fine-tuning. Wang et al. 2022; code.
  • Qwen3 Embedding - Adapts language-model backbones for instruction-aware multilingual text embeddings. Zhang et al. 2025; code.

Passage retrieval and token scoring

  • Dense Passage Retrieval - Learns separate question and passage encoders for dense-vector retrieval. Trains from question-passage examples to select contexts for open-domain question answering. Karpukhin et al. 2020.
  • ColBERT - Independently encodes queries and passages into token vectors; sums each query token’s maximum similarity to a passage token. Late interaction retains multiple vectors, whereas late chunking pools each chunk. Khattab and Zaharia 2020; code.
  • SPLADE - Learns sparse lexical representations with explicit sparsity regularization and saturated term weights. Supports retrieval with inverted indexes and an effectiveness-sparsity tradeoff. Formal et al. 2021.
  • BGE-M3 - Jointly supports dense vectors, sparse lexical weights, and token multivectors through self-knowledge distillation. Chen et al. 2024; code.

Document context and chunk pooling

  • Late chunking - Encodes the document before pooling spans into chunk vectors. Across three encoders and four BEIR datasets, mean nDCG@10 rises from 52.2 to 54.0 with 256-token chunks; individual settings include ties and regressions. Günther et al. 2024.
  • pplx-embed-context - Combines diffusion-pretrained bidirectional encoders, contrastive learning, and late chunking for document-context-aware passage vectors. Eslami et al. 2026.

Retrieval and generation

  • Retrieval-Augmented Generation - Combines a pretrained sequence generator with a learned retriever and an external passage index. Compares retrieval shared across a response with retrieval varying by token; retrieved context does not guarantee factuality. Lewis et al. 2020.
  • RAGTruth - Annotates unsupported or contradictory spans in nearly 18,000 retrieval-augmented responses. Demonstrates that providing retrieved evidence does not ensure faithful generation; retrieval quality and support for generated claims need separate evaluation. Niu et al. 2024.
  • GraphRAG - Builds entity graphs and community summaries for corpus-wide questions. On two corpora, LLM judges rate its answers more comprehensive than vector RAG’s. Gains shrink against summaries made directly from source chunks without a graph. Edge et al. 2024; code.
  • HippoRAG - Uses personalized PageRank over an LLM-extracted graph to retrieve passages across multiple hops. With ColBERTv2, top-five passage recall on 2Wiki rises from 68.2% to 89.1%. On HotpotQA it falls from 79.3% to 77.7%. Gutiérrez et al. 2024; code.
    • HippoRAG 2 - Adds passage nodes and LLM-filtered query-to-triple matching. With a Llama-3.3-70B reader, 2Wiki answer F1 rises from 61.5 with NV-Embed-v2 retrieval to 71.0. On HotpotQA it changes from 75.3 to 75.5. Gutiérrez et al. 2025; code.
  • Zep / Graphiti - Tracks fact validity intervals and superseded edges. On LongMemEval, GPT-4o accuracy rises from 60.2% with full history to 71.2% with retrieved memory, but falls from 94.6% to 80.4% on questions about the assistant’s statements. Rasmussen et al. 2025; code.
  • Direct corpus interaction - Lets an agent search raw files with terminal tools. On 100 BrowseComp-Plus queries with GPT-5.4 nano, accuracy rises from 45% with embedding retrieval to 73%, while agent cost roughly doubles and gold-document recall falls. Li et al. 2026.
  • RAG vs. GraphRAG - With Llama-3.1-8B, local GraphRAG improves HotpotQA F1 (60.04 to 61.66) but trails passage retrieval on Natural Questions (63.01 vs 64.78). Extracted graphs omit about a third of answer entities in both corpora. Han et al. 2026; code.

Long-context models

  • Transformer-XL - Combines segment-level recurrence with a positional encoding scheme to extend context across segments. Reuses prior representations while addressing fragmentation from fixed-length training windows. Dai et al. 2019.
  • Longformer - Combines local sliding-window attention with task-specific global attention. Reduces attention cost for long documents when window size and the number of global positions remain bounded. Beltagy et al. 2020.
  • Lost in the Middle - Finds position-dependent failures in long-context question answering and key-value retrieval. In the studied models, relevant information near the middle can be harder to use than information near either end. Liu et al. 2024.

Decoding and reasoning procedures

  • Nucleus sampling - Samples from a dynamically selected high-probability portion of the next-token distribution. Studies how decoding changes repetition and diversity even when the underlying language model is unchanged. Holtzman et al. 2020.
  • PICARD constrained decoding - Rejects tokens that violate incremental parsing constraints during generation. Improves validity and text-to-SQL performance on Spider and CoSQL; a syntactically valid query can still answer the wrong question. Scholak et al. 2021.
  • Chain-of-thought prompting - Supplies worked reasoning examples in the prompt before asking for a solution. Studies arithmetic, commonsense, and symbolic tasks; generated explanations are not themselves a correctness check. Wei et al. 2022.
    • Self-consistency - Samples multiple reasoning paths and aggregates their answers instead of choosing one greedy path. Improves the studied reasoning benchmarks at the cost of additional generation. Wang et al. 2023.
  • ReAct - Interleaves generated reasoning with actions that query external information or interact with an environment. Studies how observations can update task plans instead of relying only on an uninterrupted text completion. Yao et al. 2023.
  • Speculative decoding - A smaller model drafts tokens that the target model verifies in parallel. A sampling correction preserves the target distribution; speed depends on draft cost and acceptance. See multi-token pretraining for in-model draft heads. Leviathan et al. 2023.
  • Process supervision - Trains reward models from human judgments of individual solution steps rather than only final answers; introduces PRM800K for studying process supervision in mathematical reasoning. Lightman et al. 2023.
  • Reflexion - Stores verbal diagnoses of failed attempts as memory for retries, without updating weights. On 50 hard HumanEval tasks translated to Rust, GPT-4 with tests and reflection reaches 68% success, versus 60% without reflection and 52% without tests. Shinn et al. 2023; code.
  • Self-Refine - One model critiques and revises its output using few-shot prompts, without weight updates. On code optimization, ChatGPT optimizes 27.5% of programs with specific feedback versus 24.8% with revision alone. Madaan et al. 2023; code.
  • Self-correction and stopping - On 100 closed-book HotpotQA questions, GPT-4 scores 49% initially and 43% after two critique-and-revision rounds. Using known answer labels to stop revising instead gives 59%. Huang et al. 2024.
  • Reasoning budgets - At matched token or call budgets, self-consistency often beats tested debate and self-evaluated Reflexion on math and QA. On Game24, Tree-of-Thoughts beats self-consistency with GPT-4, but not GPT-3.5. Wang et al. 2024.
  • SCoRe - Trains revisions with two-stage reinforcement learning and correctness rewards. After MBPP training, Gemini 1.0 Pro’s HumanEval accuracy rises from 52.4% on its first attempt to 64.6% after one revision, without test feedback at inference. Kumar et al. 2025.
  • Recurrent-depth language models - Repeat shared layers to refine hidden states before producing tokens. See recurrent depth and adaptive computation for architectures and evidence.

Optimizing language-model programs

  • DSPy - Represents LM pipelines as typed modules and bootstraps demonstrations from successful execution traces. A compiler selects module parameters against a supplied metric; the program structure and evaluator remain design choices. Khattab et al. 2024.
  • TextGrad - Propagates LLM-written critiques through a computation graph to revise prompts, code or other variables. Its textual gradients are feedback, not numerical derivatives. Yuksekgonul et al. 2025.

Text-generation metrics

  • BLEU - Compares generated translations with references using modified n-gram precision and a brevity penalty. A corpus-level overlap metric; it does not directly establish factual or semantic equivalence. Papineni et al. 2002.
    • SacreBLEU and reproducibility - Shows that tokenization and normalization choices can make BLEU scores incomparable. Provides standardized reference processing and reporting to make evaluations easier to reproduce. Post 2018.
  • ROUGE - Evaluates summaries using overlap with reference summaries, including n-grams and sequence-based measures. Different ROUGE variants capture different forms of overlap. Lin 2004.
  • chrF - Uses character n-gram precision and recall for translation evaluation. Avoids dependence on a particular word tokenizer and exposes a parameter controlling the recall-precision balance. Popović 2015.
  • BERTScore - Matches candidate and reference tokens using contextual embedding similarity rather than exact lexical equality. Measures representation-level agreement; it is not a factuality verifier. Zhang et al. 2020.

Benchmarks and behavioral evaluation

  • SQuAD - Tests reading comprehension with questions whose answers are spans of the supplied Wikipedia passage. Measures extractive question answering rather than unrestricted open-domain knowledge. Rajpurkar et al. 2016.
  • MultiNLI - Provides entailment, contradiction, and neutral sentence pairs across genres of written and spoken English. Includes evaluation of transfer across genres. Williams et al. 2018.
  • GLUE - Combines language-understanding tasks with a diagnostic suite for linguistic behavior. Encourages shared representations across tasks rather than optimizing a single dataset. Wang et al. 2019.
    • SuperGLUE - Introduces a harder collection of language-understanding tasks following progress on GLUE. Supplies a toolkit and shared evaluation; aggregate scores still combine distinct abilities. Wang et al. 2019.
  • CheckList - Organizes behavioral tests by linguistic capability and test type. Uses generated cases to reveal failures that aggregate held-out accuracy can conceal. Ribeiro et al. 2020.
  • MMLU - Tests knowledge and problem solving across 57 academic and professional subjects. Supports per-subject analysis; performance on these questions is not a general measure of real-world competence. Hendrycks et al. 2021.
  • BIG-bench - Collects diverse tasks for examining language-model capabilities and scaling. Studies differences across tasks, model sizes, and evaluation metrics instead of treating aggregate improvement as uniform progress. Srivastava et al. 2022.
  • TruthfulQA - Tests whether generated answers reproduce common misconceptions. Targets a particular source of falsehood; passing the benchmark does not establish truthfulness on arbitrary questions. Lin et al. 2022.
  • HELM - Evaluates language models across shared scenarios and multiple criteria, including accuracy, calibration, robustness, bias, toxicity, and efficiency. Makes coverage gaps and metric tradeoffs explicit. Liang et al. 2023.
  • Memory consolidation failures - Repeatedly distilling ground-truth solutions makes previously solved ARC-AGI tasks fail for GPT-5.4. Raw-episode stores remain competitive with consolidators in WebShop, ALFWorld and AppWorld; compression can discard useful evidence. Zhang et al. 2026.
  • Raw-history memory - On MemoryAgentBench with GPT-4o-mini, tested additions of fact extraction, summary augmentation or graph expansion to dense retrieval with reranking lower accuracy. Raw-RAG trails two structured systems on PersonaMem-v2; these are offline evaluations. Bao et al. 2026; code.
  • Thinking and refusal - In four moderate-size open reasoning models, tested safety defenses often trade less harmful compliance for more benign refusals. Thinking-prefix and defense interventions concern refusal behavior, not every form of safety or frontier-scale reasoning. Ri et al. 2026.

Software

  • Transformers - Provides a library of pretrained Transformer architectures and task interfaces. The paper describes reusable implementations for NLP research and applications, rather than a single model release. Wolf et al. 2020.
  • Stanza - Provides a multilingual NLP pipeline for tokenization, morphology, syntax, and named entities. Packages neural processing components for text analysis across languages. Qi et al. 2020.