Privacy-Preserving Machine Learning

Protected information and trust models

  • Differential privacy - Bounds how output distributions change between neighboring datasets. The adjacency relation defines the protected unit; epsilon and delta describe a mechanism’s guarantee, not the probability that an individual record is safe. Dwork and Roth 2014.
  • Record-level versus user-level privacy - Changing one record and changing all records of one person are different adjacency relations. Group privacy degrades with group size; a record-level guarantee cannot be reported unchanged as protection for a whole user. Dwork and Roth 2014.
  • Local differential privacy: RAPPOR - Randomizes client reports using randomized response and Bloom filters for string statistics. Repeated-report protection needs separate accounting; local privacy trades statistical efficiency for reduced trust in a collector. Erlingsson et al. 2014.
  • Shuffle-model amplification - Anonymously shuffling locally randomized reports can strengthen the central DP guarantee. Amplification relies on the shuffling protocol and adversary’s view; removing names from otherwise revealing records is not the same mechanism. Erlingsson et al. 2019.
  • Secure aggregation - Lets a server recover aggregate client updates without seeing individual updates under stated protocol assumptions. Protects messages during computation; the aggregate or released model may still reveal training information. Bonawitz et al. 2017.
  • Encrypted inference: CryptoNets - Evaluates a transformed neural network on homomorphically encrypted inputs and returns encrypted predictions. Protects the inference input from the evaluator; this is a different target from DP protection for training examples. Gilad-Bachrach et al. 2016.

Noise mechanisms and privacy accounting

  • Sensitivity-calibrated mechanisms - Laplace and Gaussian mechanisms calibrate noise to how much a query can change on neighboring data. Bounding contributions is essential; adding noise without sensitivity analysis does not establish differential privacy. Dwork and Roth 2014.
  • Composition and post-processing - Repeated private accesses consume privacy budget; computation using only a private release preserves its guarantee. New access to raw private data is not post-processing, even when it occurs during model selection. Dwork and Roth 2014.
  • Zero-concentrated differential privacy - Controls privacy-loss tails through Rényi-divergence bounds across orders. Gives convenient composition and group-privacy analysis; its parameter must be converted before comparison with epsilon-delta DP. Bun and Steinke 2016.
  • Rényi differential privacy - Bounds Rényi divergence between outputs on neighboring datasets. Privacy costs add at each order under composition, then convert to an epsilon-delta guarantee; the selected order affects the resulting bound. Mironov 2017.
  • Subsampling amplification - Bounds privacy after sampling a subset before applying a private mechanism. Sampling probability and scheme enter the theorem; an accountant must match how the implementation actually forms training batches. Wang et al. 2019.
  • Privacy-loss random-variable accounting - Numerically composes privacy-loss distributions with bounded approximation error. Tightens accounting for repeated mechanisms; evaluates a specified mechanism without auditing whether code implements it. Gopi et al. 2021.
  • Gaussian differential privacy - Describes privacy through hypothesis-testing tradeoffs between neighboring outputs. Gaussian DP provides a one-parameter family and composition approximations; asymptotic approximations and finite-composition bounds should be distinguished. Dong et al. 2022.
  • Private hyperparameter selection - Non-private tuning can leak information even when the final training run is private. Papernot and Steinke analyze search procedures under Rényi DP; reported budgets must account for the selection procedure and its assumptions. Papernot and Steinke 2022.
  • Adaptive f-DP filter failure - Tensor products of realized privacy trade-off curves can underestimate loss when mechanism choices depend on prior outputs, even for subsampled Gaussian mechanisms. Standard composition accounting need not yield a valid adaptive filter. Tran et al. 2026.

Private optimization and knowledge transfer

  • Output and objective perturbation - Adds randomness to parameters or the optimization objective. Chaudhuri et al. prove privacy for regularized ERM under convexity, smoothness and sensitivity conditions; arbitrary neural objectives need separate analysis. Chaudhuri et al. 2011.
  • DP-SGD - Clips per-example gradients, adds Gaussian noise and accounts for sampled iterations. Clipping bounds each example’s contribution; noise scale, sample rate and training steps jointly determine the privacy guarantee. Abadi et al. 2016.
  • Private Aggregation of Teacher Ensembles - Trains teachers on disjoint private partitions and labels public inputs through noisy voting. A student learns from these labels; privacy belongs to the aggregation mechanism, not merely to using an ensemble. Papernot et al. 2017.
    • Selective teacher queries - Adds Gaussian noise to teacher votes and a noisy consensus threshold that can withhold labels. Selective querying reduces disclosure on disagreements; data-dependent privacy accounting includes the threshold test and answered queries. Papernot et al. 2018.
  • User-level private federated learning - Clips and noises user contributions rather than individual-example gradients. McMahan et al. train recurrent language models with user-level DP; grouping, participation sampling and contribution limits define the protected unit. McMahan et al. 2018.
  • DP-FTRL - Uses tree aggregation to perturb cumulative clipped gradients with noise correlated across steps. Avoids reliance on sampling/shuffling amplification; repeated contributions still need accounting. Trades noise structure against optimization and compute. Kairouz et al. 2021.
  • Private synthetic data: AIM - Adaptively selects and privately measures statistical queries, then fits synthetic data to noisy answers. Utility depends on the query workload; synthetic appearance alone is no privacy guarantee, and downstream fidelity needs evaluation. McKenna et al. 2022.

Pretraining and empirical utility

  • Private language-model fine-tuning - Li et al. combine public pretraining, matched objectives and DP-specific tuning; ghost clipping avoids materializing full per-example gradients. Privacy covers the private fine-tuning data, not an unaccounted pretraining corpus. Li et al. 2022.
  • Private vision at scale - Averages gradients over multiple augmentations before clipping each image; large batches and normalization improve tested DP-SGD accuracy. Reported privacy budgets exclude hyperparameter-search costs, so they do not cover the full tuning procedure. De et al. 2022.

Attacks, memorization and empirical auditing

  • Membership inference - Infers whether a candidate record was used for training from model responses. Shokri et al. train shadow-model attacks; success depends on access and data assumptions, while a failed attack does not prove privacy. Shokri et al. 2017.
  • Canary exposure - Inserts rare synthetic sequences and measures how unusually highly the model ranks them. Exposure quantifies tested memorization and extraction risk; it is not a universal certificate for all private information. Carlini et al. 2018.
  • Deep Leakage from Gradients - Optimizes dummy inputs to match shared gradients and reconstructs training data in studied image/text settings. Keeping raw records on-device does not ensure that transmitted gradients conceal them. Zhu et al. 2019.
  • Training-text extraction - Carlini et al. recover verbatim training sequences from GPT-2, including examples appearing in one document. Generation can reveal memorized data; membership classification and extracting an unknown sequence are distinct attack targets. Carlini et al. 2020.
  • Gradient inversion beyond toy settings - Uses magnitude-invariant gradient matching to reconstruct high-resolution images from trained networks. Architecture, batch size and attacker knowledge matter; the experiments refute treating gradient sharing as inherently private. Geiping et al. 2020.
  • Likelihood Ratio Attack - Fits per-example distributions of model confidence with and without membership. Evaluates true positives at low false-positive rates, exposing risks hidden by average attack accuracy; shadow-model assumptions affect the attack. Carlini et al. 2022.
  • One-run privacy auditing - Uses many independently included or excluded examples to audit a mechanism in one training run. Produces empirical privacy lower bounds under the audit’s assumptions; it can expose a claimed bound’s failure, not certify every possible attack. Steinke et al. 2023.

  • Data laundering detection - Rewritten training texts can evade tests that query originals. SDR searches synthetic rewrites to restore detector separation; tests use fine-tuned MIMIR models, likelihood-based scores and known nonmember references, not arbitrary API-only provenance. Li et al. 2026.

Software

  • Opacus - Implements DP-SGD in PyTorch. Its validator rejects BatchNorm, whose batch statistics couple records; alternatives include GroupNorm. Sampling and accounting must still match the executed training procedure. docs.