Scientific Machine Learning

Problems and mathematical structure

  • Solution learning - Approximates a PDE solution from equations, boundary/initial conditions and observations. The original PINN formulation fits a coordinate-to-state function; solving a new problem can require another optimization. Raissi et al. 2017.
  • Operator learning - Learns maps between function spaces, such as coefficients to solution fields. Shared parameters across discretizations distinguish the operator formulation from a fixed-grid predictor; approximation and numerical discretization remain separate errors. Kovachki et al. 2023.
  • Hybrid models - Combines known differential equations with trainable missing terms. The supplied physics restricts what must be learned; fitting observed trajectories does not uniquely identify an unknown mechanism. Rackauckas et al. 2020.

Neural solvers and equation constraints

  • Physics-informed neural networks - Uses automatic differentiation to penalize differential-equation residuals alongside data and boundary losses. Residual penalties enforce equations approximately at sampled points; they are not exact conservation constraints. Raissi et al. 2017.
  • Deep BSDE solvers - Recasts suitable parabolic PDEs as backward stochastic differential equations. Neural networks approximate solution gradients along simulated paths; terminal-condition mismatch supplies the training loss. E et al. 2017.
  • Deep Ritz - Minimizes a variational energy over neural trial functions. Unlike strong-form residual fitting, it uses the problem’s variational formulation; the chosen functional and admissible boundary treatment determine the solution sought. E and Yu 2018.
  • Deep Galerkin Method - Trains a neural solution on randomly sampled space-time points using PDE, initial and boundary residuals. Avoids a tensor-product mesh; reported high-dimensional examples do not imply dimension-independent accuracy for arbitrary PDEs. Sirignano and Spiliopoulos 2018.
  • hp-VPINNs - Combines neural trial functions with local polynomial test functions and domain decomposition. Weak residuals and h/p refinement connect physics-informed training to variational numerical methods. Kharazmi et al. 2021.
  • Residual-adaptive refinement - Adds collocation points where PDE residuals are large. DeepXDE uses this to focus PINN training; a small residual still needs a stability argument to imply small solution error. Lu et al. 2021.

Training failures and error analysis

  • PINN failure modes - Convection, reaction and diffusion examples expose optimization failures despite sufficient network expressivity. Curriculum regularization and sequential time training improve the tested cases; adding a PDE penalty alone is insufficient. Krishnapriyan et al. 2021.
  • Unbalanced loss gradients - Wang et al. trace training failures to stiffness and unequal backpropagated gradients across loss terms. Gradient-statistics-based balancing improves their examples; the equation and boundary losses need compatible optimization scales. Wang et al. 2021.
  • PINN neural tangent kernels - Infinite-width analysis reveals unequal convergence rates for residual and boundary losses. Eigenvalue-based reweighting addresses this imbalance in experiments; the limiting-kernel theory is not a finite-network convergence guarantee. Wang et al. 2022.
  • Time-ordered PINN training - Weights later-time residuals according to earlier-time errors. This respects the order of an initial-value problem and improves chaotic-system benchmarks; “causal” here refers to temporal evolution, not causal discovery. Wang et al. 2022.
  • Residual-to-solution error bounds - De Ryck et al. bound Navier-Stokes PINN errors using training error, network size and quadrature. Regularity and PDE-specific stability enter the result; approximation capacity alone does not guarantee successful optimization. De Ryck et al. 2023.
  • PINN loss curvature - Links differential-operator conditioning to loss curvature. Adam+L-BFGS beats either alone; Nyström-preconditioned Newton-CG improves stalled solutions in the tested PDEs. This compares PINN optimizers, not neural and classical solvers. Rathore et al. 2024.

Learning operators and numerical updates

  • Reduced-order operator inference - Fits linear or low-order polynomial dynamics by least squares on projected trajectories, without full solver operators. A reduced state basis limits the model; estimating dynamics adds information beyond compressing snapshots. Peherstorfer and Willcox 2016.
  • Learned discretizations - Learns spatial-derivative approximations inside a numerical time integrator. Bar-Sinai et al. obtain accurate coarse-grid dynamics on tested 1D equations; the learned component augments a solver rather than replacing its entire structure. Bar-Sinai et al. 2019.
  • Solver-in-the-loop correction - Backpropagates through solver rollouts to train state corrections. Earlier corrections change later inputs, unlike fixed-trajectory training. Gains depend on dynamics: the forced-Burgers experiment shows only a small advantage. Um et al. 2020.
  • Particle graph simulators - Represents particles as graph nodes and learns interactions by message passing. Training with perturbed states reduces accumulated rollout error in the reported simulations; long-horizon performance differs from one-step fit. Sanchez-Gonzalez et al. 2020.
  • DeepONet - A branch network encodes input-function samples; a trunk network encodes output locations. Their combination predicts an output function. The operator approximation theorem does not bound optimization or generalization error. Lu et al. 2021.
  • Fourier Neural Operator - Learns integral kernels through Fourier-mode transformations. Demonstrates resolution transfer on Burgers, Darcy and Navier-Stokes tasks; shared weights across grids do not guarantee accuracy on unseen physical regimes. Li et al. 2021.
    • Physics-Informed Neural Operator - Combines operator training on coarse data with PDE constraints evaluated at finer resolution. Connects FNO-style amortized prediction with equation-based refinement; separate pretraining cost from per-instance inference. Li et al. 2024.
  • MeshGraphNets - Passes messages over simulation meshes to predict physical updates, with adaptive meshing in studied systems. Evaluated on cloth, solids and fluids; rollout accuracy and remeshing behavior matter beyond one-step prediction. Pfaff et al. 2021.
  • PDE pretraining: Poseidon - Pretrains a multiscale operator transformer on fluid equations, then adapts to 15 tasks. Uses time conditioning and semigroup-based data pairing; transfer to new PDEs includes downstream training, not universal zero-shot solving. Herde et al. 2024.
  • Sparse molecular Hessians - Derives a local potential’s Hessian sparsity from its interaction graph, then uses coloring and Hessian-vector products for exact recovery. Linear time and storage require fixed cutoff, fixed message-passing depth and bounded local density. Thiede et al. 2026.
  • Meshfree exterior calculus - Builds conservative operators on point clouds and learns a local flux law. Transfer bounds separate discretization and flux error, but require controlled stability constants and new edge-feature coverage; one solution need not identify the flux. Shaffer et al. 2026.

Dynamics, conservation and discovery

  • Equation discovery - See Symbolic Regression for SINDy, PDE-FIND and sparse-discovery follow-ups.
  • Neural ODEs - Parameterizes a continuous-time vector field and integrates it with an ODE solver. The solver, tolerances and gradient calculation become part of the learned system; a neural vector field need not preserve physical invariants. Chen et al. 2018.
    • Latent ODEs - Uses continuous latent dynamics and an ODE-RNN inference network for irregular observations. Can model observation times with a Poisson process; irregular sampling support does not itself identify the underlying physical state. Rubanova et al. 2019.
  • Hamiltonian neural networks - Learns a scalar Hamiltonian and derives dynamics through canonical derivatives. Conservation follows for the learned continuous system; numerical integration can still introduce energy drift. Greydanus et al. 2019.
  • Lagrangian neural networks - Learns a Lagrangian and obtains dynamics from Euler-Lagrange equations. Uses positions and velocities without requiring canonical momenta; the learned dynamics and numerical solver still need validation. Cranmer et al. 2020.
  • Reservoir dynamics models - Fixed recurrent states and delayed polynomial features provide alternatives to fitting a neural vector field. See reservoir computing for the methods and chaotic-system evidence.

Data assimilation and digital twins

  • Adjoint variational assimilation - Fits a dynamically constrained trajectory to observations across time. A forward model run and backward adjoint compute gradients with respect to the initial state, avoiding one perturbation run per control variable. Le Dimet and Talagrand 1986.
  • Ensemble Kalman filter - Propagates an ensemble through the dynamics and uses its sample covariances to update states from observations. The analysis approximates nonlinear Bayesian filtering; finite ensembles also introduce sampling error. Evensen 1994, Evensen 2003.
  • Filtering connections - See Kalman filtering for linear state estimation and Sequential Monte Carlo for particle filtering and smoothing. Ensemble covariance updates and weighted-particle approximations retain different information about uncertainty.
  • Ensemble Kalman inversion - Repurposes ensemble updates for derivative-free inverse-problem optimization. Estimates remain in the span of the initial ensemble, making its choice a restriction on recoverable parameters or fields. Iglesias et al. 2013.
  • Probabilistic digital twins - Uses a dynamic decision network to update uncertain asset states and guide control. The UAV study calibrates on experimental data; in-flight damage tracking and planning use synthetic observations. Kapteyn et al. 2021.

Inverse problems and uncertainty

  • Simulation-based inference - Uses simulations when likelihood evaluation is unavailable, including learned likelihoods, likelihood ratios and posteriors. Inference concerns parameters under a simulator; posterior confidence does not validate the simulator itself. Cranmer et al. 2020.
  • Bayesian PINNs - Combines equation constraints and noisy observations in Bayesian inference. HMC gives better uncertainty estimates than tested variational/dropout alternatives in the paper; calibration remains specific to likelihood, prior and model adequacy. Yang et al. 2021.

Sequence-to-function genomics

  • Basenji - Uses dilated convolutions to predict quantitative chromatin and transcription profiles from DNA. Enlarging the receptive field improves held-out prediction; variant scores align with measured expression associations across 19 tissues. Kelley et al. 2018; code.
  • Enformer - Uses attention to integrate regulatory DNA up to 100,000 bases away. Across held-out genes, mean correlation with measured transcription rises from 0.81 with Basenji2 to 0.85; enhancer prioritization also improves against CRISPR perturbation data. Avsec et al. 2021; code.
  • Borzoi - Extends Enformer to predict RNA-seq coverage from 524,000-base DNA windows. Coverage supports scoring effects on expression, splicing and RNA processing; adding chromatin-accessibility training improves expression-variant and enhancer-linking predictions. Linder et al. 2025; code.
  • AlphaGenome - Jointly predicts expression, splicing and chromatin features from one million DNA bases, with outputs up to single-base resolution. Matches or exceeds the strongest tested external models on 25 of 26 variant-effect benchmarks. Avsec et al. 2026; code.
    • Variant-effect underestimation - On selected GTEx whole-blood genes, AlphaGenome often gives candidate causal variants weak effects relative to neighboring variants. Underestimation persists in a 1-Mb input check. Drusinsky and Pollard 2026.

Comparative evidence

  • PDEBench - Compares forward and inverse PDE tasks with FNO, U-Net and PINN baselines. Metrics include boundary, conservation and frequency-dependent errors; a single aggregate field error can conceal physically relevant failures. Takamoto et al. 2022.
  • Personal gene-expression prediction - Across 421 people, Enformer and three other models predict differences between genes but have near-zero mean correlation for differences between individuals. Some genes show reversed predicted effects. Huang et al. 2023.
  • PINNs versus finite elements - FEM was generally faster at equal or better accuracy in the tested PDEs; 1D Schrödinger results were inconclusive. Faster PINN point evaluation appeared only for 3D Poisson, after training. Grossmann et al. 2024.
  • Mass-balance audit - In two steady-state 1D groundwater cases, PINNs selected from 10,800 configurations each fit hydraulic heads well but retain larger mass-balance errors than finite volumes. Accurate fields did not ensure local or global conservation. Mamud et al. 2024.
  • The Well - Provides 16 diverse simulation datasets with common interfaces. Baselines expose differences across physical systems; evaluate rollouts and dataset-specific dynamics rather than treating performance on one fluid equation as general simulation ability. Ohana et al. 2024.
  • Perturbation-response prediction - Tested deep models did not outperform simple baselines on gene perturbations. A cell-line follow-up finds gains with metrics focused on perturbation effects. Additive baselines remain strong for near-additive double perturbations. Ahlmann-Eltze et al. 2025, Miller et al. 2026.
  • Neural-solver breakeven - Counts solves needed to repay data-generation and task-training cost against an error-matched classical solver. Some tested periodic PDE/model pairs need hundreds of thousands of calls; others never break even. Foundation-model pretraining is excluded. Zhang et al. 2026.

Software

  • DeepXDE - Implements PINNs, operator networks and adaptive sampling for differential equations. Supports forward and inverse problems; equation residuals, boundary conditions and sampling rules remain user-specified. docs.