Collection of open machine learning papers
Solution learning (2017). Approximates a PDE solution from equations, boundary/initial conditions and observations. The original PINN formulation fits a coordinate-to-state function; solving a new problem can require another optimization.
Operator learning (2021). Learns maps between function spaces, such as coefficients to solution fields. Shared parameters across discretizations distinguish the operator formulation from a fixed-grid predictor; approximation and numerical discretization remain separate errors.
Hybrid models (2020). Combines known differential equations with trainable missing terms. The supplied physics restricts what must be learned; fitting observed trajectories does not uniquely identify an unknown mechanism.
Physics-informed neural networks (2017). Uses automatic differentiation to penalize differential-equation residuals alongside data and boundary losses. Residual penalties enforce equations approximately at sampled points; they are not exact conservation constraints.
Deep Ritz (2017). Minimizes a variational energy over neural trial functions. Unlike strong-form residual fitting, it uses the problem’s variational formulation; the chosen functional and admissible boundary treatment determine the solution sought.
Deep Galerkin Method (2017). Trains a neural solution on randomly sampled space–time points using PDE, initial and boundary residuals. Avoids a tensor-product mesh; reported high-dimensional examples do not imply dimension-independent accuracy for arbitrary PDEs.
Deep BSDE solvers (2017). Recasts suitable parabolic PDEs as backward stochastic differential equations. Neural networks approximate solution gradients along simulated paths; terminal-condition mismatch supplies the training loss.
hp-VPINNs (2020). Combines neural trial functions with local polynomial test functions and domain decomposition. Weak residuals and h/p refinement connect physics-informed training to variational numerical methods.
Residual-adaptive refinement (2019). Adds collocation points where PDE residuals are large. DeepXDE uses this to focus PINN training; a small residual still needs a stability argument to imply small solution error.
PINN failure modes (2021). Convection, reaction and diffusion examples expose optimization failures despite sufficient network expressivity. Curriculum regularization and sequential time training improve the tested cases; adding a PDE penalty alone is insufficient.
Unbalanced loss gradients (2020). Wang et al. trace training failures to stiffness and unequal backpropagated gradients across loss terms. Gradient-statistics-based balancing improves their examples; the equation and boundary losses need compatible optimization scales.
PINN neural tangent kernels (2020). Infinite-width analysis reveals unequal convergence rates for residual and boundary losses. Eigenvalue-based reweighting addresses this imbalance in experiments; the limiting-kernel theory is not a finite-network convergence guarantee.
PINN loss curvature (2024). Links differential-operator conditioning to loss curvature. Adam+L-BFGS beats either alone; Nyström-preconditioned Newton-CG improves stalled solutions in the tested PDEs. This compares PINN optimizers, not neural and classical solvers.
Time-ordered PINN training (2022). Weights later-time residuals according to earlier-time errors. This respects the order of an initial-value problem and improves chaotic-system benchmarks; “causal” here refers to temporal evolution, not causal discovery.
Residual-to-solution error bounds (2022). De Ryck et al. bound Navier–Stokes PINN errors using training error, network size and quadrature. Regularity and PDE-specific stability enter the result; approximation capacity alone does not guarantee successful optimization.
DeepONet (2019). A branch network encodes input-function samples; a trunk network encodes output locations. Their combination predicts an output function. The operator approximation theorem does not bound optimization or generalization error.
Fourier Neural Operator (2020). Learns integral kernels through Fourier-mode transformations. Demonstrates resolution transfer on Burgers, Darcy and Navier–Stokes tasks; shared weights across grids do not guarantee accuracy on unseen physical regimes.
Physics-Informed Neural Operator (2021). Combines operator training on coarse data with PDE constraints evaluated at finer resolution. Connects FNO-style amortized prediction with equation-based refinement; separate pretraining cost from per-instance inference.
Learned discretizations (2018). Learns spatial-derivative approximations inside a numerical time integrator. Bar-Sinai et al. obtain accurate coarse-grid dynamics on tested 1D equations; the learned component augments a solver rather than replacing its entire structure.
Reduced-order operator inference (2016). Fits linear or low-order polynomial dynamics by least squares on projected trajectories, without full solver operators. A reduced state basis limits the model; estimating dynamics adds information beyond compressing snapshots.
Solver-in-the-loop correction (2020). Backpropagates through solver rollouts to train state corrections. Earlier corrections change later inputs, unlike fixed-trajectory training. Gains depend on dynamics: the forced-Burgers experiment shows only a small advantage.
MeshGraphNets (2020). Passes messages over simulation meshes to predict physical updates, with adaptive meshing in studied systems. Evaluated on cloth, solids and fluids; rollout accuracy and remeshing behavior matter beyond one-step prediction.
Particle graph simulators (2020). Represents particles as graph nodes and learns interactions by message passing. Training with perturbed states reduces accumulated rollout error in the reported simulations; long-horizon performance differs from one-step fit.
PDE pretraining: Poseidon (2024). Pretrains a multiscale operator transformer on fluid equations, then adapts to 15 tasks. Uses time conditioning and semigroup-based data pairing; transfer to new PDEs includes downstream training, not universal zero-shot solving.
Neural ODEs (2018). Parameterizes a continuous-time vector field and integrates it with an ODE solver. The solver, tolerances and gradient calculation become part of the learned system; a neural vector field need not preserve physical invariants.
Latent ODEs (2019). Uses continuous latent dynamics and an ODE-RNN inference network for irregular observations. Can model observation times with a Poisson process; irregular sampling support does not itself identify the underlying physical state.
Hamiltonian neural networks (2019). Learns a scalar Hamiltonian and derives dynamics through canonical derivatives. Conservation follows for the learned continuous system; numerical integration can still introduce energy drift.
Lagrangian neural networks (2020). Learns a Lagrangian and obtains dynamics from Euler–Lagrange equations. Uses positions and velocities without requiring canonical momenta; the learned dynamics and numerical solver still need validation.
SINDy (2015). Selects a sparse set of candidate functions to explain observed state derivatives. Compact equations depend on the chosen variables, function library, derivative estimates and excitation in the observed trajectories.
PDE-FIND (2016). Extends sparse equation discovery to spatial derivatives and nonlinear terms. Multiple initial conditions can distinguish equations that explain the same single trajectory; low regression error alone need not identify a unique PDE.
Bayesian PINNs (2020). Combines equation constraints and noisy observations in Bayesian inference. HMC gives better uncertainty estimates than tested variational/dropout alternatives in the paper; calibration remains specific to likelihood, prior and model adequacy.
Simulation-based inference (2019). Uses simulations when likelihood evaluation is unavailable, including learned likelihoods, likelihood ratios and posteriors. Inference concerns parameters under a simulator; posterior confidence does not validate the simulator itself.
PINNs versus finite elements (2023). FEM was generally faster at equal or better accuracy in the tested PDEs; 1D Schrödinger results were inconclusive. Faster PINN point evaluation appeared only for 3D Poisson, after training.
Mass-balance audit (2023). In two steady-state 1D groundwater cases, PINNs selected from 10,800 configurations each fit hydraulic heads well but retain larger mass-balance errors than finite volumes. Accurate fields did not ensure local or global conservation.
PDEBench (2022). Compares forward and inverse PDE tasks with FNO, U-Net and PINN baselines. Metrics include boundary, conservation and frequency-dependent errors; a single aggregate field error can conceal physically relevant failures.
The Well (2024). Provides 16 diverse simulation datasets with common interfaces. Baselines expose differences across physical systems; evaluate rollouts and dataset-specific dynamics rather than treating performance on one fluid equation as general simulation ability.
Geometric deep learning · Symbolic regression · Model predictive control · Bayesian inference · Neural nets · Evaluation
Star Issue