Distributed Learning and Parallel Training

Parallelism and synchronization

  • Data-parallel SGD - Replicates the model across workers, computes gradients on different data and synchronizes updates. PyTorch DDP averages gradient contributions; effective batch size, loss reduction and synchronization determine the update. docs.
  • HOGWILD! - Runs lock-free SGD updates in shared memory. Recht et al. analyze sparse update conflicts; its assumptions do not imply that arbitrary dense, asynchronous distributed training behaves like serial SGD. Niu et al. 2011.
  • DistBelief / Downpour SGD - Dean et al. combine model partitioning with asynchronous training replicas. Downpour lets workers update shared parameters without lockstep iterations, trading synchronization for delayed parameter information. Dean et al. 2012.
  • Parameter servers - Li et al. separate workers that compute updates from servers holding shared parameters. Flexible consistency models support asynchronous execution; consistency and fault tolerance are explicit system choices. Li et al. 2014.
  • Decentralized SGD - Workers mix parameters through a communication graph rather than a central server. Lian et al. identify bandwidth/latency regimes favoring this approach; graph connectivity and mixing affect convergence. Lian et al. 2017.
  • Local SGD - Workers take several local steps before averaging parameters. Stich derives communication reductions for analyzed convex problems; less frequent synchronization changes both communication cost and optimization behavior. Stich 2019.
  • Backup-worker SGD - Waits for the first m workers and discards stale gradients. With fixed heterogeneous worker speeds and identical gradient distributions, an appropriately chosen m achieves optimal time complexity within a logarithmic factor. Rapid speed changes can defeat it. Begunov and Tyurin 2026.

Communication compression

  • QSGD - Quantizes and encodes gradient updates, trading communication bits against added estimation variance. Alistarh et al. provide convergence results under assumptions; compression ratio alone does not determine time to accuracy. Alistarh et al. 2017.
  • Error feedback - Accumulates compression residuals and adds them to later updates. Karimireddy et al. give counterexamples for plain signSGD and show how error feedback repairs analyzed failures; compressor conditions still matter. Karimireddy et al. 2019.
  • PowerSGD - Approximates matrix gradients by low-rank factors compatible with all-reduce. Vogels et al. report wall-clock gains against an optimized SGD backend on studied CNN/LSTM workloads, accounting for compression overhead. Vogels et al. 2019.

Partitioning models and training state

  • PipeDream - Overlaps forward and backward work across stages and versions weights for consistent passes. Its asynchronous pipeline schedule differs from GPipe’s minibatch-level synchronization. Harlap et al. 2018.
  • Tensor parallelism / Megatron-LM - Splits transformer layer computations across devices. Shoeybi et al.’s intra-layer parallelism complements pipeline partitioning; communication occurs within layer execution rather than only between complete replicas. Shoeybi et al. 2019.
  • GPipe - Partitions a sequence of layers into stages and divides minibatches into microbatches. Pipelined execution increases model capacity across devices; stage imbalance and pipeline bubbles affect utilization. Huang et al. 2019.
  • ZeRO - Partitions optimizer states, then gradients and parameters across data-parallel workers in successive stages. This reduces replicated training state; activations and temporary buffers remain separate memory costs. Rajbhandari et al. 2020.
    • ZeRO-Infinity - Extends sharded training with GPU, CPU and NVMe memory plus data movement. Offloading expands capacity, but useful throughput depends on bandwidth and overlap; fitting a model does not establish fast training. Rajbhandari et al. 2021.
  • Zero Bubble pipelines - Separate backward input-gradient and weight-gradient work to fill idle slots while retaining synchronous updates. The paper reports up to 23% throughput gain over 1F1B at similar memory limits; schedules depend on stage timings and memory. Qi et al. 2023.

Activation memory, attention and experts

  • Sequence parallelism - Shards operations such as layer normalization and dropout alongside tensor parallelism. Korthikanti et al. combine this with selective recomputation to reduce replicated activations; optimizer-state sharding addresses a different cost. Korthikanti et al. 2023.
  • Ring Attention - Distributes sequence blocks and circulates keys and values for exact blockwise attention. Overlapping transfers with computation extends context capacity; hiding communication depends on block size, device throughput and interconnect bandwidth. Liu et al. 2023.
  • MegaBlocks - Implements mixture-of-experts layers with block-sparse kernels so uneven token routing needs neither token dropping nor padded dense batches. Variable workloads use sparse matrix multiplication instead of equal-sized batches for every expert. Gale et al. 2023.

Automatic parallelization

  • GSPMD - Infers distributed tensor partitioning from a small set of sharding annotations and transforms the computation graph. Supports mixed parallelism while preserving a largely single-device programming interface. Xu et al. 2021.
  • Alpa - Searches hierarchical inter- and intra-operator parallelization plans, combining data, operator and pipeline choices. Zheng et al. 2022.

Scheduling and recovery

  • Gradient coding - Replicates data blocks and encodes gradients so the server can recover a full gradient despite a bounded number of stragglers. Trades redundant worker computation for synchronous progress; recovery depends on the code’s tolerance. Tandon et al. 2017.
  • Pollux - Jointly adapts GPU allocation and training configuration using goodput: throughput adjusted for statistical efficiency. Its cluster experiments reduce average completion time, linking resource scheduling to useful training progress rather than steps alone. Qiao et al. 2020.
  • CheckFreq - Adapts checkpoint frequency from online profiling and overlaps persistence with training. Saving data-loader state preserves per-epoch sample coverage after recovery; checkpoint cost and lost work must both enter training-time comparisons. Mohan et al. 2021.

Scaling evidence and measurement

  • Large-batch ImageNet training - Goyal et al. match small-batch ResNet-50 accuracy at batch size 8,192 using learning-rate scaling and warmup. This is a tested training recipe, not a universal rule for scaling batch size. Goyal et al. 2017.
  • Gradient noise scale - Estimates the batch size beyond which extra examples give diminishing progress per update. Gradient variability relative to the mean predicts this transition approximately across tested vision, language and reinforcement-learning tasks. McCandlish et al. 2018.
    • Critical batch size by branching - Short continuations at different batch sizes estimate when token efficiency degrades. In OLMo 1B, batch-size warmup with learning-rate scaling reaches slightly lower training loss with 43% fewer updates than the small-batch control. Merrill et al. 2025.
  • MLPerf Training - Compares time to reach specified quality targets under controlled workload and timing rules. Repeated runs address stochastic training time; throughput gains that delay convergence need not improve this metric. Mattson et al. 2020.

Software

  • PyTorch DDP - Synchronizes gradients across replicated models and can overlap communication with backpropagation. It does not shard the model or automatically divide input data; sampling remains part of the training setup. docs.
  • PyTorch FSDP - Shards parameters, gradients and optimizer state across workers, gathering parameters when needed. Sharding, resharding and CPU-offload choices trade memory against communication and transfer costs. docs.
  • Horovod - Uses collective communication, including ring reduction, to support distributed training with limited changes to training code. Sergeev and Del Balso’s original paper describes TensorFlow integration. Sergeev and Del Balso 2018.