Audio and Speech

Speech, music and environmental sounds vary in timing, speaker, source and recording conditions. This review covers learning to align, recognize, represent, generate and compress audio. Generic network architectures belong in Neural Networks, and joint audio-text objectives in Multi-View Learning.

Alignment and speech recognition

  • Dynamic time warping - Aligns sequences by optimizing a nonlinear time-warping path with dynamic programming. Sakoe-Chiba studies path constraints for spoken-word matching; this is an alignment method, not a forecasting model. Sakoe and Chiba 1978.
  • CTC - Trains sequence recognition without frame-level alignments by summing paths through labels and a blank symbol. Frame output probabilities factor independently given the input sequence. Graves et al. 2006.
    • RNN Transducer - Adds a prediction network conditioned on previous output labels to an acoustic transcriber. Summing over monotonic alignments allows joint acoustic and label-sequence modelling without a supplied frame-level alignment. Graves 2012.
  • Whisper - Trains on 680,000 hours of multilingual audio and web transcripts for recognition and translation. Zero-shot tests across speech datasets show broader transfer than models tuned to a single benchmark, despite similar LibriSpeech accuracy. Radford et al. 2023.
  • Hidden-state recognition - See Modeling for hidden Markov models and their inference problems.

Self-supervised acoustic representations

  • wav2vec 2.0 - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. Baevski et al. 2020.
  • HuBERT - Predicts offline cluster labels for masked speech segments, then reclusters learned features. Consistent targets matter more than accurate phonetic labels. Pretraining matches or improves wav2vec 2.0 across tested LibriSpeech label budgets. Hsu et al. 2021.
  • BEATs - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. Chen et al. 2023; code.

Sound classification and cross-modal matching

  • Audio Spectrogram Transformer - Treats overlapping spectrogram patches as Transformer tokens. ImageNet initialization substantially improves audio classification over training from scratch, showing that image pretraining can transfer to time-frequency representations. Gong et al. 2021.
  • Audio-text alignment - See Multi-View Learning for CLAP and text-conditioned audio retrieval.

Speech and music synthesis

  • Waveform generation - See Generative Models for WaveNet and autoregressive waveform modelling.
  • Tacotron 2 - Predicts mel spectrograms from characters and uses a WaveNet vocoder to synthesize speech. In a single-speaker English test, listeners rate it 4.53 out of 5 versus 4.58 for recordings. Shen et al. 2018.
  • MusicGen - A single autoregressive Transformer generates music as EnCodec tokens conditioned on text or melody. Delays between codebook streams let it predict several streams in parallel without separate coarse and fine generative models. Copet et al. 2023.

Neural audio compression

  • SoundStream - A convolutional encoder-decoder and residual vector quantizer compress speech, music and other audio. Dropping quantizer layers during training lets one model operate at 3-18 kbps with little quality loss against separate fixed-rate models. Zeghidour et al. 2022.
    • EnCodec - Extends residual-quantizer audio coding with a multiscale spectrogram discriminator and loss balancing by gradient contribution. Listening tests cover speech and music in mono and stereo. A token language model further reduces bitrate. Défossez et al. 2023.