Speech and audio: speech recognition and synthesis

In short: Speech recognition (ASR) turns audio into text, speech synthesis (TTS) text into audio. Both are now done with neural networks, usually transformers.

In more detail: Audio is a waveform with, for example, 16,000 samples per second. That is too fine for models, so it is first converted into a spectrogram (frequencies over time) or into audio tokens.

In detail

• Spectrogram: a mel spectrogram maps frequencies the way the ear perceives them. The network sees an “image” of the sound.
• Whisper (OpenAI): encoder-decoder transformer trained on hundreds of thousands of hours of speech, recognises many languages and translates.
• CTC: loss function that connects audio frames to text without fixed alignment.
• TTS: text → phonemes/tokens → acoustic model → vocoder (waveform). Newer models generate audio tokens directly like a language model.
• Voice cloning: a voice can be reproduced from a few seconds of audio. Misuse (fraud, deepfakes) is a serious risk, see AI phishing.
• Neural audio codecs: compress audio into discrete tokens (EnCodec, SoundStream).
• Music models: generate music with similar methods.
• Real time: voice assistants need low latency, hence streaming and voice activity detection.

See also: Vision transformer and CLIP, Encoder, decoder and the model families (BERT, GPT, T5)