EMZETT.AI and music: how models create sound
In short: Music AI creates sound from text or melodies. There are two main ways: predict audio as a sequence of tokens (like a language model) or create audio by diffusion, similar to images.
The basic problem
Sound has a very high data rate: 44,100 samples per second. A three-minute song is almost eight million values. Predicting value by value directly is impractical.
Solution: compress
An audio codec (an autoencoder) turns sound into a much shorter sequence of discrete codes and back. The models work on that:
- Token models: a transformer predicts the codes one after another, conditioned on text (“calm jazz with piano”), like a language model.
- Diffusion models: music arises step by step from noise in the compressed space, see image generation with diffusion models.
Other tasks
Imitate voices, separate stems (vocals, drums, bass), recognise chords, mastering, transitions between songs.
Open questions
- Copyright: which recordings may be used for training? See Training data, copyright and privacy in AI.
- Voice clones and forgeries: Deepfakes and AI detection, C2PA and provenance.
- What counts as own creation, and how are musicians involved?
See also: Speech and audio