Video generation with AI

In short: AI video generators create moving images from text or images. Technically they are mostly diffusion models that treat time as an additional dimension.

Basic idea

In image generation with diffusion models noise is denoised step by step into an image. For video a whole stack of frames is denoised at once. The model must learn that neighbouring frames fit together.

• Temporal attention: besides attention within one frame the model looks across several frames.
• Latent space: the video is first compressed strongly (see Autoencoder and VAE), the model works on this smaller representation and a decoder turns it back into pixels.
• Conditioning: a text (or a start image) steers what is shown.

Difficulties

1. Consistency: faces, clothes and objects should stay the same from frame to frame.
2. Physics: the model knows no real laws, only learned patterns. Things can slide through each other.
3. Length and compute: every extra second means many more frames and a lot of compute time.
4. Misuse: very realistic videos make forgeries easier, see Deepfakes and AI detection and C2PA and provenance for AI content.

See also: Multimodal models