Edge AI: models on phones and the Raspberry Pi

In short: Edge AI means a model runs right on the device (phone, laptop, Raspberry Pi) instead of in the cloud. That saves waiting time and protects data, but needs small, cleverly shrunk models.

Why local?

• Privacy: inputs do not leave the device.
• Usable offline and without delay from the network.
• No running costs per request.

How big models become small

1. Smaller models from the start (1 to 8 billion parameters).
2. Quantisation: store weights with only 4 or 8 bits instead of 16. A 7-billion model then needs about 4 GB instead of 14 GB. See Quantisation.
3. Distillation and pruning: model shrinking.
4. Efficient programs: runtimes such as llama.cpp or Ollama use the device’s CPU and GPU well.

Realistic expectations

• On a Raspberry Pi small models run slowly (a few words per second).
• Modern phones have an NPU (AI chips compared) for photo and speech tasks.
• The main bottleneck is memory: the whole model has to fit in.
• Small models make more mistakes than big ones and are weaker with long contexts.

See also: Long contexts