GGUF and llama.cpp

In short: llama.cpp is an open-source program that runs language models efficiently on ordinary computers. GGUF is the file format in which the models are stored.

In more detail: llama.cpp runs on CPU, NVIDIA (CUDA), AMD (ROCm, Vulkan) and Apple (Metal). A GGUF file contains the weights (usually quantised), the tokenizer and metadata in one file. Programs such as Ollama, LM Studio and Jan build on it.

In Depth

• File names: model-q4_k_m.gguf reveals the quantisation.
• Putting layers on the GPU: With a parameter (-ngl) you set how many layers go on the card (offloading).
• Server: llama.cpp can offer an OpenAI-compatible interface.
• Licence: MIT. The models have their own licences (open models and their licences).

See also: Hugging Face, Quantisation