Local AI with Ollama and LM Studio: Language models on your own computer

In short: With Ollama or LM Studio open language models run on your own computer: data stays local, there are no API costs, but quality and speed depend on the hardware.

In more detail: Open models (Llama, Mistral, Qwen, Gemma, Phi, DeepSeek and others) can be downloaded and run locally. This is attractive for privacy, offline use, experiments and learning.

In Depth

Tools

• Ollama: simple command-line/background-service program (Windows, macOS, Linux). ollama run llama3.2 downloads and starts a model; a local API (http://localhost:11434) is OpenAI-compatible.
• LM Studio: graphical interface with model search, chat and local server.
• llama.cpp (basis of many tools, GGUF format), Jan, GPT4All, Open WebUI (browser interface for Ollama), vLLM (server operation).

ollama pull qwen2.5:7b
ollama run qwen2.5:7b "Explain DNS in three sentences."

Hardware and model size

Decisive is the memory (RAM/VRAM): a model with 7 billion parameters needs quantised (4 bit) about 4 to 5 GB, 13B about 8 GB, 70B about 40 GB. A graphics card with lots of VRAM (or Apple Silicon with large unified memory) is fastest; without a GPU it runs more slowly on the CPU (a few words per second). The smaller the model and the stronger the quantisation (Q4, Q5, Q8), the less memory but slightly worse quality. Context length increases memory need.

Advantages and limits

Advantages: privacy (nothing leaves the computer), no cost per request, offline, customisable (system prompts, fine-tuning, RAG with your own documents), no provider changing models. Limits: smaller models are weaker than the best cloud models (logic, code, factual knowledge), hallucinations remain, set-up and updates yourself, power consumption, mind the model’s licence (many allow private use, some restrict commercial use).

Practice and security

Load models only from trustworthy sources (Ollama library, Hugging Face with known providers); GGUF files are usually harmless, but scripts and pickle files are not. Do not expose the local API to the internet (only localhost or VPN), use access protection for shared use. Typical applications: chat, summaries of your own documents, code help in the editor, translation, language-learning partner, home automation. With RAG (read in and query documents) a small model becomes more useful. Compare models on your own example tasks.

See also: GGUF and llama.cpp, Quantisation, RAG, Open models and their licences, Using AI assistants sensibly and safely