Running AI models locally what you need and what to expect

[Context Sookie]

TL;DR

• Running a model locally means the model weights live on your own computer or server, so prompts and data never leave your machine
• Beginner friendly tools include Ollama and LM Studio, while llama.cpp and vLLM suit more technical users
• Memory is the main constraint: a 4 bit quantised model needs very roughly half a gigabyte to three quarters of a gigabyte of memory per billion parameters, plus room for context
• Laptops with 16 GB of memory can run small models well, 24 to 32 GB opens up mid sized models, and the largest open models need workstation or server hardware
• Popular open weight families in 2026 include Qwen, Gemma, Mistral, Llama, DeepSeek and OpenAI's gpt-oss, each with its own licence
• Expect strong privacy and control, but usually less capability and speed than the best cloud models

Why people run models locally

Cloud AI assistants are convenient and powerful, but they send your prompts to someone else's servers. For many tasks that is perfectly fine. For others, such as confidential documents, personal journals, client data or work in places with poor connectivity, keeping everything on your own hardware is appealing. Local models make that possible

Privacy is the headline benefit, but it is not the only one. Local models work offline, have no usage fees beyond electricity and hardware, and cannot be changed or withdrawn by a vendor without your consent. They are also wonderful for learning, because you can see exactly how models load, run and respond. Hobbyists, researchers and privacy conscious professionals have embraced them enthusiastically

This thread explains what you need to get started, how to choose a model, what performance to expect and which pitfalls to avoid. It is written for curious users rather than machine learning engineers. Figures are rules of thumb, because exact requirements depend on the model, the settings and the software. Please share your own measurements in the replies

Open weights explained

Local AI depends on open weight models, meaning models whose trained parameters are published for download. Open weights are not always fully open source in the traditional sense, because training data and code may not be released, and licences vary. Some licences are permissive, such as Apache 2.0, while others add conditions on commercial use or very large deployments. Always read the licence before using a model in a business

The open weight ecosystem has grown rapidly. Meta's Llama family helped start the trend, and models from Alibaba's Qwen team, Google's Gemma, Mistral AI, DeepSeek and Microsoft's Phi series followed. In August 2025, OpenAI released gpt-oss, its first open weight language models since GPT-2, in 20 billion and 120 billion parameter sizes under the Apache 2.0 licence. New releases arrive every few months, so check current versions before choosing

Model sizes are described in parameters, such as 8B for eight billion. Larger models are generally more capable but need more memory and compute. Many families offer several sizes, letting you pick the largest that fits your hardware. Some newer models use a mixture of experts design, where only part of the model is active for each token, which improves speed relative to total size

The tools

Ollama is the most common starting point. It runs on Windows, macOS and Linux, downloads models with a single command and provides a simple chat interface and a local API. Many other applications can connect to Ollama, so it doubles as a foundation for more advanced projects. For beginners, it removes most of the setup friction

LM Studio offers a friendly desktop interface for browsing, downloading and chatting with models. It can also run a local server compatible with common AI APIs, which lets other software use your local model as if it were a cloud service. People who prefer clicking to typing commands often choose it. It is a good way to compare several models side by side

llama.cpp is the engine underneath many local tools. It is a highly optimised open source project that runs models efficiently on ordinary CPUs, Apple Silicon and GPUs, using a file format called GGUF. Technical users run it directly for maximum control. For serving models to many users, tools such as vLLM are designed for high throughput on GPU servers

Hardware and memory

Memory is the deciding factor. A model's parameters must fit in memory, either the system RAM or the graphics card's VRAM, along with space for the context of your conversation. At full 16 bit precision, each billion parameters needs about two gigabytes. That would put an eight billion parameter model at around sixteen gigabytes before counting context

Quantisation makes local AI practical. It stores parameters with fewer bits, commonly four, which cuts memory needs by roughly three quarters compared with 16 bit precision. A 4 bit quantised eight billion parameter model typically needs around five gigabytes, making it comfortable on many laptops. Quality drops slightly with heavier quantisation, but 4 bit versions of good models usually perform well for everyday tasks

Context length adds to memory use. Longer conversations and larger documents require more memory for what is called the key value cache. If you plan to work with long documents, leave extra headroom beyond the model size itself. Many tools let you set the maximum context length to balance capability and memory

Apple Silicon Macs are popular for local AI because their unified memory is shared between the processor and graphics, so a Mac with 32 or 64 gigabytes can load larger models than a typical gaming graphics card. On PCs, graphics cards with 12 to 24 gigabytes of VRAM provide fast performance for small and mid sized models. Compact AI workstations with large unified memory, such as NVIDIA's DGX Spark launched in 2025 with 128 gigabytes, target users who want bigger models on a desk. Ordinary CPUs can run small models too, just more slowly

Matching models to hardware

• 8 GB of memory: very small models of 1 to 4 billion parameters, good for simple tasks and experimentation
• 16 GB: models around 4 to 8 billion parameters run comfortably; OpenAI states gpt-oss-20b can run within 16 GB
• 24 to 32 GB: mid sized models such as 12 to 30 billion parameters, including models like Gemma 3 27B on a 24 GB graphics card
• 48 to 128 GB: larger models of 30 to 120 billion parameters; OpenAI states gpt-oss-120b fits on a single 80 GB GPU
• Multi GPU servers: the largest open models with hundreds of billions of parameters

These figures are approximate and assume 4 bit or similar quantisation. Mixture of experts models can need more memory than their active parameters suggest, because all experts must usually be loaded. Model cards on Hugging Face and documentation from tool providers give specific guidance. Start with the largest model that runs comfortably, then experiment

What to expect from performance

Speed is measured in tokens per second. Reading speed for most people is roughly five to ten tokens per second, so anything above that feels responsive for chat. Small models on a modern laptop often exceed this easily. Larger models on modest hardware may produce only a few tokens per second, which feels slow but can still be useful for background tasks

Capability is the bigger trade off. The best cloud models are trained with vast resources and typically outperform local models on complex reasoning, coding and knowledge heavy tasks. Mid sized open models have become remarkably good, however, and handle summarising, drafting, classification and question answering over your own documents very well. Match the task to the model rather than expecting a laptop to rival a data centre

Energy and heat are worth noticing. Running a large model keeps your processor or graphics card busy, which uses power and produces heat and fan noise. For occasional use this is negligible. For heavy continuous use, it becomes part of the cost calculation

Practical uses

Private document assistants are one of the most popular uses. Tools can index your files locally and let a model answer questions about them without anything leaving your computer. This suits personal archives, research notes and confidential work documents. Quality depends on both the model and the retrieval setup

Coding help is another. Smaller code focused models can autocomplete, explain and refactor code locally, which appeals to developers working on proprietary software. Writing assistance, translation, transcription with local speech models and image description with local multimodal models round out common uses. Many people combine local models for private tasks with cloud models for demanding ones

Pitfalls and safety

Download models only from trusted sources, such as official organisation pages on Hugging Face or the built in libraries of tools like Ollama. Model files can be tampered with, and some older formats can execute code when loaded. Formats such as GGUF and safetensors are designed to avoid that risk. Check publisher names carefully, because lookalike uploads exist

Be careful with network exposure. Local model servers often listen on a network port so other applications can connect, and security researchers have repeatedly found such servers exposed to the internet without authentication. Keep servers bound to your own machine unless you deliberately need network access, and add authentication if you do. A misconfigured server can let strangers use your hardware or read your conversations

Remember that local does not mean infallible. Local models hallucinate like any other, and smaller ones do so more often. Verify important outputs, especially facts and code. Privacy is a strong benefit, but accuracy still requires human checking

Respect licences and content rules. Some model licences restrict particular uses, and generated content may still raise copyright or policy questions in professional settings. A quick read of the licence and your organisation's AI policy avoids surprises. Business users should involve their legal or compliance teams before deploying local models widely

A simple getting started plan

• Check your memory: system RAM on a laptop, or VRAM on a graphics card
• Install Ollama or LM Studio from the official website
• Download a small, well known model from an official source and try a few everyday tasks
• Try a larger model if your hardware allows and compare quality and speed
• Keep the server local only, and read each model's licence
• Share your hardware, model, quantisation and tokens per second here so others can compare

Closing thoughts

Running AI locally has become surprisingly easy. With a recent laptop and a free tool, anyone can chat with a capable model privately within minutes. Local models will not replace the most powerful cloud systems for every task, but they offer privacy, control and a deeper understanding of how AI works. For many people, that combination is well worth exploring

Please share your setups and results in the replies. Hardware details, model names, quantisation levels and speeds will help others choose wisely. Questions from newcomers are very welcome too


Key references for citation

• OpenAI, Introducing gpt-oss, August 2025
• Ollama, LM Studio and llama.cpp official documentation
• Hugging Face model cards for Qwen, Gemma, Mistral, Llama, DeepSeek and Phi models
• Hugging Face, safetensors documentation