Running Local Models Is Good Now

By mid-2024, if you asked a developer what local AI models were good for, the answer was usually the same: fun to play with, but not for serious work. That judgment no longer holds.

On June 15, 2026, Vicki Boykis — a tech writer who has followed local model development closely — posted her experience under a straightforward title: Running Local Models Is Good Now. Her conclusion is sharp: things that were impossible six months ago are possible today. For developers who care about privacy, offline capability, or controlling their own costs, this shift is worth examining.

The turning point: from toy to tool#

To understand how far local models have come, go back to 2023-2024.

Back then, local models were, in Boykis’s words, “slow, hard to use, and just not that accurate for most programming tasks.” Mistral 7B was the most talked-about open-weight model at the time, but the gap between it and the frontier API models was a chasm. People tried local deployment out of curiosity or principle, not because it made them more productive.

Several things changed.

OpenAI OSS-20B was the first model that made Boykis stop double-checking against an API model. Her personal metric — “do I have to verify this against a frontier model” — had been failing for every local model before OSS-20B. This one passed.

Google’s Gemma series accelerated things further. Boykis says Gemma 4 was the first local model that let her run agentic coding loops — where the model reads files, writes code, runs tests, sees errors, and fixes them autonomously — at about 75% of the accuracy and speed of frontier models. That number would have been unthinkable a year earlier.

Qwen 2.5 Coder and Qwen 3 MOE from Alibaba kept pushing the coding benchmarks. The MOE (mixture-of-experts) architecture in Qwen 3 gives better results with fewer active parameters, which means it fits on consumer hardware more comfortably.

But the real breakthrough is not any single model. It is the ecosystem. Inference engines (llama.cpp, Ollama, LM Studio), quantization techniques (GGUF, AWQ, GPTQ), and agent frameworks (Continue.dev, Pi, Aider) have all matured together over the last 18 months. What used to require hours of configuration and debugging is now close to plug-and-play.

The hardware question#

The first question everyone asks is: what hardware do I need?

Boykis runs a 2022 M2 Mac with 64GB of unified memory and 1TB of storage. That is a two-and-a-half-year-old machine, not bleeding-edge hardware.

On this setup, she regularly:

• Refactors a Jupyter notebook into a Python repo with 5-6 modules
• Adds correct generic type annotations to codebases
• Writes unit tests
• Bootstraps a two-tower recommendation model from scratch
• Runs multi-turn agentic coding loops

The key insight for Apple Silicon users is that unified memory capacity is the binding constraint, not GPU compute. Because the M-series architecture shares memory between CPU and GPU, the amount of RAM directly determines what size model you can run. 64GB handles 7B-20B parameter quantized models. 128GB or 192GB opens up 70B-class models.

For PC users, an RTX 4090 (24GB VRAM) is the current sweet spot, handling 7B-13B 4-bit quantized models comfortably. A 7B model can even run on 16GB cards.

Boykis is honest about the limits: “the K-V cache grows to 64 GB RAM” during heavy agentic workflows. High-intensity local agenting still demands serious hardware — but it no longer demands hardware that is impossible to buy.

What makes local agentic coding exciting#

The most valuable section of Boykis’s post is her hands-on experience with local agentic coding.

Agentic coding means the AI is not just helping with one-off code generation. It loops: read a file, understand context, generate code, run tests, see errors, fix them, iterate. Each cycle is a full step in a development feedback loop.

For frontier models like Claude and GPT-4, this experience is already solid — but costly. Each agentic loop means multiple API calls. A complex coding task can burn several dollars in tokens within minutes.

The advantage of local models? The API marginal cost is near zero. You can run 100 or 1,000 agentic loops. The cost shifts to hardware depreciation, electricity, and your own patience.

Boykis’s local setup uses Pi (an open-source agent harness) paired with LM Studio as the inference server, all inside a Docker container. The container is deliberately restrictive — only bash execution was allowed, no Python execution, no web browsing. Despite these constraints, she describes the result as “about 75% the accuracy and speed of Claude or GPT.”

This aligns with feedback from the broader developer community. The consensus among people using local models with tools like Aider and Continue.dev is:

• Code completion and simple refactoring: local models are already quite usable; 7B-13B models feel close to cloud-tier fluency
• Test writing: solid, because testing is a structured task
• Complex architectural design and cross-file refactoring: still needs frontier models or human intervention — the context window on local hardware is the bottleneck
• Privacy-sensitive work: health, finance, and law-sector developers especially value local deployment, because data never leaves their machine

A practical guide to getting started#

For readers who want to try this themselves, here is a condensed setup guide:

Step 1: Pick an inference engine

• LM Studio — best for beginners; GUI-based, one-click model download and API server
• Ollama — best for CLI users; minimal commands, mature ecosystem
• llama.cpp — best for advanced users; highest performance, Docker-friendly

Step 2: Download a model

Recommended coding models (June 2026):

• Gemma 4 (Google) — one of the best local models overall; the 12B QAT quantized version is especially consumer-hardware-friendly
• Qwen 2.5 Coder (Alibaba) — coding-focused; 7B and 14B versions are both mature
• DeepSeek Coder V3 — strong coding performance with good Chinese-language support
• OpenAI OSS-20B — well-rounded, but needs 48GB+ of memory

Step 3: Wire up an agent harness

The conceptual setup:

LM Studio endpoint: http://host.docker.internal:1234/v1
Pi models.json: point to your local model path
Docker container: isolated environment, bash-only permissions

Best practices:

• Run agents in Docker to isolate them from your host system
• Start with simple tasks (unit tests, code review) before attempting complex refactoring
• Use a hybrid approach: local models for daily work, frontier APIs for complex architecture or security-critical code

More than cost savings#

When people talk about local models, “saving money” is usually the first argument. But it may not be the most important one.

Privacy is the primary driver. When your codebase contains customer data, business strategy, or unreleased product logic, sending it to a cloud API carries real risk. Several companies now explicitly prohibit pasting code into AI tools. Local deployment solves this cleanly.

Offline capability is the second driver. On a plane, in a subway tunnel, or anywhere with unreliable connectivity, local models are the only option that works.

Control is the third. You choose which model version to run, when to upgrade, and what quantization strategy to use. No API version changes, no service outages you cannot control.

Boykis also highlights a less obvious value of local models: observability. You can watch tokens come in and out, adjust the context window, switch quantization schemes, change system prompts, and see exactly how each adjustment affects performance. That transparency is something a black-box API cannot offer.

She also found something funny when she used Pi to analyze her own LM Studio session logs: her most common use case was “quick documentation lookups” — treating the local model as a hyper-personalized, always-available technical Q&A assistant. Not glamorous, but surprisingly practical in daily development.

What still holds local models back#

Boykis is candid about the remaining limitations:

Inference speed. Even quantized models on consumer hardware are significantly slower than cloud inference. For real-time conversation or rapid iteration, the gap is still noticeable.

Context window. A local model’s context is bounded by your hardware memory. Cloud models routinely handle 128K or even 200K token contexts. Local models are often limited to 8K-32K, which is not enough for large codebases.

Ecosystem fragmentation. Tools like LM Studio and Ollama have lowered the barrier significantly, but prompt template mismatches and toolchain compatibility issues still occur. The HuggingFace “Use This Model” button is helping, but we are not at full plug-and-play yet.

The 75% ceiling. Boykis’s own metric — 75% of frontier model performance — captures the current state. For many tasks that is good enough. For security audits, complex math reasoning, or mission-critical code, you still need the frontier models.

Where things are going#

At the end of her post, Boykis reflects on a question that came up while testing Gemma-4-12b-qat: “If we are constrained by performance and price, what architectural tradeoffs do we need to make?”

This question was almost never asked during the token gold rush. The assumption was always: bigger model, more parameters, more compute. But Gemma 4’s QAT (quantization-aware training) architecture shows a different direction — optimizing the model for small-memory environments from the design stage.

This points to two parallel paths for local models:

1. Hardware improvement — consumer hardware keeps getting better: more memory, faster NPUs, wider adoption of unified memory architectures
2. Architectural optimization — models designed from scratch for consumer hardware: quantization-friendly, MOE sparse activation, smaller but denser architectures

Both paths are already in motion. If they continue together, we can expect local models to close more of the gap with cloud APIs over the next 12-18 months — especially for latency-sensitive and privacy-first applications.

The takeaway#

Vicki Boykis’s conclusion is blunt: “used to be impossible for local models as recently as 6 months ago.”

Local models are not the future. They are the present. For every developer using AI today, the question is no longer whether local models are usable. It is which tasks you should move local — not to fully replace cloud APIs, but to have a real option when privacy, offline capability, or control matters.

Sources#

• Vicki Boykis: Running local models is good now
• Google AI: Gemma 4
• LM Studio — Ollama — llama.cpp
• Pi agent harness — Aider
• Hacker News discussion