Local Models Are Now Fast Enough for Coding

Smaller local models have been decent at performing bounded coding tasks for a while, but they were slow even on my 48GB M4 Pro. I could not really use them in the past for work where I wanted fast iteration. Fast-forward to today and we have 30B-class Mixture of Experts (MoE) models that run fast on the same machine.

Two things made this possible: (a) MoE architecture, which reduces how much model data has to move through memory for each generated token; and (b) 4-bit quantization instead of 8-bit, thereby reducing the model’s memory footprint.

A dense 30B model uses almost all 30 billion parameters for every token it generates. A Mixture of Experts model splits some of its layers into groups of experts, and a router chooses only a few for each token. You can see this in the model name: 35B means 35 billion parameters in total, while A3B means about 3 billion active parameters per token.

The inactive experts do not leave memory. All 35 billion parameters still have to fit on the machine. The difference is that only the active subset, roughly 3 billion parameters, participates in generating each token, so substantially less data moves through memory than in a dense 35B model. That is where the speed comes from.

Quantization stores each model weight using fewer bits. At 8 bits, each weight takes one byte, so 35 billion weights need roughly 35GB. At 4 bits, each weight takes half a byte, cutting the raw weight storage in half. The actual Distilled Weight Quantization (DWQ) model is around 21GB because some values stay at higher precision and quantization adds some overhead. DWQ uses a higher-precision model as a reference while tuning the 4-bit version, preserving more quality than a basic conversion. That leaves 14GB more for the runtime and context cache.

Here is what running Qwen3.6-35B-A3B-4bit-DWQ looks like on my machine:

At 39.2 tokens/s, the model produces roughly 30 English words per second. That is slower than frontier APIs, but much faster than I can read it.

How I Use It

I still use frontier models to explore unfamiliar codebases and make architecture decisions. Once I know what needs to change and roughly where, I write the target state to a spec and give the bounded implementation work to the local model. By then, the important decisions are already in the spec, and the local model just has to follow them.

The app I use to run the model, oMLX, also handles persistent prefix caching. Coding agents resend a large stable prefix every turn, including the system prompt, tool definitions, and files already in context. oMLX keeps the reusable cache blocks in memory or on SSD so the model does not have to process them again. Without that cache, the wait for the first token in a long session gets long enough that I stop using it.

My Setup

1. Install oMLX. Download the app from omlx.ai. It requires Apple Silicon and macOS 15 or later. It runs as a menu bar app with a dashboard at http://localhost:8000/admin and an OpenAI-compatible endpoint at http://localhost:8000/v1.

2. Download a model. The Model Downloader in the dashboard pulls from Hugging Face. I primarily use these two:

Note: If your machine has less memory, search for a smaller model. oMLX also detects how much RAM is available and suggests models that should fit right off the bat.

3. Use it. You can use it via (a) the chatbot interface oMLX provides at http://localhost:8000/admin/chat, or (b) a coding agent or another chatbot that connects to its OpenAI-compatible endpoint at http://localhost:8000/v1.

My preference is the latter. I use opencode with the following configuration in ~/.config/opencode/opencode.json:

{
    "$schema": "https://opencode.ai/config.json",
    "provider": {
        "omlx": {
            "npm": "@ai-sdk/openai-compatible",
            "name": "oMLX (local)",
            "options": {
                "baseURL": "http://localhost:8000/v1"
            },
            "models": {
                "mlx-community/Qwen3.6-35B-A3B-4bit-DWQ": {
                    "name": "Qwen3.6 35B-A3B (local)"
                }
            }
        }
    }
}

Then run /models in opencode and select the model you loaded in oMLX.

That’s it. Have fun running your own fast local LLM.