Local AI vs a hosted LLM API: a developer's comparison

Disclosure first: I build Outlier, a Mac app that runs models on-device, so I'm an interested party. I've kept this to numbers you can check, and said where the hosted API wins.

What each one actually does

A local runtime is a process on your machine that loads a quantized weights file into memory, runs the forward pass on whatever accelerator you have, and keeps the KV cache in your own RAM. llama.cpp is the substrate most of the ecosystem sits on — Metal on Apple Silicon, plus CUDA, HIP, SYCL, Vulkan and CPU paths, at quantizations from 1.5-bit to 8-bit. Ollama wraps it in a CLI and a local REST API; LM Studio is a desktop app running GGUF and MLX models behind OpenAI-compatible REST, with offline document RAG. MLX is Apple's array framework built on unified memory, and vLLM is the server-shaped one: PagedAttention and continuous batching for throughput.

A hosted API is an HTTPS request: you send tokens, a vendor runs a model you can't inspect on hardware you don't own, and bills you per million tokens each way.

The money, and the ceilings attached to it

Claude API list pricing, per million tokens: Opus 4.8 $5 in / $25 out, Sonnet 5 $2/$10 introductory through August 31 2026, then $3/$15. OpenAI's list runs from gpt-5.6-sol at $5 input / $0.50 cached input / $30 output down to gpt-5.4-nano at $0.20/$0.02/$1.25.

Headline rates aren't what you pay. Anthropic prices a 5-minute cache write at 1.25x base input and a 1-hour write at 2x, but a cache read at 0.1x — a 5-minute cache pays for itself after one hit. Batch takes 50% off both directions (Opus 4.8 becomes $2.50/$12.50) and stacks with caching. The other way: hosted web search adds $10 per 1,000 searches on top of tokens, and Anthropic's newer tokenizer (Opus 4.7+, Sonnet 5, Fable 5) emits roughly 30% more tokens for the same text — identical content costs more at an unchanged rate.

The ceilings are per-organization. Start tier for Opus 4.x allows 1,000 requests, 2M input tokens and 400K output tokens per minute; Scale reaches 10,000 / 10M / 2M, and cache reads don't count toward the input limit on most models. Spend caps are $500 (Start), $1,000 (Build) and $200,000 (Scale): over a rate limit you get a 429 with retry-after, over the cap usage pauses until next month. Local inference has none of that arithmetic — though an idle API costs $0 and idle hardware doesn't.

Where the hardware ceiling actually bites

OpenAI's open-weight release states the limit cleanly: gpt-oss-120b (117B params, 5.1B active) fits on a single 80GB GPU, while only the much smaller gpt-oss-20b (21B, 3.6B active) is documented to run within 16GB — both via MXFP4 quantization of the MoE weights. No quantization setting fits a frontier-scale model in a laptop.

Sizing below it is your job. Ollama publishes no minimum RAM figure — only that required RAM scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH, and that a model which doesn't fit gets split across CPU and GPU (its example: 48%/52%).

For scale: Outlier's largest tier, a 397B-A17B MoE, is a 209 GB download needing 64 GB of RAM, and manages 2.1 tok/s by streaming experts off the SSD. I won't rank open weights against frontier models on benchmarks here — the memory ceiling is the part that doesn't need my word for it.

Data path, retention and licensing

"Cloud means they train on your prompts" is false at the paid tiers. Anthropic doesn't by default use commercial-product inputs or outputs for training, except what you submit as feedback or a bug report; OpenAI doesn't train on API data unless you opt in. The real distinctions are narrower. It generates abuse-monitoring logs for all API usage, retained up to 30 days by default, and Zero Data Retention needs prior approval and isn't available on every endpoint (/v1/conversations and /v1/assistants aren't eligible). Tiers differ: on Google's Gemini API Unpaid tier, Google uses submitted content and responses to improve its products, and human reviewers may read and annotate API input and output. A free prototyping key isn't the paid terms.

Local's claim is different in kind: Ollama's FAQ says that run locally it never sees your prompts. But that property belongs to the execution mode, not the brand — Ollama also sells hosted tiers ($20/month Pro, $100/month Max) processed server-side.

Licensing needs the same care. Runtimes are clean — llama.cpp, Ollama and MLX are MIT, vLLM Apache-2.0 — but weights are messier. gpt-oss and Qwen3-32B are Apache-2.0; Meta's Llama 4 Community License isn't, requiring a separate license request if your products exceeded 700 million monthly active users on the release date, a "Built with Llama" notice, and derivative names beginning with "Llama". Tooling too: as of mid-2026 LM Studio's documentation publishes no open-source license for the desktop app itself, though it's been free at home and at work since July 2025.

Side by side

Where each genuinely wins

The hosted API wins outright when…

• You need the capability ceiling. Million-token-context frontier models don't fit in consumer RAM — a memory limit, not a tuning gap.
• You're shipping cross-platform. No weights, no thermal budget; the same call works from a laptop, a CI runner or a phone.
• Load is spiky. Scale tier reaches 10,000 requests/minute, Batch halves bulk work, idle costs nothing.
• You'd otherwise rebuild infrastructure. Caching, batch, hosted search, code sandboxes and native tool use get reimplemented locally.
• You want upgrades for free. A new model is a one-string change, not a release plus a re-download.

Running it yourself wins outright when…

• Marginal cost is the constraint. Agent loops, corpus reprocessing and heavy best-of-N are metered by electricity, not an invoice and a spend cap.
• The data genuinely cannot leave. Air-gapped or regulated work: "the process never sees a network" beats any retention policy.
• You want the whole stack. Any open-weight model, any quantization from 1.5-bit to 8-bit, custom fine-tunes and LoRAs, your own sampling.
• You need to redistribute. MIT/Apache runtimes plus Apache-2.0 weights means shipping commercially with no vendor relationship.
• Local isn't one desktop. The same weights serve a team from a Linux box under vLLM, on NVIDIA, AMD or Intel hardware.

Who should pick which

Pick the hosted API if you're building cross-platform, your load is bursty, or you need the largest models and longest context. Budget for retries, deprecation dates and that ~30% tokenizer inflation.

Self-host if you have a fixed, heavy, repetitive workload; a hard data-residency requirement; or something to redistribute with no vendor in the loop. Read the weights license, not just the runtime license.

Use a packaged on-device app — Outlier is one example, LM Studio another — if you want local inference without owning the packaging, signing and update problem. Outlier is Apple Silicon only, with no web search, no image generation, no mobile app and no Windows build — exactly the gaps a hosted API fills.

Or both: routing cheap steps locally and escalating hard ones is a config change, not a rewrite.