Self-hosted LLM vs on-device AI: the real trade-offs

Disclosure first: I build Outlier, a Mac app that runs models on-device, so I have a side in this — which is why I want to be precise about where the server approach wins over mine. Both routes run the same open weights. What changes is where the tensors live and who pays for the silicon.

What self-hosting actually involves

The mechanism is unglamorous. You pull open-weight files from Hugging Face, load them into an inference server on a GPU machine, and that process exposes an HTTP endpoint speaking the OpenAI chat-completions schema. Every client you own points at your URL.

Three servers cover most of the field. vLLM (Apache-2.0) calls itself "a fast and easy-to-use library for LLM inference and serving" and supports NVIDIA, AMD and Intel GPUs plus x86/ARM/PowerPC CPUs. llama.cpp (MIT) ships llama-server, "a lightweight, OpenAI API compatible, HTTP server," with 1.5-bit to 8-bit quantization and CPU+GPU hybrid inference for models larger than total VRAM. Ollama (MIT) wraps it behind a REST API on localhost:11434.

The lever that makes servers different in kind is --tensor-parallel-size, vLLM's "number of tensor parallel groups": splitting one model's weights across many GPUs is how a deployment holds what no single accelerator can.

What on-device actually involves

On-device, the weights load into the same process the user is clicking around in, and inference runs on whatever accelerator the machine has. On Apple Silicon that's Metal, usually via MLX (MIT), built around unified memory: "Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device types without transferring data."

Outlier is one instance of that shape — an example here, not the conclusion. Its Core tier is Qwen3.6-27B at 15.13 GB, wants 24 GB of RAM and runs about 20.7 tok/s on an M1 Ultra. Its largest is a 397B-a17b MoE, a 209 GB download that pages experts off the SSD to keep peak RSS near 11 GB on a 64 GB Mac — at 1.59 tok/s, which is the honest shape of the ceiling. Those Core weights are the same Apache-2.0 checkpoint you'd serve from vLLM, though: on-device doesn't mean a weaker model family.

Side by side

Where self-hosting genuinely wins

These aren't concessions — they're why serious deployments are server-side.

• Model size. DeepSeek-V3 is 671B total parameters with 37B activated per token; gpt-oss-120b needs an 80 GB GPU. Those are server-class artifacts, and on-device workarounds trade throughput to get near them.
• Concurrency, the real differentiator. vLLM's PagedAttention partitions the KV cache into blocks and cuts memory waste from 60–80% to under 4%, which is what makes large batches viable. The launch post claimed "up to 24x higher throughput compared to HF" — June 2023 against 2023 baselines, so treat it as directional. The architecture point stands: one GPU serving a whole team is something only a server does.
• Bandwidth. An H100 SXM moves 3.35 TB/s. Long-context prefill and big-batch decode are bandwidth problems — where datacenter hardware has the clearest advantage over a desktop.
• One endpoint, many clients. vLLM and llama-server both speak the OpenAI API, so a single deployment backs laptops, phones, CI jobs and internal tools with one upgrade path. A per-machine app can't serve a CI runner.
• It offloads the work. No battery drain, no thermal throttling, no multi-gigabyte resident set fighting the user's apps; jobs survive a closed lid.
• Operability. Kubernetes deploys, versioned config, central logs and metrics, per-key auth via --api-key. You can answer "what did we ask the model last Tuesday" org-wide — impossible across desktop installs.
• Data control can be just as strong. On hardware you own, prompts never reach a model vendor and retention is your policy — why regulated buyers pick it.

Where on-device genuinely wins

• Zero marginal cost per user. Nothing to meter or scale — the thousandth user costs nothing.
• It works with the network off. A self-hosted endpoint can't, by construction. Any outage — provider, transit or your own cluster — takes the assistant down.
• No round trip per token, and no third party holding physical custody of the RAM the model runs in. AWS's data-privacy FAQ says "we do not access or use your content for any purpose without your agreement," but doesn't explicitly address AI/ML training use — confirm that in the service terms, which differ per provider.
• Apple Silicon is the native target. vLLM's install matrix covers CUDA, ROCm, XPU and CPU; Apple Silicon appears only "via vLLM-Metal," a separate project outside the main repo, plus CPU-only support. In practice, self-hosting means NVIDIA.
• Nothing to secure. Ollama binds 127.0.0.1, and its FAQ's answer for network access is Nginx, ngrok or Cloudflare Tunnel — auth is what the operator adds. vLLM's --api-key is a header check, a floor rather than a security model. A misconfigured bind puts an unauthenticated endpoint on the internet; an app with no socket can't.

The cost and ops math people skip

RunPod Secure Cloud lists H100 PCIe at $2.89/hr and RTX 4090 at $0.69/hr, storage billed separately. Run either around the clock — 730 hours a month — and that's roughly $2,110 and $504 respectively. Lambda's H100 SXM is $4.29/GPU/hr, about $3,100/month, with no egress fees. Hugging Face Inference Endpoints run $0.80/hr on an L4 to $4.50/hr on an H100.

Serverless softens that: RunPod bills per second "from when a worker starts until it fully stops," and Flex workers scale to zero — though the idle timeout, five seconds by default, is still billed, and the next request pays a cold start.

Then the work itself. vLLM's Kubernetes guide walks you through a GPU cluster, a shared-memory emptyDir that tensor-parallel inference requires, plus optional model-cache storage (a PersistentVolumeClaim or hostPath) and, for gated models, a Secret holding a Hugging Face token — before TLS, authz, rate limiting or on-call. Owning metal moves the problem to facilities: NVIDIA rates the H100 SXM at up to 700W configurable, per GPU, before cooling.

Check licenses too — open weights aren't uniformly open source. Llama 3.3 needs a separate license from Meta above 700 million monthly active users and mandates "Built with Llama" attribution. DeepSeek-V3's code is MIT while its weights carry a separate Model License. Open WebUI forbids removing its branding above 50 end users per 30 days. Qwen3.6-27B and gpt-oss-20b are plain Apache-2.0.

Who should pick which

Self-host if you're serving more than one person, you need a model bigger than any single machine holds, you want one endpoint behind your CI and internal tools, you need central audit logs, or compliance says inference must run on hardware you own. Self-host for throughput work too — batch classification, evals, pipeline jobs — where continuous batching pays for the GPU many times over.

Go on-device if you're one person doing one thing at a time, the recurring bill is the problem, the machine has to work without a connection, or the data is the kind you'd rather never transit a network. Outlier is one implementation: no account, no caps, nothing uploaded, free at the Nano and Lite tiers, and with no web search, image generation, mobile app or Windows build — that's what the architecture costs.

Run both — something on-device for routine and confidential work, a server for jobs needing 671B parameters or dozens of concurrent requests. They're two places to put the same weights.