Self-hosting an LLM on a server vs running it on-device

Disclosure first: I build Outlier, a Mac app that runs models on-device, so I have a side in this. That's exactly why I want to be precise about where the server approach genuinely beats mine. Both routes run the same open weights. What changes is where the tensors live, who pays for the silicon they live on, and what breaks when something goes wrong.

What self-hosting actually involves

The mechanism is unglamorous. You pull open-weight files — safetensors or GGUF — from somewhere like Hugging Face, load them into an inference server on a machine with a GPU, and that process exposes an HTTP endpoint speaking the OpenAI chat-completions schema. Every client you already own then points at your URL.

Three servers cover most of the field. vLLM (Apache-2.0) calls itself "a fast and easy-to-use library for LLM inference and serving" and supports NVIDIA, AMD and Intel GPUs plus x86/ARM/PowerPC CPUs, with plugins for Google TPUs, Intel Gaudi and Huawei Ascend. llama.cpp (MIT) ships llama-server, "a lightweight, OpenAI API compatible, HTTP server for serving LLMs," spanning Metal, CUDA, HIP, Vulkan, SYCL and WebGPU, with 1.5-bit through 8-bit quantization and CPU+GPU hybrid inference to partially accelerate models larger than total VRAM. Ollama (MIT) wraps that for macOS, Windows, Linux and Docker behind a REST API on localhost:11434.

The lever that makes servers different in kind, not degree, is --tensor-parallel-size — vLLM's "number of tensor parallel groups." Splitting one model's weights across many GPUs is how a deployment holds something no single accelerator can. An AWS g6e.48xlarge, a common target, is eight L40S cards at 48 GB each: 384 GB of VRAM in one box.

What on-device actually involves

On-device, the weights load into the same process the user is clicking around in, and inference runs on whatever accelerator the machine already has. On Apple Silicon that's Metal, usually via MLX (MIT), built around unified memory: "Arrays in MLX live in shared memory. Operations on MLX arrays can be performed on any of the supported device types without transferring data." No PCIe staging — but also no way to add memory the machine wasn't sold with.

Outlier is one instance of this shape, and I'll use it as an example rather than a recommendation. Apple Silicon only, M1 or later, macOS 12+. Its Core tier is Qwen3.6-27B at 15.13 GB, needs 24 GB of RAM, and runs about 20.7 tok/s on an M1 Ultra. Its largest tier is a 397B-a17b MoE: a 209 GB download that needs a 64 GB Mac and pages experts off the SSD to hold peak RSS near 11 GB, at 2.1 tok/s. That number is the honest shape of the ceiling — on-device can reach big models, but it reaches them by fighting for every gigabyte. Worth noting the Core weights are Apache-2.0 Qwen3.6-27B, the same checkpoint you'd serve from vLLM. On-device doesn't imply a weaker model family.

Side by side

Where self-hosting genuinely wins

These aren't concessions. They're the reasons serious deployments are server-side.

• Model size. DeepSeek-V3 is 671B total parameters with 37B activated per token; gpt-oss-120b needs an 80 GB GPU. Those are server-class artifacts, and the on-device workarounds trade throughput to get near them.
• Concurrency, the real differentiator. vLLM's PagedAttention partitions the KV cache into blocks and cuts memory waste from 60–80% to under 4%, which is what makes large batches viable. The launch post claimed "up to 24x higher throughput compared to HF and up to 3.5x higher throughput than TGI" — that's June 2023 against 2023 baselines, so treat it as directional, not current. The architectural point holds: one GPU serving forty people is something only a server does.
• Bandwidth. An H100 SXM moves 3.35 TB/s. Long-context prefill and big-batch decode are bandwidth problems, and this is where datacenter hardware most clearly beats a desktop.
• One endpoint, many clients. Because vLLM and llama-server both speak the OpenAI API, a single deployment backs laptops, phones, CI jobs and internal tools with one upgrade path. A per-machine app can't serve a CI runner at all.
• It offloads the work. No battery drain, no thermal throttling, no multi-gigabyte resident set competing with the user's real apps, and jobs survive a closed lid.
• Operability. Container and Kubernetes deploys, versioned config, central logs and metrics, per-key auth via --api-key. You can answer "what did we ask the model last Tuesday" org-wide — genuinely impossible across a fleet of desktop installs.
• Data control can be just as strong. On hardware you own or inside your own VPC, prompts never reach a model vendor and retention is your policy. That's why regulated buyers pick it.

Where on-device genuinely wins

• Zero marginal cost per user. Nothing to meter, nothing to scale. The thousandth user costs the operator nothing.
• It works with the network off. A self-hosted endpoint can't, by construction. Any outage — provider, transit, or your own cluster — takes the assistant down; on a plane it was never up.
• No round trip per token, so no network latency floor, and no third party with physical custody of the RAM the model runs in. Unless the server is in your building, prompts enter someone else's datacenter. AWS's data-privacy FAQ says "we do not access or use your content for any purpose without your agreement," but that page doesn't explicitly address AI/ML training use — you'd confirm that in the service terms, and the answer differs per provider.
• Apple Silicon is the native target. vLLM's install matrix covers CUDA, ROCm, XPU and CPU; Apple Silicon appears only "via vLLM-Metal," a separate project outside the main repo, plus CPU-only support. In practice, self-hosting means NVIDIA — a different hardware and driver world from a Mac.
• Nothing to secure. Ollama binds 127.0.0.1 and its FAQ's answer for network exposure is Nginx, ngrok or Cloudflare Tunnel; auth is what the operator adds. vLLM's --api-key is a header check, a floor rather than a security model. A misconfigured bind is an unauthenticated inference endpoint on the open internet. An app with no socket has no such failure mode.

The cost and ops math people skip

RunPod Secure Cloud lists H100 PCIe at $2.89/hr and RTX 4090 at $0.69/hr, storage billed separately. Run either around the clock — 730 hours a month — and that's roughly $2,110/month for the H100 and roughly $504/month for the 4090. Lambda lists one H100 SXM at $4.29/GPU/hr (about $3,100/month) with no egress fees. Hugging Face Inference Endpoints, if you'd rather not touch metal, run $0.80/hr on an L4 up to $9.25/hr on a B200.

Serverless softens this without erasing it. RunPod bills per second "from when a worker starts until it fully stops," and Flex workers scale to zero — with a configurable idle timeout, default five seconds, that's still billed, plus a cold start on the next request. You're trading money for time-to-first-token.

Then there's the work itself. vLLM's own Kubernetes guide wants a running GPU cluster, a PersistentVolumeClaim for model storage, a Secret holding a Hugging Face token, and a shared-memory emptyDir for tensor-parallel inference — before you add TLS, authz, rate limiting, monitoring or an on-call rotation. Buying instead of renting moves the problem to facilities: NVIDIA rates the H100 SXM at up to 700W configurable, per GPU, before host, networking and cooling.

One more check before building on any of it: open weights aren't uniformly open source. Llama 3.3's license requires a separate license from Meta above 700 million monthly active users and mandates "Built with Llama" attribution. DeepSeek-V3's code is MIT while the weights carry a separate Model License. Open WebUI, the common self-hosted front end, is BSD-3-Clause plus a clause prohibiting altering or removing its branding unless you're under 50 end users per 30 days or hold written permission. Qwen3.6-27B and gpt-oss-20b, by contrast, are plain Apache-2.0.

Who should pick which

Self-host if you're serving more than one person, you need a model bigger than any single machine holds, you want one endpoint behind your CI and internal tools, you need central audit logs and key rotation, or compliance says inference has to run on hardware your organization owns. Self-host too for throughput work — batch classification, evals, pipeline jobs — where continuous batching pays for the GPU many times over.

Go on-device if you're one person doing one thing at a time, the recurring bill is the problem, the machine has to work without a connection, or the data is the kind you'd rather never transit anyone's network. Outlier is one implementation: no account, no caps, nothing uploaded, free at the Nano and Lite tiers. It has no web search, no image generation, no mobile app, no voice mode, no Windows or Linux build and no cloud sync, because those are what the architecture costs. LM Studio, Jan and a locally built llama.cpp are the same category with different trade-offs.

Run both if that's your actual week — a small on-device model for everything routine and confidential, a server for the jobs needing 671B parameters or forty concurrent requests. They're not competing philosophies. They're two places to put the same weights.