Start here — what hardware do you have?
Verified 22 Aug 2026The single biggest constraint is your hardware. Pick your situation below to see which tools are even available to you.
LiteLLM
llama.cpp · HF Transformers
LM Studio · llama.cpp
llama.cpp · vLLM
TensorRT-LLM · Aphrodite
Model size → practical minimum hardware (4-bit quantized, short context)
2 is for K and V; bytes_per_value is 2 at FP16 or 1 at FP8. Every term comes from the model's config.json. Worked example: a 70B at 4-bit is ~35 GB of weights (~42 GB once you apply the 1.2×), and Llama 3.3 70B — 80 layers, 8 KV heads, head_dim 128 — costs about 0.33 MB per token at FP16, so roughly 2.6 GB of KV cache at 8K context for a single request. Raise the context window or serve concurrent users and that term grows linearly.
Two further caveats: quantization is not free (below roughly 4 bits per weight, quality loss becomes noticeable on most models), and macOS caps GPU-addressable memory at roughly 65–75% of unified RAM by default, so a 64 GB Mac offers ~44–48 GB to the model, not 64.
Tool comparison
Verified 22 Aug 2026Throughput is split into two columns because they measure different things, and conflating them is the most common way to pick the wrong tool. Single-stream is tokens/sec for one user with no queue — what you feel on a laptop. Concurrent is aggregate throughput across many batched requests — what matters when you serve a team. llama.cpp is competitive with vLLM at batch size 1 and falls far behind it at batch size 64.
| Tool | Platform / OS | Hardware min. | Single-stream | Concurrent | Setup | Status | Licence | Cost | Best for |
|---|---|---|---|---|---|---|---|---|---|
| ● Cloud API proxies | |||||||||
openrouter.ai/api/v1 |
Any (web API) | None | Provider | Provider | Very easy | Active | Proprietary hosted service |
Free tier ~$0.05–15 / 1M tokens varies hugely by model |
Multi-model access (OpenAI, Claude, Gemini, Llama) with one API key; fallback routing; competitive open-model prices |
self-hosted proxy |
Any (Python) | None (server: 1 GB RAM + a database) | Provider | Provider | Easy | Active | MIT enterprise tier |
Free OSS You pay providers + enterprise: contact sales |
Teams needing a self-hosted multi-provider gateway; cost tracking, rate limits, Redis cache; provider failover |
| ● Easy local runners | |||||||||
GGUF |
macOS · Linux Windows |
CPU: 8 GB RAM GPU: 6 GB VRAM optional |
Medium | Low | Very easy | Active | MIT | Free | Quickest path to local models; Docker-style pull/run; single-user dev and prototyping |
GGUF + MLX · GUI |
macOS · Linux Windows |
CPU: 8 GB RAM GPU: 6 GB VRAM optional |
Medium | Low | Very easy | Active | Proprietary free to use |
Free | GUI-first experience; no CLI needed; model discovery from HF Hub; MLX backend on Apple Silicon |
Apple framework |
macOS only (Apple Silicon) |
8 GB unified min 64 GB for 70B models |
Medium | Low | Easy | Active | MIT | Free | Best performance on Mac; unified memory makes large models feasible without a discrete GPU |
| ● Production inference servers | |||||||||
PagedAttention |
Linux (Docker on others) |
NVIDIA GPU 16 GB+ VRAM recommended |
High | Very high | Moderate | Active | Apache-2.0 | Free OSS + GPU compute |
The default production choice; continuous batching; strong throughput per dollar; multi-GPU tensor parallel |
RadixAttention |
Linux | NVIDIA GPU 16 GB+ VRAM recommended |
High | Very high | Moderate | Active | Apache-2.0 | Free OSS + GPU compute |
Shared-prefix workloads; automatic prefix caching; strong structured/constrained generation |
NVIDIA · compiled engines |
Linux · Docker NVIDIA only |
NVIDIA GPU Ampere or newer; 24 GB+ typical |
Very high | Very high | Hard | Active | Apache-2.0 | Free OSS + GPU compute |
Squeezing the last 10–30% out of NVIDIA hardware at scale; FP8/NVFP4 on Hopper and Blackwell |
vLLM fork · multi-format |
Linux | NVIDIA GPU 12 GB+ VRAM |
High | High | Moderate | Active | AGPL-3.0 copyleft — check first |
Free OSS + GPU compute |
Batched serving of quant formats upstream vLLM won't load — EXL2, GGUF, QuIP# — plus DRY/XTC samplers |
HuggingFace serving |
Linux · Docker AWS Inferentia |
NVIDIA GPU 16 GB+ VRAM; AMD ROCm ok |
High | High | Moderate | ArchivedRead-only 21 Mar 2026 | Apache-2.0 | Free OSS + GPU compute |
Not for new deployments. No new model support, features, or security patches. Listed so you can recognise it in existing stacks and older guides; migrate to vLLM or SGLang |
| ● Libraries and engines | |||||||||
reference library |
Any (Python) macOS · Linux · Windows |
CPU ok (slow) GPU: 6 GB+ VRAM for speed |
Low | Low | Moderate | Active | Apache-2.0 | Free | Research, fine-tuning, model internals access; every architecture; canonical PyTorch interface |
C++ engine · GGUF |
Any (C++) mac · lin · win · mobile |
CPU: 8 GB RAM GPU: 6 GB VRAM optional |
Medium | Low | Moderate | Active | MIT | Free | Backbone of Ollama/LM Studio; embed in C/C++ apps; broadest hardware compatibility including mobile/edge |
EXL3 · replaces ExLlamaV2 |
Linux · Windows NVIDIA-focused |
NVIDIA GPU Ampere+; 16 GB+ typical |
Very high | Medium | Moderate | EarlyPre-1.0, active | MIT | Free | Maximum quality per gigabyte of VRAM on consumer NVIDIA; fitting a 70B onto one or two cards |
Status reflects upstream development as of the verification date: Active = shipping releases; Early = active but pre-1.0 and still changing shape; Maintenance = bug fixes only; Archived = read-only, no further work. Re-check this column first if you are reading this months later. Throughput pills are relative to the others in this table on comparable NVIDIA hardware, not absolute tokens/sec. GPU compute is roughly $0.75–3/hr per GPU wherever it appears — see the section intro below. On endpoints: every tool here speaks OpenAI-compatible HTTP, including TensorRT-LLM via trtllm-serve, with two things to plan around — HF Transformers only does so through the newer transformers serve command, and ExLlamaV3 is served through TabbyAPI rather than a built-in server.
On model access: the frontier closed models (OpenAI's GPT series, Claude, Gemini) are reachable through their own APIs, through a major cloud's managed service — AWS Bedrock, Google Vertex AI, Azure — or through a proxy like OpenRouter or LiteLLM. Bedrock and Vertex are first-party managed platforms, not proxies, and are often the right answer for enterprises with existing cloud commitments. Every other tool here runs open-weight models only: Llama, Mistral, Qwen, Gemma, Phi and so on.
"Open-weight" is not the same as "open source." Llama ships under the Meta Llama Community License and Gemma under Google's Gemma Terms of Use; both carry redistribution and use restrictions that an OSI-approved licence would not. The weights are free to download and run, and you pay only for compute — but read the licence before shipping a commercial product on top of one.
Deep-dives
Verified 22 Aug 2026What each tool actually does well, and where it stops being the right answer.
Cloud API Proxies
Not inference tools — routing layers in front of external providers. Prompts leave your machine. Zero hardware required on your side.
OpenRouter
Routes to hundreds of models across all major providers with one API key and one endpoint format. The base URL is https://openrouter.ai/api/v1, and because it is OpenAI-compatible you can point an existing OpenAI SDK at it by changing one line. The strong features are automatic fallback (if one provider is down, it retries another) and price transparency — you can compare cost-per-token across backends and route to the cheapest one that meets your latency target. Open-weight model pricing here is competitive with anywhere else, though check the live model page rather than trusting any figure in a guide. Closed models route through their respective providers at near-retail prices. Main tradeoff: your prompts pass through OpenRouter's infrastructure.
LiteLLM
The self-hosted version of the same idea: a proxy that translates any provider's API into a single OpenAI-compatible endpoint. Run it on your own infrastructure and your keys never leave your environment. Beyond routing it provides per-model rate limiting, cost budgets per team or user, Redis-backed caching, and spend dashboards. SSO, audit logs, and support sit behind a paid enterprise tier, and LiteLLM does not publish standardised pricing — figures circulating in comparison articles are unreliable, so contact them directly. Budget for the real cost of self-hosting too: a server, a Postgres or Redis instance, monitoring, and someone to maintain it. Essential for teams that want to be provider-agnostic without changing application code every time they switch models.
Easy Local Runners
All free. All run offline. All support CPU inference (slow) and GPU acceleration where available. Best for individuals and prototyping.
Ollama
The tool most developers should start with. ollama run llama3.3 downloads and runs in one command. It handles model downloading, quantization selection, GPU detection, and serves a REST API on localhost:11434. Historically it was a llama.cpp wrapper, and that is still true for much of the model library — but Ollama has been moving newer architectures onto its own engine, so "it's just llama.cpp" is no longer a safe assumption when you are debugging a performance difference. Its real limitation is concurrency: it is built for single-user use and handles simultaneous requests poorly compared with vLLM. Fine for a laptop, wrong for a team API.
LM Studio
A GUI-first local runner. The model browser lets you search and download from HF Hub inside the app, and it can run as a local OpenAI-compatible server in the background. Best for non-developers, people demoing to stakeholders, or anyone who finds the CLI uncomfortable. It is not simply a skin over the same backend as Ollama: on Apple Silicon it ships an MLX backend alongside llama.cpp, which is meaningfully faster there and lets it load MLX-format models Ollama won't. Note that it is free to use but closed-source, unlike everything else in this category.
MLX
Apple's own ML framework, purpose-built for unified memory. On M-series chips the CPU and GPU share one physical RAM pool, so a Mac Studio with 128 GB of unified memory can hold models that would need multiple 80 GB datacentre cards to fit in VRAM — a capacity advantage, and a large one. Speed is a separate question, and it is bounded by memory bandwidth. Single-stream token generation reads the entire weight set per token, so throughput is roughly bandwidth ÷ model size, and real-world results land at about 60–80% of that ceiling. A 70B at 4-bit is ~35 GB of weights, so: an M3 Max at 400 GB/s has a theoretical ceiling near 11 tok/s and delivers roughly 7–9 tok/s; an M4 Max at 546 GB/s ceilings near 16 and delivers 9–12; an M5 Max at 614 GB/s ceilings near 18 and delivers 11–14; Ultra parts at ~800 GB/s go higher again. Note that binned Max chips run materially slower — 410 GB/s on the 32-core M4 Max, 460 on the 32-core M5 Max — so the configuration matters as much as the generation. The useful discipline is to divide before you believe: if a quoted figure exceeds bandwidth ÷ model size, the arithmetic rules it out regardless of who published it. MLX has its own model format (available for most popular models on HuggingFace), a Python API, and an OpenAI-compatible server. If you're on Apple Silicon with 32 GB+ of RAM, it is still the best local inference experience available to you.
Production Inference Servers
All free open-source software, though licences differ — check before commercial deployment. You pay for GPU compute: roughly $0.75–3/hr per GPU on Lambda Labs, RunPod, vast.ai, AWS, or GCP. All are Linux-first.
vLLM
The default production choice. PagedAttention manages the KV cache like virtual memory rather than preallocating contiguous blocks, eliminating most waste and enabling continuous batching — requests from different users share the GPU rather than queuing. Published figures put it in the range of 10–30× the throughput of naïve HuggingFace Transformers inference on the same hardware; treat the top of that range as a best case, not a promise. Supports tensor parallelism across multiple GPUs and a broad set of quantization formats including AWQ, GPTQ, and FP8. NVIDIA-primary, though ROCm support has matured considerably and is worth testing rather than dismissing. The main setup friction is understanding GPU memory sizing.
SGLang
vLLM's closest competitor, and it frequently wins on throughput benchmarks — especially when many requests share a long prefix. RadixAttention caches KV by common prefixes automatically, so a chatbot where every request starts with the same 2,000-token system prompt pays that cost once rather than per request. Structured and constrained generation (JSON schemas, grammar-based decoding) is a genuine strength, though vLLM has since added comparable guided decoding, so this is now an advantage of degree rather than kind. The two projects trade the throughput crown release to release — benchmark both on your own workload before committing, and don't trust a ranking more than a few months old.
TensorRT-LLM
NVIDIA's own serving stack, and the usual answer when you have exhausted what vLLM can give you. Rather than interpreting the model at runtime it compiles it into an optimised engine tuned to a specific GPU, batch size, and precision — which is where the extra performance comes from, and also where the pain comes from: that compilation step must be redone for each model and hardware target, and the engine is not portable. It leads on low-latency, high-concurrency serving and has first access to new NVIDIA precisions such as FP8 and NVFP4 on Hopper and Blackwell. One correction worth carrying forward: it no longer needs Triton in front of it. The trtllm-serve command is now the primary deployment path and speaks the OpenAI API over HTTP and gRPC, so guides telling you to stand up Triton Inference Server just to get a /v1/chat/completions endpoint are out of date. The real caveats are that it is NVIDIA-only by construction and that the build step is genuinely involved. Worth the setup cost at scale, hard to justify below it.
Aphrodite Engine
A community-maintained vLLM fork that prioritises quantization breadth over pure throughput. Its real differentiators are the GPU-native and CPU-adjacent formats vLLM won't load in one process — EXL2 via ExLlamaV2 kernels, GGUF via llama.cpp kernels, plus QuIP#, SqueezeLLM and now EXL3 — along with the full sampler set (DRY, XTC, mirostat) and a KoboldAI-compatible API alongside the OpenAI one. Be precise about what is not a differentiator: AWQ and GPTQ are supported natively by vLLM, and so is AQLM, so none of those are a reason to switch. Check the licence before deploying commercially: Aphrodite is AGPL-3.0, whose network-copyleft provision reaches services delivered over a network — materially different from the Apache-2.0 terms of every other server on this list. Fine for internal tooling, research, and personal use; a legal conversation for commercial SaaS. Note the repository has moved to the aphrodite-engine organisation.
TGI
Included as a warning, not a recommendation. HuggingFace put TGI into maintenance mode in December 2025 and archived the repository read-only on 21 March 2026. There will be no new model architectures, no new features, no performance work, and no security patches. HuggingFace's own documentation now directs users to vLLM and SGLang for serving, and to llama.cpp or MLX locally.
It earns a place here because a large amount of published advice — including guides updated well into 2026 — still names TGI as the safe, production-hardened default, and because you will meet it in existing stacks. What it was good at was real: point it at an HF Hub model ID and it handled sharding for you, ROCm support was solid and for a long time ahead of vLLM's, and it ran natively on Inferentia and SageMaker. If you are running it today, nothing breaks tomorrow — a frozen engine is stable by definition — but you are on your own for CVEs and locked out of every model released from here on. Plan the migration to vLLM or SGLang rather than deferring it, and if the Inferentia/SageMaker integration was your reason for choosing it, price that path first, since it is the piece with the least direct replacement.
Libraries and Engines
All free and open source. These are code you import, not servers you run. Maximum control, lowest abstraction — you own the inference loop.
HuggingFace Transformers
The reference Python library for every major model architecture. If a model exists, its canonical implementation is almost certainly here. Use it for fine-tuning, training, and any time you need access to model internals — hidden states, attention weights, intermediate activations. For raw inference serving it is still the slowest option here, but the reason has changed and older guides get it wrong: Transformers does now have continuous batching with paged attention, exposed through generate_batch() and used by the transformers serve command, which also gives you an OpenAI-compatible endpoint. The gap is maturity and tuning, not absence of the technique — the dedicated servers have years of kernel and scheduler work behind theirs. Treat transformers serve as a genuine convenience for testing and low-traffic internal tools, and still not a substitute for vLLM or SGLang under real load. Think of it as the lowest-level interface to model weights. Pairs with PEFT, Accelerate, and TRL for the full training ecosystem.
llama.cpp
The C++ engine underneath Ollama and LM Studio. Its standout quality is hardware reach: CPU, CUDA, Metal, Vulkan, ROCm, and on down to Raspberry Pi, iOS, and Android. GGUF quantization packs models efficiently, and GGUF is the one format with support essentially everywhere — if a model doesn't fit your GPU, GGUF with partial CPU offload is usually the fallback. It also ships llama-server, a capable OpenAI-compatible server in its own right, so you don't strictly need Ollama on top. If you're embedding inference in a C/C++ application, shipping to edge devices, or targeting hardware nothing else supports, work with it directly or via llama-cpp-python. Under concurrent load on NVIDIA it doesn't approach vLLM — it is optimised for breadth, not batched throughput.
ExLlamaV3
Start here, not at ExLlamaV2. The V2 README now states that development has stopped and continues on V3, TabbyAPI has moved EXL2 support to a legacy branch, and text-generation-webui removed the ExLlamaV2 loader in early 2026 — so advice to download an EXL2 quant is working from stale information. (The V2 repo is not formally GitHub-archived the way TGI's is, which is why it still looks alive at a glance; read the README, not the repo badge.) ExLlamaV3 is the active successor and a near-total rewrite.
Its purpose is maximum quality per gigabyte of VRAM on consumer NVIDIA cards. The EXL3 format is derived from Cornell's QTIP and supports mixed-precision quantization from roughly 2 to 8 bits per weight, automatically allocating more precision to the layers that need it — which holds up far better at low bitrates than uniform methods like GPTQ. In practice that means fitting models on hardware that shouldn't hold them: a 70B becomes viable on a single 24 GB card at aggressive bitrates, and comfortable across two. It also does 2-to-8-bit KV cache quantization, tensor-parallel multi-GPU, and speculative decoding. Serve it through TabbyAPI, the recommended backend, which provides the OpenAI-compatible endpoint. Tradeoffs: setup involves a manual quantization conversion step, the project still describes itself as early, it is best on Ada or newer (Ampere works, older doesn't), and ROCm support is not there yet. Single-stream speed is its strength; it is not a high-concurrency batching server.
Decision guide
Verified 22 Aug 2026The most common mistake is reaching for vLLM when Ollama is enough, or using HF Transformers for serving when a real inference server would be an order of magnitude faster. Identify your actual bottleneck first — capacity, single-stream latency, or concurrent throughput.
| Situation | Reach for |
|---|---|
| "I want to try a local model right now with no setup" | Ollama |
| "I want a GUI — no command line" | LM Studio |
| "I'm on Apple Silicon with 32 GB+ RAM" | MLX |
| "I need frontier closed models in my app, no self-hosting" | OpenRouter |
| "I'm on AWS/GCP/Azure already and need closed models" | Bedrock / Vertex / Azure |
| "I want many providers behind one self-hosted API" | LiteLLM |
| "I'm serving an API to many concurrent users on NVIDIA" | vLLM or SGLang |
| "Every request shares a long system prompt" | SGLang |
| "I've maxed out vLLM and need the last 20%" | TensorRT-LLM |
| "I'm hosting on AWS and want managed serving" | vLLM on SageMaker (Bedrock for closed models) |
| "I've inherited a TGI deployment" | Migrate to vLLM or SGLang (TGI is archived) |
| "My model is in a quant format vLLM won't load" | Aphrodite (AGPL — check licence) |
| "I'm fine-tuning, training, or need model internals" | HF Transformers |
| "I'm embedding inference in a C++ app or edge device" | llama.cpp |
| "I want max tokens/sec from a quantized model on one NVIDIA card" | ExLlamaV3 + TabbyAPI |
| "The model won't fit in my VRAM at all" | llama.cpp (GGUF, CPU offload) |
| "I'm on CPU-only hardware and need reasonable speed" | llama.cpp or Ollama |
| "I'm on AMD" | vLLM on ROCm, or llama.cpp (Vulkan / ROCm) |
| "I'm on Intel Arc" | llama.cpp (Vulkan or SYCL) — not ROCm |