Local LLM Serving: vLLM PagedAttention & GPU Memory Guide
By: Md Tarek · 18 Oct 2026
TL;DR: Serving a 70B model at 3,000+ output tokens/sec on four GPUs is not a model problem — it is an allocator and scheduler problem. This note covers vLLM's PagedAttention memory model, continuous batching with chunked prefill, AWQ/GPTQ quantization reality, and the exact arithmetic plus pre-flight tooling to plan GPU memory before production OOMs.
Executive Summary & Key Takeaways
- PagedAttention cuts KV-cache waste from 60–80% (naive contiguous allocation) to under 4% by storing key/value tensors in fixed-size physical blocks (default
block_size=16tokens) addressed through per-sequence block tables — non-contiguous, shareable, and grown only as tokens are generated. - Memorize the KV formula: `2 x layers x kv_heads x head_dim x dtype_bytes` per token. Llama-3.1-70B in FP16 costs 320 KiB/token across the model (80 KiB/token/GPU at TP=4). An 8K-token sequence burns 640 MiB/GPU, so 32 concurrent 8K sequences need ~20 GiB of KV cache per GPU *before* weights.
- Continuous batching with chunked prefill delivers 10–20x the throughput of naive HuggingFace `generate()` and 2–4x request-level batching, because decode steps are never blocked by a full prefill. Cap
--max-num-batched-tokens(e.g. 8192) to keep inter-token latency p99 in the 30–50 ms band. - AWQ W4A16 (group_size=128) is the pragmatic default for vLLM: ~3.9x smaller weights than FP16, sub-0.1 perplexity drift on Llama-class models with decent calibration, and Marlin kernels that keep decode memory-bound. GPTQ is the fallback when only legacy checkpoints exist — gate both behind
lm-evaluation-harnessbefore cutover. - Plan memory in this order: weights, activation/CUDA-graph overhead, then the KV pool. Set
--gpu-memory-utilization 0.90, derive--max-model-lenand--max-num-seqsfrom the KV budget, and treat--swap-spaceas an availability valve, never a latency strategy.
PagedAttention Internals: Block Tables, Sharing, and Why Fragmentation Collapses
Before PagedAttention, serving frameworks allocated one contiguous KV tensor per sequence sized to max_model_len. If the model supported 8K context but the average response was 400 tokens, you reserved 8K of VRAM and used 5% of it. Worse, because output length is unknowable in advance, every sequence had to be over-reserved for the worst case. The vLLM paper measured this at 60–80% wasted KV memory on real workloads — memory that buys nothing but shrinks how many sequences you can batch.
PagedAttention borrows the OS virtual-memory playbook: split the KV cache into fixed-size blocks (16 tokens by default) and let each sequence own a block table mapping logical block indices to arbitrary physical blocks in a global GPU pool.
Sequence A (7 tokens, block_size=4) logical block 0 -> physical block 7 : [k0 v0][k1 v1][k2 v2][k3 v3] logical block 1 -> physical block 12 : [k4 v4][k5 v5][k6 v6][ -- ] Sequence B (prefix-shared with A) logical block 0 -> physical block 7 (refcount=2) logical block 1 -> physical block 41 : [k4' ...] (copy-on-write on divergence) Physical KV pool (num_gpu_blocks = 1024 blocks x 16 tokens = 16,384 tokens) [ ... ][ 7 ][ ... ][ 12 ][ ... ][ 41 ][ ... ]
Three properties fall out of this design:
1. Internal fragmentation is bounded by `block_size - 1` tokens per sequence (15 tokens, regardless of context length). External fragmentation disappears because every block is identical in size. Measured waste: under 4%. 2. Sharing is free at block granularity. Beam-search branches, parallel sampling (n>1), and shared system prompts point at the same physical blocks with a refcount; a write triggers copy-on-write of one 16-token block, not the whole sequence. 3. Block tables make preemption cheap. When the scheduler evicts a sequence to admit a higher-priority one, it swaps blocks to CPU or drops them for recompute — the block table is the only metadata to move.
In vLLM V1 the KV pool is sized at startup: num_gpu_blocks = (gpu_memory_utilization x total_vram - weights - non_kv_overhead) / (block_size x kv_bytes_per_token). This is the # GPU blocks line printed in the startup banner, and it is the single most important number in your logs.
> War story: A team serving a 70B model on 4xA100-80GB set --max-model-len 32768 because the model card advertised 128K. The banner reported ~29,000 GPU blocks — enough for only ~13 concurrent 32K sequences. Under a 60-request burst the scheduler began swapping KV blocks to CPU (--swap-space defaults to 4 GiB/GPU) and p99 latency hit 11 s. Dropping to --max-model-len 8192 raised concurrency to ~56 sequences and cut p99 to 1.4 s with zero code changes.
Continuous Batching and Chunked Prefill: Scheduling at Iteration Level
Request-level batching (the classic generate() loop) wastes GPU time because sequences finish at different moments: the batch stalls until the longest sequence ends, then the whole batch is discarded and a new one starts. Continuous batching makes a scheduling decision every decode iteration — finished sequences leave, queued sequences join, and batch composition changes token by token.
Iteration: 1 2 3 4 5 6 7
Req A: [P][P][D][D][D][D]
Req B: [P][D][D][D]
Req C: [P][P][D][D][D]
Batch: 1 2 3 3 3 2 1
^A ^B ^C joins while A is still decoding
P = prefill chunk, D = decode stepThe catch is prefill. A 16K-token prompt is one enormous forward pass that saturates the GPU for hundreds of milliseconds. Run it inside a decode iteration and every other sequence's inter-token latency (ITL) spikes — the classic one long prompt ruins p99 for everyone failure. Chunked prefill (enabled by default in the V1 engine) splits prefill into chunks bounded by --max-num-batched-tokens and mixes them with decode tokens in the same iteration. Decode tokens are cheap and keep the SM occupancy high; prefill chunks consume the remaining compute budget.
Scheduler knobs that actually matter in production:
--max-num-batched-tokens(default 8192 in V1): per-iteration token budget for prefill plus decode. Lower it (2048–4096) for tighter ITL p99 at the cost of TTFT; raise it for throughput on latency-tolerant batch jobs.--max-num-seqs(default 256): the concurrency cap and your real memory governor. The KV pool may permit more, but admission control here prevents thrashing and preemption storms.--long-prefill-token-threshold: prompts above this prefill length are scheduled in dedicated chunks rather than aggressively interleaved.--enable-prefix-caching(on by default in V1; disable with--no-enable-prefix-caching): hashes each full block (block_hash = H(parent_hash, token_ids)) and reuses physical blocks across requests. For RAG or agent workloads with a 2K-token shared system prompt, this removes 2K tokens of prefill from every request — commonly a 40–60% TTFT reduction.
> War story: A support-agent workload averaged 1,400-token prompts with a shared 900-token tool-definition prefix, running at 35% GPU utilization with TTFT p99 of 2.6 s. After enabling prefix caching and pinning --max-num-batched-tokens 4096, TTFT p99 fell to 780 ms and throughput rose 2.1x on identical hardware. The only change was a flag — block-hash reuse did the rest.
Quantized Weights: AWQ, GPTQ, and the Kernel Reality
Quantization changes the memory equation in two places: weights shrink (more VRAM for KV, more concurrency) and weight reads shrink (faster decode). The second effect is the subtle one.
Decode is memory-bandwidth-bound at small batch sizes: emitting one token requires streaming every weight matrix through the SMs. On a 4xA100-80GB node (HBM2e at ~2,039 GB/s per GPU, ~8.2 TB/s aggregate), a 141 GB FP16 70B model implies a single-stream ceiling near 58 tokens/s; at AWQ's ~37 GB it rises toward ~200 tokens/s before kernel overheads. As batch size grows, arithmetic intensity rises and decode becomes compute-bound — which is why quantization's *latency* win shrinks at high concurrency while its *VRAM* win (more KV capacity, bigger batch) persists.
| Scheme | Layout | Bytes/weight | vLLM kernels | Calibration | Typical delta PPL (Llama-3 70B) | | --- | --- | --- | --- | --- | --- | | FP16 / BF16 | W16A16 | 2.0 | cuBLAS, FlashAttention | No | 0.00 (reference) | | FP8 (W8A8) | W8A8 dynamic | 1.0 | CUTLASS FP8 (Hopper+) | No | +0.02 | | AWQ | W4A16 g128 | 0.5 (+~5%) | Marlin / AWQ fused | Yes (128–512 samples) | +0.05–0.10 | | GPTQ | W4A16 g128 | 0.5 (+~6%) | Marlin (act_order off) | Yes (Hessian, 128+ samples) | +0.07–0.15 | | GGUF Q4_K_M | W4A16 | ~0.56 | llama.cpp (not vLLM-native) | No | +0.10–0.20 |
AWQ (Activation-aware Weight Quantization) protects the ~1% of weight channels with the largest activation magnitudes by scaling them before rounding, then applies per-group (default 128) integer scales. Because it optimizes for *activations*, it preserves quality better than plain round-to-nearest at the same bit width, and vLLM's Marlin kernels keep 4-bit weights in registers while dequantizing on the fly.
GPTQ is layer-wise post-training quantization using second-order (Hessian) information to minimize reconstruction error. act_order=True reorders columns by activation magnitude and improves quality slightly, but disables the fastest Marlin path in vLLM. For new deployments, AWQ is usually the better default; GPTQ's advantage is the sheer volume of pre-quantized community checkpoints.
Load-time validation you must not skip: vLLM silently falls back to unquantized or slower kernels when the kernel does not support your head dim, group size, or GPU architecture. Grep the startup log for marlin, awq, or gptq, and confirm reported weight memory matches your quantization expectation. A 70B AWQ model reporting ~70 GiB of weights means quantization was ignored.
GPU Memory Planning: The Arithmetic That Prevents OOM
Every vLLM deployment is an exercise in one inequality:
weights_per_gpu + non_kv_overhead + kv_pool <= gpu_memory_utilization x total_vram
Where:
weights_per_gpu = (params x bytes_per_weight x quant_overhead) / tp_size, plus replicated embeddings and lm_head — budget +2–4% for TP>1.non_kv_overhead= activations, CUDA-graph capture pools, NCCL buffers, framework allocations. Empirically 1.5–3.0 GiB/GPU for 70B-class models with piecewise CUDA graphs; more with--enforce-eagerand large--max-num-batched-tokens.kv_poolis the remainder, converted to tokens viakv_bytes_per_token_per_gpu.
Worked example — Llama-3.1-70B, TP=4, 4xA100-80GB, `--gpu-memory-utilization 0.90`:
| Quantity | FP16 | AWQ W4A16 | | --- | --- | --- | | Usable VRAM/GPU (0.90 x 80 GiB) | 72.0 GiB | 72.0 GiB | | Weights/GPU | 33.9 GiB | 8.9 GiB | | Non-KV overhead | 2.5 GiB | 2.5 GiB | | KV pool/GPU | 35.6 GiB | 60.6 GiB | | KV per token (80 KiB) | 80 KiB | 80 KiB | | Max KV tokens | ~466K | ~794K | | Max concurrent 8K sequences | ~56 | ~96 | | Recommended --max-num-seqs | 48 | 80 |
Quantization buys roughly 1.8x more concurrency here — the real reason to quantize at scale, more than the per-token latency improvement.
Practical Implementation
The script below does the three things production deployments actually need: measure real VRAM, compute a plan from model architecture constants, and launch vllm serve only if the plan fits — with a health check and clean shutdown. Use it as a pre-flight gate in your container entrypoint.
#!/usr/bin/env python3
# plan_and_serve.py — vLLM pre-flight memory planner + hardened server launcher.
#
# Example:
# python plan_and_serve.py --model meta-llama/Llama-3.1-70B-Instruct --tp 4 --quant awq \
# --layers 80 --kv-heads 8 --head-dim 128 --params-b 70.6 --target-seq-len 8192 --launch
#
# Exit codes: 0 = plan fits / clean shutdown, 2 = plan does not fit, 3 = runtime error.
from __future__ import annotations
import argparse
import json
import logging
import signal
import subprocess
import sys
import time
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass, field
from typing import Final
import torch
LOG = logging.getLogger('vllm-plan')
BYTES_PER_WEIGHT: Final[dict[str, float]] = {'fp16': 2.0, 'fp8': 1.0, 'awq': 0.5, 'gptq': 0.5}
QUANT_OVERHEAD: Final[dict[str, float]] = {'fp16': 1.00, 'fp8': 1.02, 'awq': 1.05, 'gptq': 1.06}
NON_KV_OVERHEAD_GIB_PER_GPU: Final[float] = 2.5
GIB: Final[int] = 1 << 30
@dataclass(frozen=True)
class ModelSpec:
params_b: float
num_layers: int
num_kv_heads: int
head_dim: int
dtype_bytes: int = 2 # fp16/bf16 KV cache
@dataclass
class GpuPlan:
gpu_count: int
tp_size: int
total_vram_gib: float
usable_vram_gib: float
weights_per_gpu_gib: float
kv_pool_gib: float
kv_bytes_per_token_per_gpu: int
max_kv_tokens: int
max_concurrent_seqs: int
recommended_max_num_seqs: int
recommended_max_model_len: int
fits: bool
warnings: list[str] = field(default_factory=list)
def measure_vram_gib(index: int = 0) -> float:
# Total VRAM in GiB for the given CUDA device index.
if not torch.cuda.is_available():
raise RuntimeError('CUDA is unavailable: cannot plan GPU memory.')
if index >= torch.cuda.device_count():
raise RuntimeError(f'Requested GPU {index}, but only {torch.cuda.device_count()} visible.')
_free_bytes, total_bytes = torch.cuda.mem_get_info(index)
return total_bytes / GIB
def kv_bytes_per_token(spec: ModelSpec, tp_size: int) -> int:
# KV cache cost per token, per GPU, after tensor-parallel head sharding.
if spec.num_kv_heads % tp_size != 0:
raise ValueError(
f'num_kv_heads={spec.num_kv_heads} is not divisible by tp_size={tp_size}; '
'choose a TP size that divides the KV head count (GQA constraint).'
)
kv_heads_per_gpu = spec.num_kv_heads // tp_size
return 2 * spec.num_layers * kv_heads_per_gpu * spec.head_dim * spec.dtype_bytes
def build_plan(spec: ModelSpec, *, gpu_count: int, tp_size: int, quant: str,
gpu_memory_utilization: float, max_model_len: int,
avg_seq_len: int) -> GpuPlan:
if not 0.5 <= gpu_memory_utilization <= 0.98:
raise ValueError('gpu_memory_utilization should be within [0.5, 0.98].')
if tp_size > gpu_count:
raise ValueError(f'tp_size={tp_size} exceeds visible GPUs ({gpu_count}).')
warnings: list[str] = []
total_vram = measure_vram_gib(0)
usable = total_vram * gpu_memory_utilization
weights_total_gib = spec.params_b * 1e9 * BYTES_PER_WEIGHT[quant] * QUANT_OVERHEAD[quant] / GIB
if tp_size > 1:
weights_total_gib *= 1.03 # replicated embeddings, lm_head, TP communication slack
weights_per_gpu = weights_total_gib / tp_size
if weights_per_gpu > 0.75 * usable:
warnings.append(
f'Weights consume {weights_per_gpu / usable:.0%} of usable VRAM; the KV pool will be tiny. '
'Quantize harder or add GPUs.'
)
kv_pool = usable - weights_per_gpu - NON_KV_OVERHEAD_GIB_PER_GPU
if kv_pool <= 0:
return GpuPlan(gpu_count, tp_size, total_vram, usable, weights_per_gpu, 0.0, 0, 0, 0, 0,
max_model_len, False, warnings + ['KV pool is negative: OOM guaranteed.'])
kv_per_token = kv_bytes_per_token(spec, tp_size)
max_kv_tokens = int(kv_pool * GIB / kv_per_token)
max_concurrent = max_kv_tokens // max(1, avg_seq_len)
# 15% headroom for block tables, preemption churn, and prefix-cache refcounts.
recommended_seqs = int(max_concurrent * 0.85)
if kv_pool * GIB / kv_per_token < max_model_len:
warnings.append('KV pool cannot hold a single max_model_len sequence; reduce --max-model-len.')
if quant == 'gptq':
warnings.append('GPTQ with act_order=True disables the fastest Marlin kernels; prefer AWQ for new work.')
return GpuPlan(
gpu_count=gpu_count, tp_size=tp_size, total_vram_gib=round(total_vram, 1),
usable_vram_gib=round(usable, 1), weights_per_gpu_gib=round(weights_per_gpu, 2),
kv_pool_gib=round(kv_pool, 2), kv_bytes_per_token_per_gpu=kv_per_token,
max_kv_tokens=max_kv_tokens, max_concurrent_seqs=max_concurrent,
recommended_max_num_seqs=max(1, recommended_seqs),
recommended_max_model_len=max_model_len, fits=recommended_seqs >= 1, warnings=warnings,
)
def wait_for_health(port: int, timeout_s: float = 900.0) -> bool:
url = f'http://127.0.0.1:{port}/health'
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(url, timeout=5) as resp:
if resp.status == 200:
return True
except (urllib.error.URLError, ConnectionError, OSError):
time.sleep(2.0)
return False
def launch_server(model: str, plan: GpuPlan, quant: str, port: int, extra: list[str]) -> int:
cmd = [
sys.executable, '-m', 'vllm.entrypoints.openai.api_server',
'--model', model,
'--tensor-parallel-size', str(plan.tp_size),
'--max-model-len', str(plan.recommended_max_model_len),
'--max-num-seqs', str(plan.recommended_max_num_seqs),
'--gpu-memory-utilization', '0.90',
'--block-size', '16',
'--enable-prefix-caching',
'--disable-log-requests',
'--port', str(port),
]
if quant != 'fp16':
cmd += ['--quantization', quant]
cmd += extra
LOG.info('launching: %s', ' '.join(cmd))
proc = subprocess.Popen(cmd, start_new_session=True)
def _shutdown(signum, _frame):
LOG.warning('signal %s received; terminating server group', signum)
proc.terminate()
signal.signal(signal.SIGTERM, _shutdown)
signal.signal(signal.SIGINT, _shutdown)
if not wait_for_health(port):
LOG.error('server failed health check within timeout; dumping plan')
proc.terminate()
return 3
LOG.info('server healthy on port %s — plan: %s', port, json.dumps(asdict(plan)))
return proc.wait()
def main() -> int:
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(name)s %(message)s')
p = argparse.ArgumentParser(description='vLLM memory planner + launcher')
p.add_argument('--model', required=True)
p.add_argument('--tp', type=int, default=1)
p.add_argument('--quant', choices=sorted(BYTES_PER_WEIGHT), default='fp16')
p.add_argument('--layers', type=int, required=True)
p.add_argument('--kv-heads', type=int, required=True)
p.add_argument('--head-dim', type=int, default=128)
p.add_argument('--params-b', type=float, required=True)
p.add_argument('--max-model-len', type=int, default=8192)
p.add_argument('--avg-seq-len', type=int, default=4096)
p.add_argument('--gpu-memory-utilization', type=float, default=0.90)
p.add_argument('--port', type=int, default=8000)
p.add_argument('--launch', action='store_true')
p.add_argument('--dry-run', action='store_true')
args, extra = p.parse_known_args()
spec = ModelSpec(params_b=args.params_b, num_layers=args.layers,
num_kv_heads=args.kv_heads, head_dim=args.head_dim)
plan = build_plan(spec, gpu_count=torch.cuda.device_count(), tp_size=args.tp,
quant=args.quant, gpu_memory_utilization=args.gpu_memory_utilization,
max_model_len=args.max_model_len, avg_seq_len=args.avg_seq_len)
print(json.dumps(asdict(plan), indent=2))
for w in plan.warnings:
LOG.warning(w)
if not plan.fits:
return 2
if args.dry_run or not args.launch:
return 0
return launch_server(args.model, plan, args.quant, args.port, extra)
if __name__ == '__main__':
raise SystemExit(main())Then gate the rollout with the planner, launch, and benchmark with vLLM's own load generator:
# 1) Dry-run the planner before downloading weights. python plan_and_serve.py --model meta-llama/Llama-3.1-70B-Instruct --tp 4 --quant awq --layers 80 --kv-heads 8 --head-dim 128 --params-b 70.6 --max-model-len 8192 --avg-seq-len 8192 --dry-run # 2) Launch the OpenAI-compatible server with the plan's derived limits. python plan_and_serve.py --model meta-llama/Llama-3.1-70B-Instruct --tp 4 --quant awq --layers 80 --kv-heads 8 --head-dim 128 --params-b 70.6 --max-model-len 8192 --avg-seq-len 8192 --launch # 3) Measure p50/p90/p99 TTFT, TPOT and ITL against the real endpoint. vllm bench serve --backend openai-chat --base-url http://127.0.0.1:8000 --model meta-llama/Llama-3.1-70B-Instruct --endpoint /v1/chat/completions --dataset-name random --random-input-len 1024 --random-output-len 256 --num-prompts 2000 --request-rate 32 --max-concurrency 64 --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90,99
Performance Benchmarks & Trade-Offs
Representative measurements for Llama-3.1-70B-Instruct, 4xA100-80GB SXM (NVLink), vLLM v0.8.x V1 engine, 1,024-token prompts / 256-token outputs, 64 concurrent streams, --max-num-batched-tokens 8192. Treat as directional and re-measure on your silicon.
| Configuration | Output tok/s | TTFT p50 / p99 | ITL p50 / p99 | KV pool/GPU | Max 8K seqs | delta PPL | | --- | --- | --- | --- | --- | --- | --- | | HF transformers FP16, batch_size=1 | ~48 | 890 ms / 4.2 s | 21 ms / 96 ms | n/a (contiguous, 8K over-reserved) | 1 | 0.00 | | vLLM FP16, TP=4, chunked prefill off | 1,980 | 180 ms / 1.9 s | 16 ms / 340 ms | 35.6 GiB | ~56 | 0.00 | | vLLM FP16, TP=4, --max-num-seqs 48 | 1,850 | 240 ms / 610 ms | 16 ms / 41 ms | 35.6 GiB | ~56 | 0.00 | | vLLM FP16 + prefix caching (2K shared prefix) | 2,340 | 105 ms / 280 ms | 16 ms / 39 ms | 35.6 GiB | ~56 | 0.00 | | vLLM AWQ W4A16 g128, TP=4, --max-num-seqs 80 | 3,050 | 215 ms / 520 ms | 12 ms / 31 ms | 60.6 GiB | ~96 | +0.06 | | vLLM GPTQ W4A16 g128 (Marlin), TP=4 | 3,180 | 205 ms / 500 ms | 11 ms / 29 ms | 60.2 GiB | ~96 | +0.09 | | vLLM FP8 W8A8, 4xH100-80GB SXM, TP=4 | 4,420 | 130 ms / 300 ms | 8 ms / 21 ms | 55.0 GiB | ~72 | +0.02 |
How to read this table:
- Chunked prefill is a p99 instrument, not a throughput instrument. Turning it off barely moves throughput but explodes ITL p99 from 41 ms to 340 ms — one long prompt stalls every decoding sequence.
- Prefix caching is the cheapest TTFT win available. It costs a hash table and no extra VRAM per se, and it cut TTFT p99 by more than 50% here.
- 4-bit quantization buys concurrency first, latency second. AWQ raised the KV pool from 35.6 GiB to 60.6 GiB (1.7x more sequences), while per-token ITL improved only ~25%.
- FP8 on Hopper is the best quality/perf point if the hardware exists — near-FP16 accuracy with a large throughput jump, no calibration pipeline to maintain.
Pro-Tips & Common Pitfalls
- Never push `--gpu-memory-utilization` above 0.92 on a shared GPU. CUDA-graph capture and NCCL buffers sit outside the planner's model. 0.90 with
--max-num-batched-tokens 8192is the safe default. If startup OOMs, lower--max-num-seqsfirst, then--max-model-len. - Do not confuse `--max-model-len` with your KV budget. A 128K-context model with a 10 GiB KV pool admits ~2 concurrent full-length sequences. Advertise the context you can *serve concurrently*, not the context the weights support.
- `--swap-space` is an availability valve, not a latency strategy. CPU-swapped KV blocks turn 40 ms ITL into 300+ ms. If
num_preemptionsclimbs in Prometheus, you are over-admitting. - GQA plus TP divisibility:
num_kv_heads % tensor_parallel_size == 0, or vLLM duplicates KV heads (num_kv_head_replicas) and wastes memory. Llama-3 70B has 8 KV heads — TP 1/2/4/8 works; TP 3 or 6 does not. - Watch for silent quantization fallback. If the startup banner shows FP16-sized weights for an AWQ checkpoint, the kernel rejected your head dim or group size. Grep for
awqormarlinin the logs. - Prefix caching plus dynamic prompts equals cache thrash. Hash reuse requires byte-identical prefixes. Timestamps, request IDs, or per-user JSON in the system prompt destroy hit rate — keep the cacheable prefix static and push variables into the user turn.
- Benchmark with the real length distribution. Random synthetic lengths hide prefill-spike behavior. Replay production traces (or at minimum mix 256/1K/4K/16K prompts) before trusting p99.
- Tokenize before admission control. A 40K-token prompt that exceeds
max_model_lenshould be rejected at the gateway, not after it consumes a scheduler slot and a 30-second prefill. - Pin versions and re-run evals on upgrade. V1 engine defaults, chunked prefill behavior, and Marlin kernel coverage changed materially between vLLM 0.6 and 0.9.
- Monitor exactly four metrics:
vllm:gpu_cache_usage_perc(above 0.9 means admission pressure),vllm:num_requests_waiting(queue depth),vllm:time_to_first_token_secondsp99, andvllm:num_preemptions_total.
Frequently Asked Questions (FAQ)
What is PagedAttention and why does it matter more than any other vLLM optimization?
PagedAttention is vLLM's block-based KV cache allocator. Instead of one contiguous tensor per sequence sized to max_model_len, the cache is a pool of fixed-size physical blocks (16 tokens by default) referenced by per-sequence block tables — the same indirection an OS uses for virtual memory. This removes external fragmentation entirely, bounds internal waste to at most 15 tokens per sequence, and makes block-level sharing possible via refcounts and copy-on-write for beam search, parallel sampling, and prefix caching. The practical effect is that KV waste drops from 60–80% to under 4%, which directly multiplies the batch size your GPUs can sustain — and throughput scales with batch size, not with any single kernel-level tweak.
Should I use AWQ or GPTQ for a production vLLM deployment?
AWQ by default for new deployments: it is activation-aware, delivers better quality at the same 4-bit width, and has mature Marlin kernels in vLLM. Use group_size=128 unless your eval shows a specific task regression, in which case try 32 at a small throughput cost. Choose GPTQ mainly when you must reuse an existing checkpoint (the public selection is larger) and accept slightly higher quality risk; avoid act_order=True if you want the fastest Marlin path. In either case, gate the cutover on lm-evaluation-harness across MMLU, GSM8K, and a domain-specific slice, and treat more than 1% task degradation as a stop signal. If you have H100/H200 hardware, FP8 W8A8 beats both on quality and throughput with no calibration pipeline.
How do I calculate `--max-model-len` and `--max-num-seqs` that fit my GPUs?
Follow four steps. 1) Compute KV bytes per token per GPU: 2 x layers x (kv_heads / tp) x head_dim x dtype_bytes. For Llama-3.1-70B at TP=4 that is 2 x 80 x 2 x 128 x 2 = 80 KiB/token/GPU. 2) Compute weights per GPU: params x bytes_per_weight x overhead / tp; FP16 gives ~33.9 GiB, AWQ ~8.9 GiB. 3) Subtract weights and a 2.5 GiB non-KV overhead from gpu_memory_utilization x total_vram (72 GiB at 0.90 on an 80GB card) to get the KV pool — 35.6 GiB FP16, 60.6 GiB AWQ. 4) Divide the pool by KV-per-token for total tokens (~466K FP16), then divide by your p99 sequence length. At 8K tokens that is ~56 sequences; set --max-num-seqs to about 85% of that (48) to leave room for preemption. The plan_and_serve.py script above automates exactly this arithmetic and refuses to launch when the plan does not fit.
