Speculative Decoding.
A small draft model proposes tokens ahead; the resident model verifies them in one pass. Quality is unchanged. Sharded placements only — and on the one pool that has been measured, it bought nothing.
A draft model proposes several tokens ahead; the big model verifies them in one forward pass and keeps whatever matches its own greedy output. Every emitted token is still exactly what the big model would have produced alone, so quality is unchanged, and a batch of draft tokens is verified for roughly the cost of generating one.
docs/BENCHMARKS.md closes Config B by saying the speedup should not be claimed until a pool is measured that actually shows it, so this page doesn’t claim one. Sharded placements only
internal/llama emits the draft flags only when the launch options carry RPC backends — that is, only when the placement is pipeline-sharded across machines. A whole-model replica never receives them, on the host or on a node.
The reason is the network hop. A shard pays a round trip at every pipeline stage boundary, so its per-token cost is dominated by latency rather than compute, and verifying k drafted tokens in one pass amortises that hop across the batch: the cost goes from k × (compute + hops) to k × compute + hops. A whole replica has no hop to amortise. Measuring one showed speculation doing nothing whatsoever — 49.5 against 49.55 tok/s — so the flags are withheld there rather than paying for a second set of weights nobody benefits from.
llama-server (pin it with RELAY_DRAFT_GPU_LAYERS) and only the verification pass traverses the pipeline — which is exactly the expensive step speculation exists to batch. If you are reading older notes that say “host replicas only”, they describe the inverted behaviour and no longer match the code. The host has to have room for it, and nothing in the fit-check knows that
The placement fit-check does not know the draft model exists — not the planner’s per-stage sizing, not replicaNeedMB. draftModelFor in internal/daemon is the guard that keeps a plan the planner called valid from OOM-ing at launch. It drops the draft model, quietly and with a log line, in three cases:
| When | Why |
|---|---|
| The plan puts no layers on the host at all | Speculation runs where llama-server runs. With no host stage there is nothing for the draft model to sit beside. |
| The draft file cannot be sized | A stat failure means its VRAM cost is unknown, and an unknown cost cannot be checked against free VRAM. |
| The host's own stage plus the draft would exceed free VRAM | The draft model is charged against the same card as the host's share of the model. Loading both when only one fits is the OOM this guard exists to prevent. |
This is on the launch path rather than advisory — draftModelFor’s return value is what populates Options.DraftModelPath. So a configured draft model can be silently absent from a given placement even on a shard, and that is working as intended.
The draft model must share a tokenizer
A draft’s proposed token IDs are meaningless to the big model unless they share a vocabulary. In practice that means a smaller model from the same family: Qwen2.5-Coder-0.5B drafting for Qwen2.5-Coder-14B, not an unrelated model.
Configuration
| Variable | Default | Purpose |
|---|---|---|
RELAY_DRAFT_MODEL | — (empty) | Path to the draft GGUF; empty disables speculative decoding entirely (the default). Set, it still only takes effect on a sharded placement whose host stage has room for it. |
RELAY_DRAFT_MAX | 16 | --spec-draft-n-max: max tokens the draft may propose ahead of verification. Must be >= 0. |
RELAY_DRAFT_MIN | 0 (flag omitted) | --spec-draft-n-min: minimum draft tokens before llama-server verifies a batch. At 0 the flag is omitted and llama-server's own default applies. |
RELAY_DRAFT_GPU_LAYERS | 99 | --spec-draft-ngl: draft model layers offloaded to GPU. 99 offloads it in full, which is also how you keep it resident on the machine running llama-server. |
DefaultDraftMax is 16, not llama-server’s own default of 3: a coding assistant’s replies lean heavily on predictable boilerplate (braces, imports, repeated identifiers) that a small same-tokenizer draft model gets right far more than three tokens’ worth of the time. Tune it down if a particular draft model’s hit rate is low.
DefaultDraftGPULayers is 99 (offload the draft model in full) because a usable draft model is, by construction, a small fraction of the size of the model it primes; offloading it fully rarely competes for VRAM the main model needs.
RELAY_DRAFT_MODEL that does not resolve to a file (missing, or a directory) is logged as a warning and cleared at startup rather than failing the daemon. A stale or typo’d path is a tuning knob to fix later, not a reason the pool should refuse to serve its main model. The emitted flags
Order matters: --rpc must precede --device and --tensor-split; the KV-cache type and flash-attention flags follow --alias; the draft flags are appended last, and only when RPCBackends is non-empty.
llama-server --model /models/qwen2.5-coder-14b-q3_k_m.gguf \
--rpc 10.0.0.24:50052 --device RPC0 --tensor-split 20,28 \
--host 127.0.0.1 --port 8384 --cont-batching --parallel 2 --jinja \
--n-gpu-layers all --ctx-size 16384 --alias qwen2.5-coder-14b \
--cache-type-k f16 --cache-type-v f16 \
--spec-draft-model /models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
--spec-draft-n-max 16 --spec-draft-ngl 99The same launch without --rpc — a whole-model replica on the host — gets no --spec-draft-* flags at all, whatever RELAY_DRAFT_MODEL is set to.
With RELAY_KV_CACHE_TYPE=q8_0 the cache flags become --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on. See Concepts.