Benchmarks.

The question this page answers is not “how many tokens per second.” It is: which real workloads are viable on a pool, and which are not. A pool that can load a 32B model but takes nine seconds to start answering is a demo, not a tool.

How this is measured

Every number is wall-clock, measured at the client, over the real network path — not llama.cpp’s own internal timings, which cannot see the hop from editor to hub to node and back. That hop is the thing under suspicion, so it has to be inside the measurement.

Cold start is tracked separately: it is paid once, and is annoying rather than disqualifying provided it is not paid often.

Viability thresholds

Fixed before any measurement, from how people actually work, so results are judged rather than rationalised.

Interactive chat

A person reads prose at roughly 250 words per minute, about 5.5 tok/s. Generation slower than that means watching the model type.

Inline code completion

This is the hardest bar in the document, and the one most sensitive to the network hop.

Agentic automation (a coding agent driving the pool)

This is where a pool either earns its place or does not, and it is arithmetic rather than feel: a task of ~20 tool turns pays TTFT twenty times, not once; context grows through the task, often to 10k–50k tokens; and each turn re-processes that context unless it is cached. At 500 tok/s of prompt processing, a 30k-token context costs 60 s per turn before a single token is generated — 20 minutes of pure overhead across a task. At 5,000 tok/s the same turn costs 6 s.

Prompt caching changes this picture substantially and is measured separately — llama.cpp’s per-slot cache already gives a large speedup on repeated prefixes, which is exactly the shape of an agent loop.

Concurrency

Method

scripts/bench_matrix.py --url <hub> --key <api key> --cacert <hub CA> \
    --models solo,coder-14b --contexts 32,2000,8000 \
    --runs 5 --concurrency 1,2,4 --max-tokens 128 \
    --out bench-<config>.json --note "<hardware description>"

Each configuration writes a JSON record per request, so results across different hardware are compared from data rather than from memory.

Config A — hub GTX 1060 (6 GB) + node RTX 4070 SUPER lending 8 GB

Hub and node on the same LAN, 1 ms RTT, node reached over Tailscale, client on the hub machine. solo is a 4.4 GB model placed as a whole replica on the node; qwen3-4b was sharded across host and node because solo’s replica already held the node’s VRAM.

Whole-model replica on a remote node (solo)

Concurrency (solo, ctx 2,000, 128 tokens each)

Throughput rises with load rather than collapsing, and no request failed.

Sharded across two machines (qwen3-4b, 9 layers host / 27 node, before adaptive slot sizing existed)

The warm row is the prefix cache, not the fabric: a repeated prompt skips prompt evaluation almost entirely even on a shard. It is reported because an agent loop genuinely sees it, but it must not be read as the cost of a cold context, which is the 13.2 s row.

Verdict against the thresholds

A whole-model replica on a pooled machine is comfortably viable for interactive and agentic work. The network hop costs about 25 ms on a warm request — nowhere near the budget. Generation at 93 tok/s is roughly 17x reading speed.

Sharding a model across machines is a different product. Prompt processing drops to ~135 tok/s, roughly 33x slower, because activations cross the network at every stage boundary. A 4,000-token prompt then costs about 30 s before the first token. Sharding is what makes a too-big model run at all; it is not what makes it pleasant.

In this single-stream test, the concurrency caveat is honest but mild: TTFT rises 9x from a very low floor (22 ms to 203 ms) and remains well inside the 1 s chat budget. That table sends each client the same prompt, though, so every request past the first lands on a cache hit — it is not a measurement of genuinely concurrent load. Config C, below, redoes this with a unique prompt per client and finds a different picture at 4 concurrent requests.

The controlled replica-vs-shard comparison

Adaptive slot sizing makes the same model placeable both ways on this hardware, so the cost of sharding can be measured directly rather than inferred across two different models. qwen3-4b, ctx 2,000, identical hardware, identical prompts — only the placement differs.

36x, on both numbers. Two machines’ VRAM added together is not the same product as one machine holding the model: crossing the network at every stage boundary costs more than an order of magnitude. Sharding earns its place by making a too-big model run at all, and should be described that way rather than as the pooling story’s happy path. The corollary is that anything which pushes a model off the replica path and onto the shard path is expensive out of all proportion to the VRAM it saves — which is why the slot count is sized to the node rather than fixed.

What this configuration could not test

Config B — speculative decoding on a sharded placement

The premise: a shard pays a network round trip at every pipeline stage boundary, so drafting k tokens and verifying them in one pass should amortise that hop. Published work measures 2.5x in that regime, and a llama.cpp discussion reports 1.84–2.21x for a target split across two RTX 4070s. On this pool it produced no gain. See Speculative Decoding for how the mechanism works.

Target Qwen2.5-Coder-14B-Instruct-Q3_K_M, sharded 20 layers on the host (GTX 1060, Vulkan) and 28 on the node (RTX 4070 SUPER), ctx 4096 x 2 slots. Draft Qwen2.5-0.5B-Instruct-Q4_K_M, same tokenizer family, --spec-draft-ngl 99. Identical benchmark either side.

Every difference is inside run-to-run noise.

The likely reason it did not help is that the hop is not this pool’s bottleneck. Speculation pays when link latency dominates compute; the published regime is roughly 3x compute < latency < 10x compute. Here the host holds 20 of 48 layers on a GTX 1060 over Vulkan, and at 10.4 tok/s a token costs ~96 ms — far more than a small activation tensor crossing a 1 GbE LAN. Amortising a hop that was never the expensive part buys nothing.

What this does not establish is that speculation never helps a shard. The reported 1.84x pairs two RTX 4070s, where host compute is fast enough for the hop to matter — a pool of two fast, evenly matched cards is the case still untested here. What it does establish is that the win is conditional on the hardware, and this pool is not in the regime.

The change is kept because it costs nothing when RELAY_DRAFT_MODEL is unset (the default) and it fixed a real VRAM-accounting gap on the way. But the speedup should not be claimed until a pool is measured that actually shows it.

Config C — what concurrency does to each placement

The earlier configurations measured one request at a time. That flattered the shard: it compared throughput, and throughput was merely bad. Measured under concurrent load, the two placements do not differ in degree. They differ in kind.

Same pool as Config A. solo (4.4 GB, 28 layers) is a whole-model replica on the node; a 14B Q3_K_M model, sharded 20 layers on the host and 28 on the node, stands in for the shard path. Context ~2,000 tokens, with a unique nonce per request so no request can hit another’s prefix cache.

Aggregate generation throughput, tokens/sec across all concurrent requests

Time to first token at the same load

A replica scales: four concurrent requests produce 2.8x the tokens of one, and the slowest client still sees its first token in under 2.3 s.

A shard does not scale at all. Four concurrent requests produce the same total tokens as one — the per-request rate collapses from 11.0 to as low as 1.2 tok/s — while the slowest client waits a minute and a half for its first token. Nothing errors and no request is dropped; the pool simply stops making progress proportional to what is asked of it.

Cold start compounds this. A sharded model loading from an evicted pool took 96.5 s to first token, of which roughly 95 s is weight loading and RPC setup rather than compute; a second measurement on a 14B shard reached 3m44s. A user who has not been warned assumes the process has hung.

What this means for what relay is

Sharding is not a slower version of pooling. It is a different product, and the concurrency numbers are what make that unambiguous.

• A whole-model replica on a lent GPU is a serving tier. It scales with load, it answers in under a second, and a household or small team can share it. This is the case relay should be sold on.
• A shard is a capability tier. It runs a model no single machine in the pool could hold at all, at roughly a thirtieth of the prompt throughput and a ninth of the generation throughput, for one user at a time. It is the answer to “can I run this model” and never the answer to “can we all use this model.”

The planner already encodes this preference — replica first, shard as fallback — and relay status already warns when a placement fell back. These numbers are the justification for both, and the reason the fallback warning should stay loud.