Concepts.

Vocabulary

The hub owns every model

Weights live on the hub's disk only. Nodes never fetch from the internet; the hub pushes a hash-verified copy of a GGUF into a node's cache before it can run anything, and that cache is a copy the hub granted, not a source of truth. Nodes are capacity, not independent inference servers.

 client ──/v1──▶ HUB ─────────────────────────────────────────────────────
                  │ scheduler: replica (host) │ replica (node) │ shard
                  │ store: models, nodes, replicas, shards, files
                  │ llama-server (host replica / shard head)
                  │ tunnel terminator ── one yamux session per node
                  │     streams: control │ rpc │ http │ file
                  ▼
              NODE (relay up / relay-tray)
                  │ ggml-rpc-server  ← rpc streams   (shard stage)
                  │ llama-server     ← http streams  (replica)
                  │ model cache      ← file streams  (hub pushes weights)
                  │ availability: idle/busy/paused, schedule, tray toggle

Placement order: replica first, shard as fallback

For a request naming model M, Ensure tries, in order:

A replica is torn down when its node disconnects or pauses. On the pause path the hub also tells the node to stop its llama-server (30s timeout) so the card is actually released; on reconnect, a node reporting a replica the hub no longer expects is told to stop it.

Shard device order: the hub's GPU runs last

Within a shard, the pipeline stages are ordered so the hub's own GPU carries the last layers, not the first — the planner's free-VRAM/throughput split runs as normal, then hostStageLast moves whichever stage landed on the host to the end of the sequence and renumbers every stage's layer range to match.

The output layer and sampling always run on whichever device holds the last stage, so putting the hub there means the ~600 KB logits vector at the vocabulary size this pool uses never has to cross the network on every generated token — only a hidden state does. Measured on this pool's own fingerprint, that reordering alone raised a 14B shard's generation throughput 32-40% and was the difference between serving one of four concurrent clients and serving all four.

How much VRAM a replica needs

KV       = kv_bytes_per_token × ctx × slots × kv_scale
overhead = weights × 3% + 384 MB + KV × 5%
need     = weights + KV + overhead

ctx      = RELAY_CONTEXT_SIZE, or min(RELAY_REPLICA_CTX, RELAY_CONTEXT_SIZE) when RELAY_REPLICA_CTX > 0
slots    = RELAY_PARALLEL_SLOTS
kv_scale = 1.0 (f16) · 0.5 (q8_0) · 0.28 (q4_0)   ← RELAY_KV_CACHE_TYPE

weights is the registered required_vram_mb, falling back to the file size. The KV term is zero when the GGUF header gave no per-token estimate. The breakdown is logged at debug level on every fit-check.

KV-cache quantization

RELAY_KV_CACHE_TYPE sets the K/V cache precision (--cache-type-k/--cache-type-v) for every llama-server the daemon launches or asks a node to launch: host replicas, node replicas, and shards. f16 is llama-server's own default. q8_0 halves the KV footprint at close to no quality loss; q4_0 roughly quarters it at a measurable cost. Because the fit-check uses the same scale, quantizing the cache is the intended lever for a model whose weights fit a node but whose full-precision KV reservation pushes it a few hundred MB over.

llama.cpp needs flash attention to run a quantized V cache, so a quantized type forces --flash-attn on unless RELAY_FLASH_ATTN says otherwise. The type in effect is reported as kv_cache_type in /status.

Mixture-of-experts placement

A GGUF header can carry expert_count and expert_used_count: how many experts each layer holds, and how many of them a single token is routed to. Both zero means dense — every model that predates this field — so a dense model's plan is unchanged. A model counts as MoE only when both are set: Qwen3-30B-A3B holds 128 experts per layer and routes each token to 8, so a sixteenth of each layer's expert weights is read for any given token and the rest sit idle.

A dense shard (step 4 above) moves a whole layer — attention and every expert alike — to whichever device the pipeline plan assigned that layer to, so every token passing through a remote layer pays a crossing. llama.cpp also fuses a layer's experts into one tensor per projection (blk.N.ffn_gate_exps.weight holds all 128), so there is no tensor name that selects a range of expert indices — placement can move a layer's whole expert block to a device, never a slice of its experts. That is the split this implements: for an MoE model, the daemon reuses the exact layer boundaries the shard's pipeline plan already committed to and emits an extra --override-tensor per remote stage, naming only that layer range's expert tensors. Attention, the router, and the embedding are never named in an override, so they stay wherever --device/--tensor-split already put them — on the host for every layer, since those tensors are read on every token regardless of routing. The result: a dense shard pays a crossing per token per layer boundary, while this pays a crossing only when a token's router sends it to an expert whose block lives on a remote device.

The layer ranges come from the shard's own pipeline plan, not from PlanExperts's proportional division by free VRAM — disagreeing with the layers llama.cpp was actually told to load would pin tensors to a device that doesn't hold them. A remote stage the daemon can't resolve to a device name is left out of the override set entirely rather than emitted malformed, and any error building the overrides falls back to a plain layer split with a logged warning — never a placement that pins tensors somewhere wrong.

Replicas scale linearly; sharding is turn-taking

Measured on two machines over Tailscale (3 ms RTT): a GTX 1060 6 GB hub and an RTX 4070 SUPER 12 GB node.

Why sharding does not scale with users: llama.cpp's layer split executes the whole batch through stage 1, then stage 2. Devices take turns; nothing overlaps during decode. This is the inference engine's own scheduler, not something the hub can work around. Sharding is valuable for exactly one case, a model no single device can hold, and the wrong tool for serving many people, which wants replicas: whole-model instances, each a full-speed independent stream. Five machines that can each hold a model means five full-speed concurrent streams.

A live two-machine run confirms the shape at larger scale: Qwen3-14B Q4_K_M (8.9 GB) split [15 25] across the GTX 1060 and RTX 4070 SUPER loaded cold in 4 m 06 s at 8.4 tok/s; Qwen3-32B IQ3_XS (13.4 GB, fits on neither card alone) split [24 40] loaded in 4 m 57 s at 9.4 tok/s, with a measured network share of 7% once the initial weight push had finished.

Availability means the pool can host it right now

A model is available iff the current pool can place it. A node leaving that drops the pool below what a model needs makes that model unavailable, and /v1 says so explicitly (503 model_unavailable, naming the shortfall) rather than silently degrading to CPU.

Pausing is the machine owner's act, from the tray (or an opt-in --auto-pause GPU-busy detector, off by default because a desktop's own CUDA contexts made it flap). A paused node is treated as offline for placement, and the pool re-plans around it either way.