Known Issues.
Behaviours that are real, understood, and not bugs to report — what each one costs, and whether anything is planned.
0.1.0-alpha.N → -beta.N → -rc.N → 0.1.0 ladder. The wire protocol between hub and node, the on-disk state layout and the CLI surface can all change without a deprecation period; /v1 tracks the OpenAI shape and is the most stable thing here. Nothing is accredited or covered by a compatibility promise.Third-party mDNS tools cannot see a relay hub
avahi-browse, dns-sd and Bonjour will not list a hub even with RELAY_ADVERTISE_LAN=1 set and the hub genuinely advertising.
relay’s responder answers a query unicast, directly to the asker, rather than re-multicasting. RFC 6762 clients discard unicast replies to a standard multicast query, so they structurally cannot see it. relay up finds the hub normally, because it is the client this protocol was written for.
The trade was deliberate: a unicast responder is fully testable without a real socket. It is revisitable if third-party discovery ever matters.
Sharding is slower than one card that fits the model
A pipeline stage computes while the others wait, so a sharded model does not get faster with more machines, and a single sharded stream is measurably slower than the same model on hardware that could hold it outright — 30.3 → 22.5 tok/s (−26%) on the pair measured in Concepts.
Sharding exists to run a model nothing in the pool could hold otherwise. Replicas are the common case and scale linearly. Prompt processing is the exception: it batches and pipelines well, landing mostly on whatever device is fastest.
Only linux/amd64 is executed before a release
Relay ships five targets — linux and macOS on amd64 and arm64, plus windows/amd64; windows/arm64 is dropped because no GPU-capable llama.cpp build exists for it. One of them is executed before release. CI runs on self-hosted runners only, and those are Linux; the other four are cross-compiled and checked statically — they exist, file(1) reports the expected OS and architecture, they are not truncated, and their bundled llama.cpp hashes match internal/llama/pins.json.
Static checks establish that a binary is the right shape. They do not establish that it starts. So “does relay.exe run on Windows” is not proven by CI, and neither is any macOS or arm64 build.
This is a deliberate position rather than an oversight: verifying those platforms means owning runners for them, and the project would rather state the limit than imply a coverage it does not have.
What is known from testing outside CI:
| Target | Status |
|---|---|
| linux/amd64 | Exercised on every change: built, run, and relay doctor asserted on |
| windows/amd64 | Run by hand against a real two-machine pool (2026-09-10): enrolled over mTLS, sandboxed its child process, served both a replica and a shard. A one-off CI run the same day found two defects — a socket-path limit that used macOS's ceiling everywhere, and an API-key file mode Windows does not implement — both since fixed |
| linux/arm64 | Built and run once, successfully, in that same one-off run |
| darwin/amd64, darwin/arm64 | Never executed by anyone |
If you hit a startup failure on an unverified target, that is worth an issue — it is precisely the class of thing nothing upstream of you would have caught.
GPU-contention routing needs headroom to help
A node reports its GPU busy only when a foreign process holds more than busyIgnoreMB (256 MB) — below that is noise, not somebody’s game.
On a card with little spare VRAM the feature cannot help: if a model needs 5.6 GB of a 6 GB card, there is no room for a foreign consumer to clear the floor and still fit the model. Contention routing pays off when the model is comfortably smaller than the GPU.
Shard placements do not count toward a node’s workload
Request-level routing tracks in-flight work per node, but only for replicas. A Target carries no list of the nodes a shard spans, and sharding is the last-resort fallback used only after replica ranking has already failed.
A node busy serving a shard therefore looks idle to replica ranking. In practice a pool that is sharding has already run out of room for replicas.
Throughput weighting needs one replica run first
Shard layers are weighted by measured per-device throughput, learned from real replica runs — llama.cpp’s own server-side figure, normalised by model weight so readings from different models stay comparable.
A node that has only ever served shards has no reading. Those nodes keep a neutral weight and the split falls back to VRAM-proportional, exactly as before the feature existed. Nothing regresses; the pool simply gets more accurate with use.
Size caps are whole gigabytes, and warn only
RELAY_CACHE_MAX_GB and RELAY_MODELS_MAX_GB are integers. There is no way to express 500 MB.
Crossing one never deletes anything. A node lends disk on hardware its operator does not own, so eviction is not relay’s decision to make silently — the cap makes the situation visible and relay cache prune stays the explicit action. The honest cost: a warning nobody reads is the same as no cap at all, which is why relay status reports usage against cap rather than relying on a log line.
Cache.capBytes — a physical disk-safety ceiling derived from free space — does evict. It is a different mechanism with a different job. Pre-1.0
Releases are 0.1.0-alpha.*. The wire protocol between hub and node, the on-disk state layout, and the CLI surface can all change without a deprecation period. /v1 tracks the OpenAI shape and is the most stable thing here.