GPU Backends.

Release archives bundle a real llama.cpp build

The tar.gz/zip release archives — and the install.sh one-liner that fetches them (bash, including Git Bash on Windows) — put a real llama.cpp build beside relay, pinned to an exact upstream release and sha256-verified before it is ever packaged. Nothing to compile yourself for relay serve/relay up to work.

Backend per platform

A GPU is required — there is no CPU-only path

Every machine in a pool, hub or node, needs a GPU device llama.cpp itself reports. relay serve and relay up each probe for one at startup and refuse to run without it — they don't degrade to CPU, they exit:

relay needs a GPU on every machine: llama-server reports no GPU device — a
CPU-only llama.cpp build or a missing GPU driver (run `relay doctor`)

A unified-memory machine — a DGX Spark, or an Apple Silicon Mac — counts as a GPU here; the check is "does llama.cpp report a device," not "is it a discrete card."

Find out which one you actually have

relay doctor's llama-backend check runs the resolved llama-server binary's own --list-devices and reports exactly what it finds — e.g. Vulkan0: NVIDIA GeForce RTX 4090, or CPU only if no GPU backend loaded, either because the bundle for your platform doesn't have one or a GPU-capable bundle found no usable driver on this machine. relay doctor also reports whether the resolved binary matches this project's known-good hash table (llama-sha256) and the llama.cpp build it's validated against (llama-build).

relay doctor [POOL_URL]

Only the release archive carries the bundle

Package installs (.deb/.rpm/.apk) put relay at /usr/bin/relay only — no bundled llama.cpp, since /usr/bin is a shared system directory. Use the tar.gz/zip archive (or install.sh) if you want the bundle, or install llama.cpp from your own distro's packages and let relay find it on PATH.

Same story building from source — go install or a plain git clone + make build carries no bundle either. Package managers, go install, and go build do not carry it, so on those relay serve falls back to llama-server on PATH and warns if it can't find one.

The source and releases open soon. Until then there is nothing to install.

Bring your own build

Install or build llama.cpp yourself with the backend you want (-DGGML_CUDA=ON, -DGGML_HIP=ON, etc., plus -DGGML_RPC=ON) and point relay at it — or replace the bundled binaries beside relay; either works, and an env override always wins.

If you already had a llama-server/rpc-server on PATH before upgrading to a release that bundles its own, relay doctor's llama-shadow check (and a one-line startup log) tells you which one actually wins.

Bundle integrity

relay doctor --bundle       # one redacted .zip for a bug report — version, doctor
                            # report, pool status, config and agent log
relay doctor --pin-llama    # record the resolved rpc-server/llama-server binaries
                            # as trusted (TOFU) for future checks

"Redacted" means every machine name, node name, model name, API key, join token, CA fingerprint, email address, and any run of 32 or more hex characters is swapped for a placeholder before the archive is written. The same value always gets the same placeholder throughout the bundle, so a log line naming a node and the status row for that node still read as the same machine once the real name is gone — only the identity is stripped, not the shape of the problem. A loopback address is left as-is; it identifies nobody. A final pass over the finished bundle checks that none of those original values survived, and relay doctor --bundle refuses to write the file if any did.