Quickstart.

What you need before starting

A hub and a node are separate roles that normally run on separate machines: one owns the models and answers requests, the others lend GPU memory to it. A single machine can do both at once — the hub also has a card, so it can serve on its own before anything joins it.

• A GPU on every machine — hub and every node. A release archive bundles a working llama.cpp backend for every platform except Windows arm64, so there is nothing to build to get GPU offload on a supported card — which backend that is, and how to check what a given machine actually detected, is on GPU Backends. There is no CPU fallback: relay serve and relay up both refuse to start on a machine where llama.cpp reports no GPU device, whether that means a CPU-only build or a driver it can't see. A unified-memory machine such as a DGX Spark or Apple Silicon Mac counts as a GPU.
• Disk for model weights. A GGUF file runs from several hundred megabytes to tens of gigabytes depending on the model; relay model catalog (step 5 below) lists concrete sizes before anything downloads.
• Only linux/amd64 is run in this project's own CI before a release ships. The other four platform archives are cross-compiled and checked statically — right shape, right architecture, expected llama.cpp hashes — but nothing starts them. windows/amd64 has been run by hand against a real pool; macOS has not been run by anyone yet. If relay fails to start on your platform, that is a bug worth reporting, not a sign you configured something wrong.

1. Install

The one-liners resolve a release, download the archive for your platform, and verify its sha256 against that release's published checksums file before extracting or installing anything — on a mismatch the script exits non-zero and installs nothing. That is what makes piping either of these into a shell defensible: the verification happens inside the script itself, not by trusting the transport it downloaded over.

The source and releases open soon. Until then there is nothing to install.

There is no PowerShell installer — on Windows, run the same line in Git Bash, which comes with Git for Windows.

The script accepts a version pin and a channel. --version installs one exact tag and always wins when given. --channel picks which track "latest" means and defaults to auto: try the newest stable release first and — since no stable release exists yet on this project's alpha/beta/rc ladder — fall back to the newest release of any kind. prerelease (alias rc) does that newest-release resolution explicitly, and stable errors clearly instead of falling back.

The source and releases open soon. Until then there is nothing to install.

The script installs to /usr/local/bin if that is writable (or you are root) and ~/.local/bin otherwise, and tells you if the directory it chose isn't onPATH yet.

Direct download

Grab the archive for your platform from the release's assets page instead, if you would rather verify by hand or skip the script entirely:

Each archive contains relay (or relay.exe) plus llama.cpp's llama-server/ggml-rpc-server and the shared libraries they need, laid out beside relay — nothing else has to be installed separately for GPU offload to work.

Verify the checksum yourself before extracting:

sha256sum --ignore-missing -c relay_<version>_checksums.txt

Every release also uploads a versionless alias of each archive (relay_linux_amd64.tar.gz, relay_checksums.txt, and so on), for a link that always means "whatever the current release is":

The source and releases open soon. Until then there is nothing to install.

/latest/ resolves to the newest non-prerelease release only. Until this project ships its first stable 0.1.0, that URL resolves to nothing — link an alpha/beta/rc's alias by its exact tag instead: /releases/download/<tag>/relay_linux_amd64.tar.gz.

Linux packages

.deb, .rpm, and .apk packages are published for amd64 and arm64, installing relay to /usr/bin/relay with no post-install scripts — nothing starts or enables automatically; run relay install-service yourself once you've configured relay serve or relay up. These packages carry no bundled llama.cpp, since /usr/bin is a shared system directory, not somewhere to unpack an unrelated project's shared libraries. Use the tar.gz archive or install.sh instead if you want the bundle, or install llama.cpp from your own distro's packages and let relay find it on PATH.

sudo dpkg -i relay_<version>_linux_amd64.deb      # Debian/Ubuntu
sudo rpm -i relay_<version>_linux_amd64.rpm       # Fedora/RHEL
sudo apk add --allow-untrusted relay_<version>_linux_amd64.apk   # Alpine

Signature and provenance verification

Beyond the sha256 check, every release's checksum file is signed keylessly with Sigstore cosign and every archive carries a GitHub build-provenance attestation — see Security for the exact commands and what they do and do not cover.

From source

No cgo anywhere in this repo, so a plain Go toolchain is enough:

The source and releases open soon. Until then there is nothing to install.

Neither path bundles llama.cpp — see GPU Backends for what that means for relay serve.

2. Verify the install

relay version

Prints one line: version, commit, and the protocol version a node and hub have to agree on.

If anything about the install seems off — a GPU that isn't detected, a missing llama-server, a sandbox that won't engage — run relay doctor. It reports what this machine actually has and starts nothing itself; it exists to tell you why a pool would not come up before you try to bring one up.

relay doctor

3. Configure and start the hub

Every setting has a working default, so this is a complete hub with no environment set at all:

relay serve

The first relay serve on a machine generates a default API key, persists it under the state directory at mode 0600, prints it exactly once, and requires it as a bearer token on /v1. Copy it now — relay token --api re-shows it later, and relay token --api --rotate replaces it.

Generated a new default API key (shown once — copy it now):

  4f8a1c…

Use it as a bearer token: Authorization: Bearer 4f8a1c…
Re-show or rotate it later with `relay token --api`.

/v1 is served over TLS. Clients must trust or pin the hub CA fingerprint:

  9f2c7b…

Re-show this later with `relay status`.

Next:

  1. Find a model that fits this pool
       relay model catalog

  2. Lend another machine's GPU to it (run the printed line on that machine)
       relay token

  3. Send it a request
       curl https://localhost:8383/v1/models \
         -H "Authorization: Bearer 4f8a1c…"

`relay status` shows all of this again at any time.

That first-run block only prints once, keyed to the API key having just been generated — a later restart of an already-initialized hub does not repeat it, but relay status shows the same three things again at any time.

A handful of settings matter on a first run; the full table, including TLS, node transport and placement tuning, is on Configuration.

4. Register a model

A fresh machine has no GGUF file yet. Three ways to get one registered, none needing flags in the common case — relay reads layer count, context length, and VRAM needs straight from the GGUF header:

# a short curated starter list, each entry marked with where it would
# run in this pool right now
relay model catalog

# search Hugging Face for GGUF files, with size and the same fit verdict
relay model search qwen3

# download one straight from Hugging Face and register it (defaults to Q4_K_M)
relay model pull Qwen/Qwen3-8B-GGUF

# or register a GGUF you already have on disk
relay model add qwen3-8b /models/qwen3-8b-q4.gguf

5. Send the first request

The pool speaks the OpenAI API. A client needs the bearer token from step 3, and — because /v1 is signed by the hub's own CA rather than a publicly-trusted one by default — either that CA certificate or the fingerprint printed in step 3. --cacert is curl's flag for it; an SDK takes a CA-bundle setting.

curl --cacert <state dir>/tls/ca.pem https://localhost:8383/v1/chat/completions \
  -H "Authorization: Bearer $RELAY_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"hello"}]}'

<state dir> is the RELAY_STATE_DIR from step 3. Full client setup — Python/SDK examples, the name-matching rule for the hub's certificate, and how to avoid the CA step with a publicly-trusted certificate instead — is on Chat & Clients.

6. Add a node to the pool

On the hub, mint a single-use join token:

$ relay token --label "workstation-2"
Run this on the machine that will lend VRAM (valid until 3:47PM):

  relay up --pool https://hub.internal:8443 --token 647ed1a6….9f2c…

The token works once and pins this hub's TLS certificate — the connection
is encrypted and authenticated from the very first byte, no VPN required.
This is the same CA that signs the /v1 API when it is served over TLS —
re-show its fingerprint any time with `relay status`.
Set RELAY_ADVERTISE_URL on the hub to fill in the address automatically.
Trust: replicas (default) — this machine may run a whole-model replica and will see the plaintext prompt/completion for requests routed to it. Use --shard-only to restrict it to activations only.

Run that printed line on the machine that will lend its GPU. It enrolls once, stores a credential, and starts lending — a later relay up with no flags reuses it. The join token itself works exactly once. Enrollment, what the token actually pins, and the trust levels a node can hold are covered in full on Nodes & Windows and Security.

relay up --pool https://hub.internal:8443 --token 647ed1a6….9f2c…

7. Keep it running

Both commands above run in the foreground. For a hub or a node that should survive a logout or a reboot, relay install-service prints a systemd unit (or, on Windows, a scheduled task) to install yourself — it prints the definition rather than writing it, so it's safe to run anywhere and readable before you install it. Windows service behavior, the desktop tray app, and the VRAM-to-lend control are on Nodes & Windows.

relay install-service          # hub
relay install-service --node   # node