CLI.

relay — pool GPU memory across machines and serve it as one OpenAI-compatible endpoint

  relay serve                        run the pool daemon (the hub)
  relay status                       show pool capacity, node health, and layer placement
  relay bench [--model NAME]         measure tokens/sec and where the layers ran
  relay chat [--model NAME]          streaming terminal chat REPL against the pool
  relay doctor [POOL_URL]            diagnose why this machine is not working
  relay doctor --bundle              write one redacted .zip for a bug report:
                                     version, the doctor report, pool status,
                                     config and the agent log, with machine
                                     names, addresses and keys replaced
  relay token [--label NAME] [--shard-only]
                                     mint a single-use join token; --shard-only
                                     restricts the joining node to pipeline
                                     shards, never a whole-model replica
  relay up --pool URL --token TOKEN  join a pool and start lending VRAM
  relay up --token TOKEN             (no --pool) discover a hub advertising on
                                     the LAN and join it — see RELAY_ADVERTISE_LAN
  relay down                         stop lending and forget the pool

  relay node ls                      list enrolled nodes and their trust level
  relay node set ID --trust T        change a node's trust (replicas|shard-only);
                                     downgrading to shard-only stops its
                                     running replica immediately
  relay node rm ID [--wipe]          remove an enrolled node; --wipe also
                                     orders it to erase its local file cache

  relay model add NAME PATH          register a GGUF the hub owns
  relay model catalog                a short starter list, each entry marked
                                     with where it would run in THIS pool
  relay model search QUERY           find GGUF files on Hugging Face, with size
                                     and the same fit verdict, before downloading
  relay model pull REPO[:FILE]       download a GGUF from Hugging Face and register it
  relay model ls                     list registered models and host-only status
  relay model set NAME --host-only[=false]
                                     keep a model on the hub only — no node
                                     replica, no shard — even if that makes it
                                     unavailable when the host is full
  relay model rm NAME                unregister a model

  relay token --api [--rotate]       show (or rotate) the default /v1 API key
  relay key ls                       list named API keys and their quotas
  relay key add NAME [flags]         create a named, quota-bearing API key
  relay key rm ID                    remove a named API key
  relay audit tail [-n N]            show the most recent /v1 requests

  relay cache prune [--dir PATH] [--max-gb N]
                                     (on a lending node) free disk by removing
                                     unpinned cached files right now, down to
                                     N GB if given, or all of them if not

  relay install-service [--node]     print a systemd unit
  relay uninstall [--yes] [--keep-models]
                                     remove relay's state, credential and login
                                     item from this machine, and print what
                                     needs root (units) or cannot be self-deleted
                                     (the binary)
  relay version                      print version and protocol

Environment: RELAY_LISTEN, RELAY_STATE_DIR, RELAY_LLAMA_BIN, RELAY_VRAM_RESERVE_MB,
             RELAY_PARALLEL_SLOTS, RELAY_PIPELINE_DEPTH, RELAY_API_KEY, RELAY_DEBUG,
             RELAY_ADVERTISE_LAN (hub, off by default — see README's Security posture)
             RELAY_CACHE_MAX_GB (node), RELAY_MODELS_MAX_GB (hub) — visibility-only
             disk budgets: warn in relay status and after a push/pull, never delete

The usage footer lists only the most common environment variables. The full set is on Configuration.

Serving and using the pool

relay serve

Runs the hub. --listen overrides RELAY_LISTEN for this process. On a fresh state dir it also generates the default /v1 API key and prints it exactly once — see Security.

That same first run — keyed to the API key having just been generated, not to whether the state dir already existed — also prints a Next: block: find a model that fits this pool (relay model catalog), lend another machine’s GPU (relay token), then a curl line to send the pool its first request. A later restart of an already-initialized hub does not repeat it — relay status shows the same three things again at any time.

relay status

Field reference for the JSON document: API.

The plain-text output carries two things worth knowing about before you reach for --json. The per-node table has a DISK column alongside GPU, LENDING, FREE, TRUST, LINK, STATE and LAST SEEN. And on an https pool a tls line appears under pool: with a Let's Encrypt domain it names the domain and says no pinning is needed, otherwise it prints the CA fingerprint to pin in every client that talks to /v1. That fingerprint is the value relay chat --ca-fingerprint and relay up --ca-fingerprint expect — see Security.

Below that, a WORKLOAD table lists every live whole-model replica — where it is running, its state, requests in flight, when it last served one, and how long it has been resident — busiest first. A sharded model does not get a row here: it already appears under serving with its layer split, and listing it twice would read as two workloads instead of one. And when something worth explaining has happened recently — a placement refused, a fallback to sharding, an eviction — a short recent placement decisions block follows, newest first: the same detail string the API error and the hub’s own log carry, so there is one wording to remember rather than three. Successful placements are not in it; the ring is for what needed explaining, and would be pushed out in seconds by ordinary traffic otherwise.

relay chat

A streaming terminal REPL against the pool’s /v1/chat/completions, with an agentic tool loop scoped to the current repository. Pure Go, no CGO.

With no models registered on the pool it exits rather than opening an empty session. RELAY_API_KEY is read from the environment for the bearer token.

The agent’s tools are read_file, write_file, list_dir, search, and run_command. run_command asks for confirmation before every call unless /run on turns that off — treat that switch the way you would any other “run whatever it suggests” mode.

relay bench

Joining a pool and lending VRAM

relay up

The first run enrolls and stores a credential at the default credential path; a later relay up with no flags reuses it and re-applies the previously stored settings (name, reserve, rpc-port, auto-pause, sandbox mode) rather than reverting to defaults.

--sandbox takes one of three values, validated before anything starts: auto sandboxes when a mechanism is available on this machine and falls back to running unsandboxed when none is; require fails the launch outright rather than running unsandboxed; off disables sandboxing entirely. Empty means auto.

relay down

Stops the running agent process (found via a PID file beside the credential), not only the credential. An earlier version left the process serving layers with no way to reclaim its identity, which was the worst of both outcomes.

relay token

Run on the hub; hand the line it prints to whoever owns the GPU.

The token works once and pins the hub’s TLS certificate. What each trust level can actually see is on Security; relay token --api is a different thing entirely and is covered under API keys below.

Models

relay model ls

One row per registered model: name, VRAM, layers, host-only, pinned, path.

relay model add NAME PATH

Takes no flags in the common case; GGUF header parsing fills in everything. Overrides exist for the rare header that doesn’t say enough:

relay model catalog

Prints a short, curated starter list — seven entries, from a 398 MB smoke-test model up to an 18.6 GB mixture-of-experts model — spanning the sizes an ordinary desktop pool actually has. Each row marks where that model would run in this pool right now: the host, a specific node, shards (slow) if it only fits split across the pool, or that it does not fit at all. With no hub reachable it still prints the list, just without a verdict, since reading this before relay serve has ever run is a normal way to arrive at it. The fit check adds this pool’s KV cache reservation on top of the GGUF’s own size — the same comparison placement itself makes — but the authoritative number comes from the file’s metadata once it is actually downloaded.

relay model search QUERY

Searches Hugging Face for repositories carrying GGUF files and, per repository, lists each file’s size and the same fit verdict catalog prints — before anything downloads. By default only files that fit this pool are shown, with the rest counted rather than silently dropped; a split GGUF set (numbered parts) is folded into one entry carrying the total size, since sizing one part on its own reads as far smaller than the model actually is. HF_TOKEN is forwarded for gated repos.

relay model pull REPO[:FILE]

With no filename, a default quantization is picked from the repo’s file listing (Q4_K_M by preference, else the smallest). The pick is printed along with the REPO:FILENAME syntax for choosing a different one, and the download then starts — there is no confirmation prompt. HF_TOKEN is forwarded for gated repos.

relay model set NAME

At least one of the two flags must be given explicitly, or the command prints its usage and stops. Both are booleans you reverse with =false.

relay model rm NAME

Unregisters the model. The GGUF file itself is not deleted.

Nodes

relay node ls

One row per enrolled node: id, name, GPU, lendable VRAM, trust, link latency, state, and last-seen time.

relay node set ID --trust replicas|shard-only

--trust is required and must be one of those two values. Downgrading a node to shard-only stops any replica it was running right then, rather than waiting for the scheduler to reclaim it. What each level lets a node see, and how the hub enforces it, is on Security.

relay node rm ID

Without --wipe, removal revokes the credential and leaves the cache on disk.

API keys and the audit log

relay token --api

If RELAY_API_KEY is set in the environment it overrides the stored key, and --rotate reports that it had no effect rather than silently rotating a key nothing is using.

relay key ls

Named keys and their quotas: id, name, tokens/day, max concurrency, requests/min, and tokens used today. A key with no rate limit of its own shows default — it inherits RELAY_V1_RATE_LIMIT rather than being unlimited.

relay key add NAME

The secret is printed once, at creation, and never again.

relay key rm ID

Removes a named key; it stops authenticating against /v1 immediately.

relay audit tail

One row per /v1 request: time, key id, model, placement, prompt and completion token counts, latency, and status. Prompt and completion text is never recorded.

Diagnostics

relay doctor [POOL_URL]

An optional positional URL is what the three hub-side checks run against: hub (GET /healthz), tls-handshake (the mutual-TLS handshake) and v1-tls. Omit it and all three report as skipped — the middle one under the shorter label tls, which is the only place that spelling appears; once it actually runs it reports as tls-handshake throughout. Every check reports on stdout, including whether a sandbox mechanism is actually available on this machine — which is reported as skipped when --sandbox off or RELAY_SANDBOX=off is in effect. See Troubleshooting for what each check reports.

--bundle re-runs the same checks with their output captured rather than printed, and writes it into a .zip alongside the build version, relay status, the config, and a tail of the agent log — one file to attach to a bug report instead of several pasted terminals. It is redacted before the archive is written: hostnames, paths, the API key, node names, and the TLS fingerprint are all replaced with stable placeholders, and a verification pass has to confirm none of them survived before anything is written to disk.

relay version

Prints one line: version, commit, and protocol version. --version and -v are accepted as aliases.

relay help

Prints the command list above and exits 0. --help and -h are accepted as aliases, and relay with no arguments at all does the same thing. An unknown command also prints it, but exits non-zero with unknown command.

Housekeeping

relay cache prune

Run on a lending node, not the hub. It frees disk now by removing unpinned cached files, and prints how much it freed.

Safe to run while the node is lending: both this command and the live agent’s LRU eviction only ever remove files, and a file removed out from under a running push is simply re-verified and re-written.

relay install-service [--node]

Prints the unit/task definition rather than writing it, so it’s safe to run anywhere and readable before install. On Windows, --node generates a logon-time scheduled task rather than a service: a Windows Service runs in session 0 with no desktop, so it can lend VRAM but never show the tray icon. --boot only changes the hub’s Windows task, trading that same visible-session convenience for surviving past a logout. See Nodes & Windows.

relay uninstall [--yes] [--keep-models]

Removes what relay put on this machine for the current user: the node credential, settings and model cache; the hub’s state directory (database, TLS material, API key, and downloaded models unless --keep-models is given); and, if one was registered, the “start on login” item. Nothing absent from this machine is listed — a hub-only install has no node credential, and claiming one was removed would misdescribe what just happened to the disk. It stops the running node agent first if one is up, prints exactly what it removed and what failed, and prompts with This cannot be undone. unless --yes was given.

Two things it deliberately never does, both printed as commands to run yourself: it never touches a systemd unit under /etc/systemd/system, since that belongs to whoever installed it and a tool that quietly sudoes is worse than one that prints the command; and it never deletes the running binary — impossible on Windows, and on Unix it would take this command’s own output down with it if anything failed halfway through.