Nodes and the desktop app.

How a node attaches to a hub

A node does not join over the /v1 port. The hub binds a second listener for node traffic — RELAY_NODE_LISTEN, default :8443 — separate from RELAY_LISTEN (default :8383, which carries /v1). It is bound on every relay serve and there is no flag that turns it off.

That listener is TLS from the first byte, always, whatever RELAY_V1_TLS is set to. It presents the hub's own server certificate, sets ClientCAs to the hub CA with VerifyClientCertIfGiven, and floors the handshake at TLS 1.3 — nothing but relay itself ever dials it, so there is no legacy client to accommodate. Client authentication is if given rather than required because enrollment has no client certificate yet: enrollment is what issues one. Any certificate that is presented is verified against the hub CA before a handler sees it, and the tunnel handler rejects a connection that arrives without one.

Run relay token on the hub and it prints the exact line to run on the machine that will lend:

Run this on the machine that will lend VRAM (valid until 3:04PM):

  relay up --pool https://hub.internal:8443 --token 647ed1a6….a91f4c2b…

The token works once and pins this hub's TLS certificate — the connection
is encrypted and authenticated from the very first byte, no VPN required.
This is the same CA that signs the /v1 API when it is served over TLS —
re-show its fingerprint any time with `relay status`.
Set RELAY_ADVERTISE_URL on the hub to fill in the address automatically.
Trust: replicas (default) — this machine may run a whole-model replica and will see the plaintext prompt/completion for requests routed to it. Use --shard-only to restrict it to activations only.

The address in that line is derived from RELAY_ADVERTISE_URL's host with the node listener's port and the scheme forced to https — never the /v1 port. With RELAY_ADVERTISE_URL unset the hub has no address to hand out and the CLI substitutes https://<this-machine>:8443 for you to fill in.

What the token is, and what it pins

The join token is a compound string: <redeem-token>.<ca-fingerprint> — 32 bytes of crypto/rand hex-encoded, a dot, then the SHA-256 fingerprint of the hub's CA certificate. Token bytes are lowercase hex and never contain a dot, so the last dot separates the two halves unambiguously. This is the same compound-token bootstrap kubeadm uses for --discovery-token-ca-cert-hash: the operator copies one opaque string out of band, and everything after that is verified against it.

A token works once, and only for as long as RELAY_ENROLL_TTL (default 15m) allows. Only its SHA-256 hash is stored; redemption and node creation happen in one transaction, so two machines racing on one token cannot both win.

Enrollment is unauthenticated by construction — it is how a machine gets its credential — so it is rate limited instead: a burst of 10 per minute, keyed on the transport peer address rather than any forwarded header, since a header is attacker-controlled. Over the limit is a 429 with a fixed Retry-After: 60.

Four routes live on the node listener, and nothing else does:

The desktop app: relay-tray.exe and relay-tray

There are two ways to lend VRAM. relay up --pool URL --token TOKEN is the CLI path: a console program whose taskbar button (if any) belongs to the terminal that launched it, not to Relay. The desktop face (cmd/relay-tray) is the same lending agent with a notification-area icon instead — red while lending, grey while offline, click to toggle.

It is not Windows-only. It ships as relay-tray.exe for Windows amd64, built for the GUI subsystem so it opens no console window, and under the same binary name, relay-tray, for Linux amd64 and arm64, drawing over the session D-Bus with the StatusNotifierItem protocol every mainstream Linux desktop implements. Both are CGO-free.

There is no tray build for macOS, and none for Windows arm64 either — but for different reasons. macOS's notification area is Cocoa, which has no CGO-free binding, and building with CGO would cost the single-static-binary property for a cosmetic feature; lending itself still works there, with relay up. Windows arm64 is not a release platform for relay at all, tray included — no GPU-capable llama.cpp build is published upstream for it, so there is nothing for the agent to drive on that architecture.

# Windows
relay-tray.exe --pool https://hub.internal:8443 --token 647ed1a6….a91f4c2b…
# or, once already joined, with no flags at all — the normal way to launch it
# from a logon task or a double-click
relay-tray.exe

# Linux
relay-tray --pool https://hub.internal:8443 --token 647ed1a6….a91f4c2b…
# or, once already joined, with no flags at all
relay-tray

Both refuse to start rather than fail halfway: if this machine has no notification area the app exits saying to use relay up instead, and if it has never joined a pool and no --token was given it prints the two-step instruction to mint one. An icon that appears and vanishes reads as a crash; a message says what to do.

A GUI-subsystem process has no stderr, so both slog and the standard logger are redirected to relay-tray.log beside the credential file — on every platform, not just Windows. Capturing the standard logger matters more than it looks: the systray library reports its own failures through log.Printf, and without the redirect a failed registration produces no icon, no error, and no trace of either. That file is the only place to look if the icon never appears.

The VRAM-to-lend control

A notification-area menu has no slider; the platform APIs offer only items and checkboxes. The tray exposes a set of discrete steps instead — None, 25%, 50%, 75%, Everything — each labelled with the figure it works out to on this card, because they are derived from the card's actual total VRAM rather than fixed round numbers: "half your card" means something to an owner, "8 GB" only means something if you already know your card's size. Picking one sets the reserve, and the menu ticks whichever step matches the current reserve, so the choice survives a machine whose free VRAM moved since the menu was built.

Pause / resume

Pause lending and Resume lending are owner-initiated from the tray: the node keeps its tunnel but lends nothing until resumed, and the pool re-plans around it. When a node pauses, the hub drops its replica rows and — because the tunnel is still open on this path — asks the node to actually stop its llama-server (30s timeout) rather than only forgetting the bookkeeping row, so the card is genuinely free. Nothing pauses a node on its own by default.

Settings (name, reserve, rpc-port, cache directory and cap, sandbox mode, auto-pause, lend windows, and an owner-initiated pause flag) persist to settings.json beside the credential, so a later launch with no flags at all (a logon task, a tray double-click) reproduces the same configuration instead of reverting to defaults. Which fields count as "unset" is decided by the flag package, not by comparing against defaults — a flag left at its default is not the same as a flag deliberately given that value.

Single-instance lock

Both relay up and the desktop app acquire the same PID-file lock — agent.pid, beside the credential — before lending. Two lending agents on one machine share one credential, and the second's handshake knocks the first's session down; the first redials and returns the favour, which the owner sees as the icon flapping between online and offline. The lock makes that impossible rather than merely unlikely, and it treats a PID that is not a live relay process as stale, so a crash or a power cut never wedges a machine into "already running".

Process lifecycle: a node never leaks a GPU process

Every llama-server and rpc-server a node (or the hub) launches goes through internal/spawn, which gives two guarantees: the child opens no console window, and the child cannot outlive the process that started it.

As a second line of defense, each supervisor writes a PID file (0600) for its child and, on the next start, kills a stale PID only if the live process's image name is one it owns (llama-server, rpc-server, ggml-rpc-server, or bwrap — and bwrap, being a generic Linux sandboxing helper, must additionally name one of those binaries on its own command line before it counts as a match). A reused PID belonging to something else is left alone.

Tunnel loss. The agent reconnects with exponential backoff (2s doubling to 60s). A node replica keeps serving through a short outage, but if the tunnel has been down for 60 s (replicaDisconnectGrace) the agent stops its own llama-server unasked, once per outage, so a node cut off from its hub does not hold a card indefinitely. On the hub side the same disconnect drops the node's replica rows after RELAY_REAP_THRESHOLD. On reconnect, a node still reporting a replica the hub no longer expects is told to stop it.

doctor: diagnosing a machine that lends nothing

relay doctor [POOL_URL]

Every check exists because its failure is silent. In order: this build's own Windows icon resource; rpc-server presence; whether a different llama.cpp binary on PATH is being shadowed by the one beside the executable; GPU detection; which offload device llama-server actually reports; the process sandbox; the llama.cpp build number; the llama-sha256 binary check; tray support (and, on Windows, whether the icon is actually promoted to the visible tray or hidden in the overflow); whether this machine has joined a pool; how the node credential is protected at rest; and tls-cert, the state of this node's issued client certificate.

Given a pool URL it adds three more: hub (does the hub answer GET /healthz), tls-handshake — which dials the mutual-TLS node listener presenting this node's own client certificate and then hits /api/nodes/whoami, because a bare handshake succeeds right up until the tunnel rejects a revoked certificate — and v1-tls.

tls-cert is the check to watch before turning on RELAY_REQUIRE_MTLS. It skips with not joined yet, skips with still on the legacy transport and the exact --upgrade-tls command for a node that has no client certificate, fails on a corrupt certificate or on an expiry that has passed, and fails distinctly on a certificate that is not valid until a future date — which is almost always this machine's clock being behind rather than a real problem with the certificate.

Where the llama.cpp binaries must live

relay resolves llama-server and rpc-server (also accepted as ggml-rpc-server, which is what upstream renamed it to) the same way, in order: an explicit env override (RELAY_LLAMA_BIN, RELAY_RPC_SERVER_BIN), which must exist or resolution fails loudly; then the directory containing the running executable, resolved through symlinks, for each candidate name with the platform's .exe suffix; then PATH via exec.LookPath. This is the "unzip relay and llama.cpp into one folder" layout the archives are laid out for — the shared libraries have to sit next to llama-server for the dynamic linker to find them at all.

Autostart: systemd on Linux, a logon task on Windows, a LaunchDaemon on macOS

relay install-service prints a definition and never installs one — nothing on the machine changes until you pipe that output into tee, save it to a file, or hand it to launchctl yourself. That is what lets you read exactly what you are about to install first. Pass --bin when generating for a different machine than the one you are on — generating a Windows definition from Linux otherwise emits a Linux path that silently fails at logon, so cross-target output gets an obvious placeholder (C:\Program Files\Relay\relay.exe on Windows, /opt/homebrew/bin/relay on macOS) instead of a plausible-looking wrong answer.

On Linux, both roles are an ordinary systemd unit. The node unit reuses the stored credential, so no token appears in the unit file and nothing secret is written there; because a system service has no desktop session, it passes --tray=false — there is nowhere to draw the icon. Run relay up once to join before installing it. The hub unit needs no prior join step.

# Hub
relay install-service | sudo tee /etc/systemd/system/relay.service
sudo systemctl daemon-reload && sudo systemctl enable --now relay

# Node
relay install-service --node | sudo tee /etc/systemd/system/relay-node.service
sudo systemctl daemon-reload && sudo systemctl enable --now relay-node

On Windows, only the node role is available, and the generated definition is not a Windows Service: a service runs in session 0 with no desktop, so it could lend VRAM but could never show the tray icon. relay install-service --os windows --node instead prints a bash script — there is no PowerShell anywhere in relay's Windows tooling — that registers a scheduled task starting at logon, in the user's own session, via schtasks against a Task Scheduler XML definition. Run it from Git Bash. The generated task runs relay up.

relay install-service --os windows --node --bin "C:\Program Files\Relay\relay.exe" > install-relay-node.sh
bash install-relay-node.sh

On macOS, likewise only the node role is available, and it comes back as a launchd property list — specifically a LaunchDaemon, not a LaunchAgent: a LaunchAgent is tied to a login session the way a Windows logon task is, and the point of installing this at all is to survive a logout. A LaunchDaemon lives under /Library/LaunchDaemons, owned by root, so installing one needs sudo for the copy, the chown, and the launchctl load. KeepAlive is set with SuccessfulExit=false — launchd restarts the process if it fails, but leaves it stopped after a clean exit, so launchctl unload and relay down both mean what they say instead of being fought by an unconditional respawn. There is no tray icon on this path: macOS's notification area needs CGO, which the desktop app avoids everywhere (see above), so the plist runs relay up --tray=false headless — --tray=false is baked into the plist's arguments rather than left to be discovered missing. Run relay up once to join before installing it, since the plist runs on the stored credential and never carries a token.

relay install-service --os darwin --node > relay-node.plist
sudo cp relay-node.plist /Library/LaunchDaemons/com.opalengine.relay.node.plist
sudo chown root:wheel /Library/LaunchDaemons/com.opalengine.relay.node.plist
sudo launchctl load -w /Library/LaunchDaemons/com.opalengine.relay.node.plist