Configuration.
Every setting has a working default and is environment-overridable. There is no config file: a single-binary daemon under systemd already has a natural place for environment (the unit's Environment= lines).
bash
# Every default works. This is a complete, running hub.
relay serve
# Overrides are environment variables — there is no config file to edit.
RELAY_LISTEN=127.0.0.1:8383 RELAY_CONTEXT_SIZE=16384 relay serveThree defaults are not what an older transcript would suggest
/v1 is authenticated by default — an unset RELAY_API_KEY makes relay serve generate one, not leave the endpoint open. /v1 is served over TLS by default on any bind that is not loopback-only, which includes the default :8383. And every key is rate limited to 300 requests a minute unless RELAY_V1_RATE_LIMIT says otherwise. Relay is alpha and unaccredited; these are the defaults in the code, not a compliance posture. Common
| Variable | Default | Purpose |
|---|---|---|
RELAY_LISTEN | :8383 | Public API bind address: /v1, the Ollama-compatible /api/* routes, /healthz, and — unless RELAY_REQUIRE_MTLS is set — the legacy plaintext node enroll and tunnel routes. The default's empty host means every interface, not loopback, which is what makes RELAY_V1_TLS=auto resolve to TLS. |
RELAY_STATE_DIR | $XDG_DATA_HOME/relay (~/.local/share/relay), or /var/lib/relay as root | SQLite file, model store, the generated API key, the audit log, and the hub's TLS material. |
RELAY_SOCKET | $XDG_RUNTIME_DIR/relay/relayd.sock, /run/relay/relayd.sock as root, else relayd.sock in the default state dir | CLI control socket (0600). The path is capped by the platform's unix-socket limit — 103 bytes on macOS, 107 everywhere else — and a longer one fails startup with an error naming the limit. |
RELAY_LLAMA_BIN | — (resolved) | llama.cpp server binary. Unset resolves beside the relay executable first, then PATH, and falls back to the bare name llama-server when neither has one. |
RELAY_ADVERTISE_URL | — (empty) | Public URL of the hub. Supplies the host of the join URL relay token prints (with RELAY_NODE_LISTEN's port, scheme forced to https) and an extra SAN on the hub's server certificate. Unset leaves the join URL blank. |
RELAY_API_KEY | — (empty, which is not open) | Bearer token required on /v1 and on the Ollama-compatible /api/* routes. Unset does not mean open: relay serve generates a 32-byte random key at first start, stores it at <RELAY_STATE_DIR>/api-key (mode 0600), and prints it exactly once. Set this to pin a key of your own — a key supplied by the environment is never persisted and never reprinted. |
RELAY_CONTEXT_SIZE | 16384 | Served context window, and the value /v1/models reports as context_length. At 0 the daemon omits llama-server's --ctx-size, so llama-server's own default applies — but /v1/models still reports 16384, the built-in default constant, so a client sizing itself from that field would be reading a number nothing negotiated. |
RELAY_HUB_NAME | this machine's hostname | The name this hub reports to nodes at enrollment. Read by the enroll handler, not by Load(). |
RELAY_DEBUG | — (empty) | Any non-empty value enables debug logging. Read by the CLI at startup, not by Load(). |
An unset RELAY_API_KEY is not an open endpoint
relay serve calls ensureAPIKey before it constructs the daemon: an empty key is generated, persisted at <RELAY_STATE_DIR>/api-key, and printed once — on that run only. Re-show or rotate it with relay token --api. A rotated key takes effect on the next relay serve, because the default key is read once at startup. See Security. The context default is a trade against concurrency, not against VRAM
KV cache costs
RELAY_CONTEXT_SIZE × RELAY_PARALLEL_SLOTS and nothing else, so the VRAM a replica reserves is fixed by their product: 8192×4, 16384×2 and 32768×1 all reserve exactly the same memory. The defaults spend that budget on context rather than width — 16k across two slots — because a coding agent accumulates a transcript and re-sends it, it does not issue four simultaneous requests. The slots are not useless: four concurrent clients did raise aggregate throughput on the reference pool (226 tok/s against 93). They are simply idle at one or two users, while the halved context is felt on every single turn. Raise the context further only with headroom, or quantize the KV cache (below) instead. TLS and the node transport
None of these existed before the security work, and every one of them has a default that is already in effect on a hub started with no environment at all.
| Variable | Default | Purpose |
|---|---|---|
RELAY_NODE_LISTEN | :8443 | A second listener, separate from RELAY_LISTEN and bound on every relay serve — there is no flag that turns it off. It is TLS from the first byte regardless of RELAY_V1_TLS, with a TLS 1.3 floor and the hub's own CA as its client-certificate authority. Node enrollment, the TLS upgrade path, the tunnel, and the whoami diagnostic all live here. |
RELAY_V1_TLS | auto | One of auto, on, off; anything else fails startup. auto serves /v1 over TLS unless RELAY_LISTEN is loopback-only — and the default :8383 has an empty host, which means every interface, so the default resolves to TLS. off is a migration escape hatch, not a normal setting. The /v1 listener's floor is TLS 1.2, one below the node listener's, because arbitrary OpenAI-compatible SDKs are the intended clients. |
RELAY_V1_RATE_LIMIT | 300 | Default requests per minute, per key, on /v1. 0 disables the limit entirely. It is keyed on the key, not the daemon: a named key's own --requests-per-minute overrides this for that key alone, and 0 there means inherit this value rather than unlimited. |
RELAY_TLS_DIR | <RELAY_STATE_DIR>/tls | Where the hub's CA and server certificate live — ca.pem, ca-key.pem, server.pem, server-key.pem, every one written 0600. Generated on first relay serve and reused byte-for-byte afterwards, so a fingerprint pinned yesterday is still correct today. |
RELAY_REQUIRE_MTLS | false | Unmounts the legacy plaintext /api/nodes/enroll and /api/nodes/tunnel on RELAY_LISTEN; both then answer 410 Gone with code mtls_required and a message pointing at RELAY_NODE_LISTEN. Off by default so a pool whose nodes have not migrated to client certificates is never bricked by a hub upgrade alone. Any non-empty value other than 0, false, no or off turns it on. |
RELAY_TLS_DOMAIN | — (empty) | A public hostname that already resolves to this hub. Set it and /v1 presents a publicly-trusted certificate obtained via ACME instead of the hub CA, so clients need no pin at all. It requires RELAY_V1_TLS to be auto or on, and a non-loopback RELAY_LISTEN — a certificate authority's validator dials in from outside. Either violation fails startup. The node listener stays on the hub's private CA regardless. |
RELAY_ACME_LISTEN | :80 | Plain-HTTP bind address for the ACME HTTP-01 challenge, used only when RELAY_TLS_DOMAIN is set. Setting it to the empty string disables that listener and leaves TLS-ALPN-01, which only works when RELAY_LISTEN is on port 443. This is the one variable where explicitly empty and unset are different choices. |
RELAY_ACME_EMAIL | — (empty) | Optional contact address the certificate authority may use to warn about a certificate nearing expiry. Never required. |
RELAY_CA_FINGERPRINT | — (empty) | Client-side, not daemon-side: the hub CA fingerprint the relay CLI pins for an https:// pool URL. Read by the CLI, not by Load(). Resolution order is the --ca-fingerprint flag, then this, then the local daemon's own status, then this machine's node credential matched by hostname, then the system trust store. |
A default hub speaks https, and a client has to trust the CA
On the default
RELAY_LISTEN=:8383, RELAY_V1_TLS=auto resolves to TLS against the hub's own CA — so a client configured for http:// does not connect. The trust anchor is the CA's SHA-256 fingerprint, not a certificate file path; relay serve prints it once and relay status re-prints it. It is not secret. RELAY_TLS_DOMAIN is the alternative that needs no pin, and RELAY_V1_TLS=off is the migration escape hatch. Setting RELAY_API_KEY with a network-reachable RELAY_LISTEN and RELAY_V1_TLS=off is refused outright — the bearer token would cross the network in cleartext. Memory and placement
| Variable | Default | Purpose |
|---|---|---|
RELAY_VRAM_RESERVE_MB | 4096 | VRAM kept for the machine's owner; a node lends (total − reserve). Must be ≥ 0. |
RELAY_PARALLEL_SLOTS | 2 | Continuous-batching width (--parallel); must be ≥ 1. Multiplies the KV reservation. A ceiling, not a fixed width: a node that cannot hold this many slots' worth of KV is admitted at a narrower one rather than being pushed onto the much slower sharded path. |
RELAY_PIPELINE_DEPTH | 4 | Max stages a shard may spread across; must be ≥ 1. |
RELAY_REPLICA_CTX | 0 (use the served context) | Context the replica fit-check sizes its KV term on. 0 sizes on RELAY_CONTEXT_SIZE × RELAY_PARALLEL_SLOTS; a positive value is clamped to RELAY_CONTEXT_SIZE, so it can only narrow the check, never widen it. Lower reserves less and places more replicas, at a real OOM risk if every slot later saturates. |
RELAY_KV_CACHE_TYPE | f16 | K/V cache precision for every llama-server the daemon launches (host replicas, node replicas, shards): f16, q8_0, or q4_0. Anything else fails startup. See Concepts. |
RELAY_FLASH_ATTN | — (empty) | Overrides llama-server's --flash-attn: on, off, or auto. Empty is also accepted and means no explicit choice — llama-server's own default applies at f16, and on is forced when the KV cache is quantized. |
RELAY_FLASH_ATTN=off with a quantized KV cache is allowed, and logged loudly
llama.cpp needs flash attention to run a quantized V cache at all. The combination is not rejected at startup; it is logged as a warning, and llama-server is what fails.
Sandbox, budgets, and LAN discovery
| Variable | Default | Purpose |
|---|---|---|
RELAY_SANDBOX | auto | How strictly the llama.cpp helper processes relay spawns — llama-server and rpc-server — are sandboxed: auto (sandbox when a mechanism is available on this machine, else run unsandboxed with a warning), require (fail the launch rather than degrade), or off. An empty value means not configured and resolves to auto; anything else fails startup. A node persists its own choice via relay up --sandbox and falls back to this when neither a flag nor a stored setting is present. |
RELAY_CREDENTIAL_PLAINTEXT | — (unset) | Set to exactly 1 to force the plaintext sealer for a node's credential file on every platform, skipping DPAPI on Windows and Keychain on macOS — for a headless service whose secret store is locked, or CI with no store at all. It changes only how new credentials are sealed; an already-sealed file is always opened with the sealer its own envelope names. |
RELAY_CACHE_MAX_GB | 0 (no cap) | A node operator's self-reported ceiling on the local file cache, in GB. Visibility only — crossing it warns and shows a figure in relay status, and never evicts or deletes. relay cache prune is the explicit action. Must be ≥ 0. |
RELAY_MODELS_MAX_GB | 0 (no cap) | The hub's equivalent for RELAY_MODELS_DIR. Same posture: warn, never delete; relay model rm is the explicit action. Must be ≥ 0. |
RELAY_ADVERTISE_LAN | false | Answer mDNS so relay up --token TOKEN can find this hub without --pool. Off by default, because announcing that a GPU pool exists is a disclosure of its own on a shared network. The record it publishes carries only a name, address, port and protocol version — never a token, model name, node identity or VRAM figure — and it advertises the node listener's actual bound port. |
Timing and paths
| Variable | Default | Purpose |
|---|---|---|
RELAY_LLAMA_ADDR | 127.0.0.1:8384 | Bind address of the supervised llama-server. Loopback only; reached through the proxy. |
RELAY_RPC_SERVER_BIN | — (resolved) | rpc-server on a node. Unset: rpc-server / ggml-rpc-server beside the relay executable, then PATH. |
RELAY_RPC_LISTEN | 127.0.0.1:0 | Loopback address llama-server dials per node backend. Must be host:port, must not be every-interface, must be loopback; anything else is rejected at startup. |
RELAY_HEARTBEAT | 15s | How often a node reports capability over the tunnel. |
RELAY_REAP_THRESHOLD | 90s | How long a node may go silent before the hub flips it offline and re-plans. |
RELAY_ENROLL_TTL | 15m | How long a minted join token stays redeemable. |
RELAY_LOAD_TIMEOUT | 15m | Max time a single model load may take before it is abandoned. |
RELAY_DB | <RELAY_STATE_DIR>/relay.db | Embedded SQLite file. |
RELAY_MODELS_DIR | <RELAY_STATE_DIR>/models | Directory where GGUF weights are stored. |
Speculative decoding
| Variable | Default | Purpose |
|---|---|---|
RELAY_DRAFT_MODEL | — (empty) | Path to a small, same-tokenizer GGUF; empty disables speculative decoding. Host replicas only. |
RELAY_DRAFT_MAX | 16 | Max tokens the draft model may speculate ahead (--spec-draft-n-max). |
RELAY_DRAFT_MIN | 0 (flag omitted) | Min draft tokens before verification (--spec-draft-n-min). |
RELAY_DRAFT_GPU_LAYERS | 99 | Draft model layers offloaded to GPU (--spec-draft-ngl). |
See Speculative Decoding for the behavior behind the four RELAY_DRAFT_* variables.
GPU detection overrides
Relay probes the GPU through nvidia-smi, AMD's sysfs nodes, or Metal on Apple silicon. These override that probe rather than repairing it, and they are believed exactly as given: the free-VRAM override is returned without the clamp a probed figure passes through, so an overstated one produces an out-of-memory failure at load time instead of a refusal at placement time.
| Variable | Default | Purpose |
|---|---|---|
RELAY_VRAM_TOTAL_MB | probed | VRAM this machine reports as installed, in MB. On the hub it short-circuits the whole probe: a positive value is taken as both total and free, with RELAY_GPU_NAME as the name and a GPU count of 1. |
RELAY_VRAM_FREE_MB | probed | VRAM this machine reports as free, in MB. Returned as given, skipping the clamp to what the node is actually lending. |
RELAY_GPU_NAME | probed, else unknown | Display name reported for this machine's GPU. |
RELAY_GPU_COUNT | probed, else 1 | Number of GPUs on this machine. A value that does not parse as an integer is ignored. |
RELAY_GPU_BACKEND | derived from the probe | Compute backend reported for this machine, lowercased as given. cuda, metal and unknown are the values the rest of the code recognises. |
RELAY_GPU_DRIVER | — (empty) | Free-text driver string reported alongside the GPU. |
A misconfigured RPC listen address fails startup, not silently
RELAY_RPC_LISTEN must resolve to a loopback address; binding it to a network interface would hand the machine's GPU to anyone who can reach it, since llama.cpp's RPC wire has no authentication of its own. relay serve refuses to start rather than warn. A stale draft model degrades, it doesn't crash the daemon
A configured
RELAY_DRAFT_MODEL that doesn't resolve to a file (missing, or a directory) is logged as a warning and cleared at startup. Speculative decoding turns off; the pool still serves its main model. A typo in a numeric or duration value is not an error
Integer and duration variables that fail to parse fall back to their default silently, and an empty string is treated as unset for every variable except
RELAY_ACME_LISTEN. Check relay status (or relay serve's startup log) if a value seems ignored.