Troubleshooting.

Run the doctor first

relay doctor [POOL_URL]

Every check exists because its failure is otherwise silent: a missing rpc-server makes relay up exit before anything appears anywhere, an undetected GPU makes a node lend nothing, and a stale Windows icon cache hides an icon that is genuinely present in the binary. None of those announce themselves.

The checks run in this order, each printing [ ok ], [FAIL] or [ -- ] (skipped) with a one-line detail. The last three need a pool URL; omit it and all three skip.

Logs

Everything goes to stderr through slog. There is no log file — in the foreground it is on your terminal; as a service, journalctl is the log.

journalctl --user -u relay -n 200 -f          # hub, as a service
journalctl --user -u relay-node -n 200 -f     # node

Prompt and completion text is never logged, so output is safe to share — check it for hostnames and model names you would rather not, first.

model_unavailable (503)

This is intended behavior, not a bug: no silent fallback to CPU, no hang — a 503 naming the exact shortfall.

{
  "error": {
    "message": "\"qwen3-14b\" needs about 12800 MB VRAM: the host has 6144 MB free, the best node has 0 MB free, and the pool aggregate (6144 MB) is not enough to shard it either — free VRAM, add a node, or pick a smaller/more-quantized model",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

Read the numbers before you change anything. the best node is scoped to nodes that are eligible to run a whole replica, so a shard-only node’s free VRAM is deliberately not named there — it would suggest a placement this pool will never make. the pool aggregate does include shard-only nodes, because sharding is exactly what their trust level allows. Either line may carry the suffix ; N MB are held by busy or pinned residents (…), which is the daemon telling you why eviction already tried and could not clear the room.

Check which models lost their only viable placement with relay status --json | jq '.replicas' — a model that still fits without the departed node (e.g. it only ever ran on the host) is unaffected. Rejoining is relay up with no flags on the node; relay status shows it online again within one RELAY_HEARTBEAT (default 15s), and the next request to an affected model re-places it automatically — nothing to restart on the hub.

The same status for a different reason: a host-only model

A model marked host-only never leaves the hub — not as a node replica, not as a shard — so placement never looks past the host at all. It fails with the same 503 and the same model_unavailable code, but a different body and a different fix: free VRAM on the host, or lift the restriction. A node’s free VRAM is memory this model can never use, which is why the message does not mention any.

{
  "error": {
    "message": "\"qwen3-14b\" is host-only (a privacy setting — it never leaves this hub, not even as a shard): needs about 12800 MB VRAM, the host has 6144 MB free — free VRAM on the host or pick a smaller/more-quantized model",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

Confirm which it is with relay status --json | jq '.models' — the flag rides every model row as host_only. Clear it deliberately with relay model set NAME --host-only=false.

A node will not join

Join tokens are single-use. A second relay up with the same token fails by design — mint another with relay token on the hub rather than retrying the one that already worked.

Use the URL relay token prints, https:// included. On a shared network a hub with RELAY_REQUIRE_MTLS set only accepts nodes that have completed the TLS migration — for an existing node, run relay up --upgrade-tls on a trusted network first. If TLS is the failure, relay doctor https://hub:8383 says so specifically rather than reporting a generic connection error.

Node flapping between online / offline / paused

paused is not the same problem as offline: paused means the owner chose Pause lending in the tray, or enabled --auto-pause and the GPU looked busy — the tunnel is still up. The planner treats a paused node as offline for placement, but it is expected to come back the moment the owner resumes.

A node whose tunnel stays down for 60 s stops its own replica llama-server, so a replica that vanished after a longer outage is expected: while the tunnel is down the hub cannot reach the node to tell it anything, so the node has to notice on its own that nobody can be routing to it. The next request re-places it once the node is back.

A model almost fits a node

The fit-check reserves KV cache for the full served context across every slot. When the 503 body's "needs about N MB" is only a few hundred MB above a node's free VRAM, set RELAY_KV_CACHE_TYPE=q8_0 (or q4_0) on the hub rather than lowering RELAY_CONTEXT_SIZE; see Concepts. If llama-server exits immediately after that change, check for RELAY_FLASH_ATTN=off: a quantized V cache needs flash attention, and the daemon only warns about the combination rather than refusing to start.

Common doctor findings

Size caps warn — they never delete

RELAY_CACHE_MAX_GB (a node’s local file cache) and RELAY_MODELS_MAX_GB (the hub’s models directory) both default to 0, meaning no cap. Crossing either one never removes a file. A node lends disk on hardware its operator does not own, so eviction is not relay’s decision to make silently: the cap makes the situation visible and the explicit, human-invoked action stays the thing that actually reclaims space — relay cache prune on a node, relay model rm on the hub.

So a disk that keeps filling up despite a cap is working as designed. Usage against cap is reported rather than enforced: relay status carries models_used_mb/models_max_mb for the hub and cache_used_mb/cache_max_mb per node. Both values are whole gigabytes — there is no way to express 500 MB.

Cold-load timing over the tunnel

A cold shard load streams gigabytes of layers to remote machines before the server can answer /health — a 14B over a home link can take several minutes, which is why RELAY_LOAD_TIMEOUT defaults to 15 minutes rather than something tighter: a short bound turns a slow-but-working load into a spurious failure.

Windows icon cache and tray overflow

A build with a genuine icon resource can still show a blank icon in Explorer or Task Manager if the Windows icon cache is holding a previous build — the cache is keyed on path, so a rename also clears it.

ie4uinit.exe -show

Separately, Windows 11 hides every newly-seen notification-area icon behind the overflow arrow by default and records that as IsPromoted=0 — the app registered successfully and reports no error, and is still invisible.

reg add "HKCU\Control Panel\NotifyIconSettings\<key>" /v IsPromoted /t REG_DWORD /d 1 /f
taskkill /IM explorer.exe /F

Or pin it by hand: Settings → Personalization → Taskbar → Other system tray icons.

GPU busy / no VRAM detected

relay doctor's GPU check probes the same way the daemon does at startup; a no VRAM detected result names the detection source it tried (e.g. nvidia-smi) so you know whether to fix PATH or the driver.

When detection cannot be fixed — an unusual driver, a container without the probe tool — a node reads RELAY_VRAM_TOTAL_MB and RELAY_VRAM_FREE_MB and uses them in place of what it probed. Both are read on the node, not the hub, and an unparseable value is ignored rather than treated as zero. They are a last resort: you are telling the pool how much VRAM this machine has, and the planner will believe you.