Chat & Clients.

relay chat

relay chat                                          # picks the model, or asks if there's more than one
relay chat --model qwen3-8b                         # skip the picker
relay chat --url https://hub:8383                   # talk to a remote pool instead of the local daemon
relay chat --url https://hub:8383 --ca-fingerprint <fingerprint>  # pin the hub's CA explicitly

--url’s https:// pool is TLS by default (RELAY_V1_TLS=auto), signed by the hub’s own CA — relay chat resolves a fingerprint to pin automatically in most cases (this machine’s own daemon, or a node credential enrolled with that hub); otherwise pass --ca-fingerprint or set RELAY_CA_FINGERPRINT to the fingerprint relay status prints on the hub. See Security’s “Migrating clients”.

Pure Go, no CGO — works the same in a Linux terminal, macOS Terminal.app, and Windows Terminal.

relay chat — model qwen3-8b
Type /help for commands, /exit or Ctrl-D to quit, Ctrl-C to cancel a reply in progress.
you › explain the pool's placement order
assistant › Relay places a model on the cheapest strategy that fits: ...
2.1 tok/s

• Fenced code blocks render distinctly from prose.
• Ctrl-C cancels the reply in progress without leaving the REPL; a second Ctrl-C (or Ctrl-D / /exit) quits.
• A line ending in \ continues onto the next line, for pasting multi-line input.
• If the model has to load for the first time, relay chat prints “loading model across the pool…” instead of sitting silent — cold loads on a pool are measured in minutes, not seconds, so a hang would be indistinguishable from progress otherwise.
• The conversation (including any /system prompt) lives in memory for the session only; nothing is written to disk.
• Respects RELAY_API_KEY — set it in the environment before running relay chat if the pool requires one.

Slash commands

/status is the one worth knowing about: run it mid-session any time you want to see whether a node paused, whether a replica moved, or how much VRAM is free before loading something bigger.

Pointing another client at the pool

Nothing on the client side needs to know a pool exists. /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models all speak the plain OpenAI dialect, gated by an optional bearer token (Authorization: Bearer $RELAY_API_KEY) if the hub set one. Point any of these at https://<hub>:8383/v1 — /v1 is TLS by default (RELAY_V1_TLS=auto), signed by the hub’s own CA, so a client also needs to trust or pin that CA (<RELAY_STATE_DIR>/tls/ca.pem, or the fingerprint relay status prints); RELAY_TLS_DOMAIN on the hub gets you a publicly-trusted certificate instead, needing no client-side trust setup at all, and RELAY_V1_TLS=off is the escape hatch back to plain HTTP while a client migrates (loopback bind only — with an API key present, the default, the hub refuses plaintext /v1 on a network address). See Security’s “Migrating clients” for the full detail.

curl --cacert <RELAY_STATE_DIR>/tls/ca.pem https://hub:8383/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer '"$RELAY_API_KEY" \
  -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"hello"}]}'

from openai import OpenAI
import httpx

client = OpenAI(
    base_url="https://hub:8383/v1",
    api_key="unused",
    http_client=httpx.Client(verify="<RELAY_STATE_DIR>/tls/ca.pem"),
)
resp = client.chat.completions.create(
    model="qwen3-8b",
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="")

Ollama-native clients work too, on their own surface

A client that speaks Ollama’s wire dialect rather than OpenAI’s does not need an OpenAI-compatible mode: /api/tags, /api/chat and /api/generate are mounted on the same listener as /v1, behind the same API key. Point the client at the pool’s base URL and it works unchanged. It is a translation layer rather than a byte-for-byte proxy — Ollama’s format is newline-delimited JSON rather than server-sent events — so see the Ollama surface on API for which fields survive the trip.

What does not work

• LM Studio’s own client protocol — not OpenAI-shaped.
• Anything that loads weights in-process (HF transformers, torch, llama.cpp bindings used directly) — there’s no remote-GPU shim; the pool is a server you talk HTTP to, not a CUDA driver replacement.

A request that gets a 503

If the pool genuinely can’t place the requested model right now — nothing fits it as a replica and it can’t be sharded across what’s left either — you get an explicit error naming the shortfall, not a hang or a silent CPU fallback:

{
  "error": {
    "message": "\"qwen3-14b\" needs about 12800 MB VRAM: the host has 6144 MB free, ...",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

See Troubleshooting for exactly when this fires and how to read the numbers in the message.