API.

Public routes (RELAY_LISTEN)

The public mux composes three families of routes plus a health check: the OpenAI-compatible /v1 surface, the Ollama-compatible /api surface, and the legacy plaintext node routes. Every inference route on both client dialects sits behind the same bearer-key gate.

Transport

RELAY_V1_TLS defaults to auto, which resolves to TLS unless RELAY_LISTEN is loopback-only. An empty host — :8383, the default — means every interface, which is not loopback, so a stock hub serves /v1 over TLS with its own CA. Plain HTTP happens only on an explicitly loopback bind or with RELAY_V1_TLS=off. A client configured for http:// against a default deployment does not connect.

relay serve prints the hub CA's SHA-256 fingerprint once at startup, and relay status re-prints it; that fingerprint is what the relay CLI and a joining node pin, and it is not secret. For an arbitrary HTTP client there is no pinning flag in the code — trust the hub CA instead, which is on the hub at <state dir>/tls/ca.pem. Setting RELAY_TLS_DOMAIN swaps the private CA for a publicly-trusted ACME certificate, and then no client needs either.

Auth

Every route on both client dialects is gated by requireAPIKey, and an unset RELAY_API_KEY does not mean the endpoint is open. relay serve populates the key before it builds the daemon: an env-provided key is used as-is and never persisted, and if none is set it generates one — 32 bytes of crypto/rand, hex-encoded — persists it in the state dir at mode 0600, and prints it exactly once, on the run that created it. Re-show or rotate it later with relay token --api. Clients send Authorization: Bearer <key>, compared in constant time.

Beyond that single default key, relay key add NAME mints named keys that carry their own quotas — --tokens-per-day, --max-concurrency and --requests-per-minute, each defaulting to 0. Only a SHA-256 hash of a named key is persisted; the secret is shown once at creation and never again. A presented token is matched against the default key first and then against every stored hash. Neither matching is 401 invalid_api_key — never 404, because there is no resource identity to hide here.

Request bodies are capped at 8 MiB. Prompts are text; anything past a few megabytes is a mistake or an attack, not a conversation.

GET /v1/models

{
  "object": "list",
  "data": [
    {
      "id": "qwen3-8b",
      "object": "model",
      "created": 1735689600,
      "owned_by": "relay",
      "context_length": 16384,
      "state": "warm",
      "fits": true,
      "fit_reason": "",
      "placement": "host"
    }
  ]
}

Everything past owned_by is additive — an SDK that does not know these fields ignores them and still gets a working OpenAI-shaped list. state, fits, fit_reason and placement come from the pool's own view of the model: a client's model picker is meant to let a user select a warm model, while cold, loading and unavailable ones are listed with fit_reason explaining why an unavailable one cannot be picked.

POST /v1/chat/completions

# /v1 is TLS by default; copy the hub's <state dir>/tls/ca.pem to the client.
curl https://hub.internal:8383/v1/chat/completions \
  --cacert ./ca.pem \
  -H "Authorization: Bearer $RELAY_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-8b","messages":[{"role":"user","content":"hello"}],"stream":true}'

When model is omitted, the pool uses the sole registered model if there is exactly one; with zero or several registered models it returns 400 invalid_request naming the ambiguity instead of guessing.

A cold model load streams weights across the pool before it can answer, so the handler gives it RELAY_LOAD_TIMEOUT plus one minute of headroom, long enough for the supervisor's own timeout (which produces the useful error) to fire first.

Streaming responses are proxied with FlushInterval: -1, so tokens arrive as tokens rather than one buffered blob, on both the host/shard path (loopback dial) and the node-replica path (one tunnel stream per request, no keep-alive across requests).

The Ollama surface (/api)

A large share of the local-AI ecosystem speaks Ollama's wire dialect rather than OpenAI's, so /api/tags, /api/chat and /api/generate are mounted alongside /v1 on the same listener. This is a translation layer, not a byte-for-byte proxy. llama-server already speaks OpenAI's shape, so /v1 forwards bytes unchanged; Ollama's format is different — newline-delimited JSON objects rather than server-sent events, with its own field names — so each request here is translated onto the same call handleInference would make and the response translated back into Ollama's NDJSON shape. Placement, error mapping, the bearer gate, the quotas, the rate limit and the audit log are all shared with /v1 verbatim; only the wire shape differs.

Two differences worth knowing. Real Ollama has no auth of its own, but these routes are gated anyway — an operator who has locked down /v1 must not have that undone by a second, unauthenticated door onto the same pool — so send the same key in the same Authorization: Bearer header. And /api/tags lists only models that currently fit the pool, where /v1/models lists every registered model and reports fit as a field.

Error codes

A placement failure produces one of three distinct 503 bodies, all in the same OpenAI envelope. The first is captured from a real run; the other two are the format strings the scheduler builds them from.

{
  "error": {
    "message": "\"qwen3-14b\" needs about 12800 MB VRAM: the host has 6144 MB free, the best node has 0 MB free, and the pool aggregate (6144 MB) is not enough to shard it either — free VRAM, add a node, or pick a smaller/more-quantized model",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

{
  "error": {
    "message": "\"qwen3-14b\" is host-only (a privacy setting — it never leaves this hub, not even as a shard): needs about 12800 MB VRAM, the host has 6144 MB free — free VRAM on the host or pick a smaller/more-quantized model",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

{
  "error": {
    "message": "\"qwen3-70b\" needs about 48000 MB and the whole pool tops out at 30720 MB",
    "type": "model_unavailable",
    "code": "model_unavailable"
  }
}

Control plane: GET /status

The control API is served on the unix socket (RELAY_SOCKET, created 0600 and owned by the daemon's user), not on RELAY_LISTEN. It has no authentication of its own; the filesystem permission is the whole access-control story. relay status --json prints the /status document verbatim. The other routes back the relay node, relay model and relay key subcommands: GET/PATCH/DELETE /nodes, POST /nodes/token, POST /nodes/{id}/wipe, GET/POST/PATCH/DELETE /models, POST/DELETE /keys and POST /keys/default/rotate.

The control plane is versionless. It carries no version field, makes no compatibility promise, and moves with the binary — the CLI is the only intended client, so fields are added and reshaped whenever the CLI needs them. Read it for operator tooling if you like, but pin the relay version you read it with.

None of the control plane is part of the client contract. It is the CLI's own channel over the unix socket; a client of the pool — an OpenAI-compatible SDK, an Ollama-shaped tool, an editor plugin, an agentic coding tool — consumes only the public inference surface, and needs nothing from relay but a base URL and a key to do it. That narrowness is deliberate, but /v1 is not the whole public API: the pool speaks two client dialects, OpenAI's on /v1 and Ollama's on /api, so nothing has to be written against relay specifically in either ecosystem.