Security.

Three things are on by default and were not always: /v1 is served over TLS, /v1 requires a bearer key, and enrolment and the node’s tunnel are mutual TLS pinned by the join token. None of those is a setting you have to find. What is still yours to do is migrating any node that enrolled before mutual TLS existed, deciding whether a node is allowed to run a whole-model replica at all, and getting your clients to trust the hub’s certificate.

What is on by default

Pick your deployment shape

The distinction that used to decide everything here — whether an attacker can put a device on the same broadcast domain as the hub — stops deciding the tunnel once a node has migrated and RELAY_REQUIRE_MTLS is set. It never decided /v1 on its own once RELAY_V1_TLS defaulted to auto. It still applies, in full and unchanged, to any node that has not migrated.

How a node joins

relay serve generates a private certificate authority the first time it runs and persists it under RELAY_STATE_DIR/tls/ (ca.pem, ca-key.pem, server.pem, server-key.pem, every one of them mode 0600). Nothing about it depends on a public CA, DNS, or port 443 being reachable. A second start against the same directory returns byte-identical material rather than regenerating — the fingerprint a token pinned yesterday is still correct today.

# On the hub
relay token --label workstation

# On the joining machine — the token is <redeem-token>.<ca-fingerprint>
relay up --pool https://hub:8443 --token <token>

The token is one compound string: a single-use redeem token, a dot, and the hub CA’s SHA-256 fingerprint — the same shape kubeadm uses for --discovery-token-ca-cert-hash. The joining node verifies the presented certificate against that pinned fingerprint before sending a single application byte, not against a system trust store and not by hostname, since a home pool is routinely reached by a bare LAN IP. A wrong fingerprint fails the handshake itself. Tokens are single-use and expire after RELAY_ENROLL_TTL (15 minutes by default), and redemption plus node creation happen in one transaction, so two machines racing on one token cannot both win.

Enrolment issues the node a client certificate from a CSR the node generates itself. The node’s private key is never transmitted, in either direction, at enrolment or at a later migration; the hub signs the request with the node’s UUID as the certificate’s CommonName and returns only the certificate. From then on that certificate is the node’s credential, and the hub retires the old shared secret in the same write — a certificate-less connection claiming that node ID is refused outright, with no fallback. The node, symmetrically, keeps re-verifying the fingerprint it pinned out-of-band on every reconnect, including a resumed TLS session, so a hub swapped out later is caught the same way an impersonator at join time would be.

TLS for /v1

The same CA and server certificate cover /v1, with no client certificate — it is meant for arbitrary OpenAI-compatible clients, so its floor is TLS 1.2 rather than the node listener’s 1.3.

On a TLS start the daemon prints the CA fingerprint once, and relay status reprints it any time — it is not a secret, it is exactly what a join token embeds. A client trusts the hub one of three ways: pin the fingerprint (--ca-fingerprint, RELAY_CA_FINGERPRINT, or automatically for relay chat and relay bench against a pool this machine already knows), point the client at the CA file itself, or set RELAY_TLS_DOMAIN so /v1 presents an ACME-issued, publicly-trusted certificate and no pin is needed at all.

relay status                       # prints the CA fingerprint and the scheme in effect
relay token --api                  # prints the key to send as the bearer token

curl --cacert "$RELAY_STATE_DIR/tls/ca.pem" https://hub:8383/v1/models \
  -H "Authorization: Bearer <key>"

RELAY_TLS_DOMAIN requires RELAY_V1_TLS at auto or on and a non-loopback bind, since the CA validates by dialling in from outside. Challenges are answered on RELAY_ACME_LISTEN (default :80); explicitly setting it empty falls back to TLS-ALPN-01, which only works when RELAY_LISTEN is on 443. The node listener stays on the hub’s private CA either way.

API keys, quotas, and the audit log

relay serve calls ensureAPIKey before it constructs the daemon: if RELAY_API_KEY is unset it generates 32 bytes of crypto/rand, stores the hex at RELAY_STATE_DIR/api-key (mode 0600, written atomically), and prints it exactly once — on the run that created it, never again at startup. A key supplied through the environment is left completely alone: never persisted, never reprinted. A missing or wrong key is always 401 invalid_api_key, never a 404: there is no resource identity to hide here, so an unauthenticated caller should learn exactly one thing.

relay token --api                  # re-show the default key (reads the file; no daemon needed)
relay token --api --rotate         # rotate it — takes effect on the next `relay serve` restart

relay key add ci --tokens-per-day 2000000 --max-concurrency 4
relay key ls                       # ID  NAME  TOKENS/DAY  MAX CONCURRENCY  REQUESTS/MIN  USED TODAY
relay key rm <id>

relay audit tail -n 50             # oldest first; -n defaults to 50

A named key from relay key add is an additional bearer token with its own budget: --tokens-per-day, --max-concurrency and --requests-per-minute, each defaulting to 0. For the first two 0 means unlimited; for the third it means inherit the daemon default, not unlimited. Only a SHA-256 hash of the secret is persisted, in api_keys.json alongside the counters — the secret itself is shown once, at creation, and never again. Token cost is charged after the response, from the usage the upstream reported, because it is not knowable in advance; a concurrency slot is reserved up front. A key can therefore overshoot its daily cap very slightly on the request that crosses it, never before.

The rate limit is per key, not daemon-wide, and it is only ever evaluated after a key has matched — an unauthenticated caller gets 401 and can never probe limiter state. Node enrolment, which is unauthenticated by construction, has its own separate limiter keyed on the transport peer address rather than a forwarded header.

Every /v1 request writes one compact JSON line to RELAY_STATE_DIR/audit.jsonl (append-only, created 0600, the same file name on every install so a log shipper can rely on it): timestamp, key_id, model, placement (host, shard, or node:<name>), prompt_tokens, completion_tokens, latency_ms and status. There is deliberately no field for prompt text, completion text, the bearer key, or any request body — key_id is an id, never the key. A write failure at close time is surfaced rather than swallowed.

One more key exists that you did not create: enrolment mints a per-node convenience key named node:<node name> with an empty (unlimited) quota and returns it in the enroll response, so a script running on a lending machine can call the hub without an operator configuring one by hand. It shows up in relay key ls like any other, and relay key rm removes it.

What an attacker on your network can do today

• Against a node that has not migrated: everything the old model allowed. The legacy plaintext /api/nodes/enroll and /api/nodes/tunnel routes stay mounted on RELAY_LISTEN until RELAY_REQUIRE_MTLS is set — a credential in the clear on enrolment, prompts and completions in the clear on the replica path, weights and activations in the clear in flight. This is the gap that stays open until an operator closes it.
• Read everything on /v1 — but only if you turned TLS off. RELAY_V1_TLS=off restores the old plaintext behaviour, and the daemon will only start that way on a loopback bind while a key is configured.
• Learn that a pool exists here, if you set RELAY_ADVERTISE_LAN=1 on a network you do not administer. The mDNS record carries only a name, address, port and protocol version — no token, model, node identity or VRAM figure — but the advertisement is itself a disclosure, which is why it is off by default.
• Not impersonate the hub to a migrated node. The node pins the CA fingerprint it got out-of-band and re-verifies it on every reconnect, resumed sessions included.
• Not use your models. /v1 and /api/* answer 401 without a valid key, whether or not you ever set RELAY_API_KEY.

What is protected regardless of network

• Enrolment and the tunnel are mutual TLS, fingerprint-pinned, for every migrated node.
• Legacy node credentials: 32 bytes of crypto/rand, only a SHA-256 hash stored, compared in constant time.
• Join tokens: single-use, 15-minute default (RELAY_ENROLL_TTL), redeemed inside one database transaction.
• llama.cpp’s unauthenticated RPC wire never leaves loopback — relay refuses to start otherwise. It reaches the network only inside the tunnel.
• Nodes open no inbound ports; every node dials out. RELAY_NODE_LISTEN is a second hub-side listener, not a node one.
• Every file a node receives is hash-verified while streaming before it is renamed into the cache; on a migrated node the hub additionally signs the file’s manifest with its CA key, and the node refuses to start a replica on a file whose manifest did not verify.
• TLS private keys are 0600, and the hub never holds a node’s — both enrolment and migration exchange a CSR for a certificate, never a key.
• A node’s credential.json is sealed with the platform’s own secret store where one exists (DPAPI on Windows, an AES-256-GCM key held in the login Keychain on macOS); a sealer failure is an error, never a silent downgrade to plaintext.
• The llama-server and rpc-server processes run inside an OS-level sandbox by default — filesystem, PID, IPC and privileges, on the platforms that can — and neither is ever handed the node’s credential path, the hub’s TLS material, or its database.
• Both llama.cpp binaries are hashed and classified at startup and by relay doctor, against an embedded known-good table and your own pin file.
• The control socket is 0600 and carries no other authentication — holding it is the privilege. The control API is not on the public mux at all.
• Prompts are never logged. Request logging records method, path, status and duration; the audit log adds model, placement and token counts. Neither ever records text.

Trust model: replicas vs. shards

Replica — the node runs a whole-model llama-server on a file the hub pushed it and serves requests end to end, so it sees the complete, unencrypted prompt and completion for every request routed to it. This is the common placement: most models fit on one device somewhere in the pool. Shard — a model too big for any single device is split by layer, with rpc-server on each node executing the compute graph for its slice. A shard node sees activations for the layers it holds, not plaintext, but a determined operator of that machine could still learn a great deal from them, and rpc-server runs whatever graph the hub sends without treating that as a security boundary.

Pooling is still a way to combine machines that already trust each other, not a way to borrow compute from strangers safely. But trust is not only a placement outcome you have to accept: it is a column on every node, assigned by the operator, and the scheduler enforces it.

Two controls, and what each costs

replicas is the default for a new node, deliberately: replica placement is the pool’s primary value, so shard-only is the opt-in rather than the opt-out. The agent applies its own level as defence in depth — a node that reads shard-only on its credential refuses a start-replica request locally, in case a hub’s bookkeeping has drifted or been tampered with. Trust is a column in relay node ls, so you can see the whole pool’s at once.

Process sandboxing and the llama.cpp binaries

internal/sandbox wraps the two llama.cpp helper processes this pool ever runs — rpc-server on a node and llama-server on the hub or a node replica. RELAY_SANDBOX selects the mode on both sides: auto (the default) sandboxes when a mechanism actually works on this machine and otherwise runs unsandboxed with a warning; require fails the launch instead of degrading silently; off disables it. A node persists its own choice with relay up --sandbox auto|require|off.

The binaries themselves are consumed from upstream llama.cpp, not built here. relay doctor runs two checks against both of them: llama-build, which compares --version against the build this project’s placement math and RPC assumptions are validated against (a binary that does not implement --version is skipped, not failed), and llama-sha256, which hashes the resolved binary and classifies it against an embedded known-good table of official-release hashes and your own pin file. The four outcomes are: matches a known-good release binary; matches your pin; changed since you pinned it (the one failure); or unpinned. A finding warns — it never blocks a startup, because a node already lending VRAM has every reason to keep doing so while you decide what to do about it.

relay doctor --pin-llama    # records {path, sha256, build, pinned_at} in llama-pins.json

This is trust-on-first-use, not a signature. It does not vouch for what is being pinned right now; it only lets a later relay doctor notice that the file at that path has since changed — which is why it prints exactly what it recorded rather than an unqualified “ok”.

Trust model: the node file cache

A cached file is a copy the hub granted, not something the node fetched itself — nodes never reach out to the internet or any other source for weights. The cache is not wiped when a node leaves: relay down stops the tunnel and (unless --keep-credential) removes the credential, but pushed files persist until LRU eviction against the cache’s byte cap (default min(200 GiB, free disk − 10 GiB)) or a deliberate delete. Treat “leaving the pool” and “deleting the cache” as two separate actions: relay node rm <id> --wipe orders the node to erase its local cache the next time it tries to reconnect, and relay cache prune, run on the node itself, is the manual equivalent.

Hardening on a network you do not fully control

Nothing below is required to make a pool safe on an untrusted network — enrolment, the tunnel and /v1 are already TLS. These cover a browser-trusted certificate, or a VPN you may already run for other reasons.

# On both machines
tailscale up

# Join over the tailnet address; https:// still gets you mutual TLS
relay up --pool https://hub-hostname:8443 --token <token>

hub.example.com {
    reverse_proxy https://127.0.0.1:8383 {
        transport http {
            tls_trusted_ca_certs /path/to/RELAY_STATE_DIR/tls/ca.pem
        }
    }
}

Revoking access

relay node ls
relay node rm <id>           # its credential — and its client certificate — stops working
relay node rm <id> --wipe    # same, plus: order it to erase its local file cache

relay key rm <id>            # a named /v1 key — it no longer authenticates

Revocation is deletion, not a CRL: relay node rm removes the node’s row, and the next connection attempt — shared secret or certificate, cryptographically valid or not — fails the post-handshake identity lookup the same way an unknown node always has. A node with a live tunnel is dropped when that tunnel next reconnects, so restart the hub if you need an immediate cut. relay doctor can tell a node this before a failed relay up does: it dials the node listener with its own certificate and hits a whoami route running the identical check the tunnel does, reporting “removed from the hub” distinctly from a wrong hub or clock skew.

Known gaps

• An unmigrated node stays plaintext indefinitely. Migration is opt-in by design so a hub upgrade cannot brick a pool — which means a node nobody runs relay up --upgrade-tls against never gets the protection.
• /v1 clients must trust or pin a self-signed CA unless you set RELAY_TLS_DOMAIN, which needs a real DNS name pointed at the hub and port 80 or 443 reachable for validation. Short of that, a client that does not pin will reject the connection like any other self-signed certificate.
• On Windows the hub’s api-key file is not owner-restricted. Mode 0600 is a POSIX guarantee; Go’s Chmod on Windows maps a mode onto a read-only attribute, so the file inherits the state directory’s ACL instead. Keep RELAY_STATE_DIR somewhere only its owner can read. A node’s credential.json is unaffected — it is sealed with DPAPI.
• No CRL and no OCSP. Revocation is row deletion, which is sufficient for this threat model, but a hub restored from a backup that still has the row would accept that certificate again. Treat hub backups the way you would treat a leaked credential store.
• Client certificates do not expire on a short cycle and there is no renewal flow yet; relay node rm is the only way to invalidate one early.
• A node’s file cache is not remote-wiped by default. Plain relay node rm revokes identity only, and --wipe takes effect on the node’s next connection attempt — never instantly on one that is offline.
• The audit log and the named-key quota file are not encrypted at rest. They record key ids and token counts, never prompt text or the keys themselves, but a compromise of the state directory still discloses usage metadata.
• A Linux (or other Unix) node’s credential stays 0600 plaintext. There is no ubiquitous, unlocked-by-default OS secret store a headless server process can reach, so filesystem permissions are the whole boundary there.
• No sandbox unshares the network namespace (macOS restricting network to localhost is the one exception). The wrapped process can still reach whatever loopback ports are reachable from outside it.
• A TOFU llama pin is only as good as the binary resolved at pin time. Pin an already-tampered binary and you get no warning about it — only about a later change to that same file.

Found something not listed here? Open a private security advisory on the repository rather than a public issue.