Tester.
One command that joins someone else's pool as a node, benchmarks your GPU standalone and sharded, and hands back a redacted results archive.
0.1.0-alpha.N → -beta.N → -rc.N → 0.1.0 ladder. The wire protocol between hub and node, the on-disk state layout and the CLI surface can all change without a deprecation period; /v1 tracks the OpenAI shape and is the most stable thing here. Nothing is accredited or covered by a compatibility promise.This is a different path than joining a pool yourself — it's for someone who wants to lend a GPU to your pool for a one-off benchmark and hand results back, without a lasting install. One script installs everything it needs, benchmarks the tester's GPU on its own, joins your pool as a node, benchmarks the pool, packs the results, and removes everything it put on their machine. It runs on Windows (Git Bash) and Linux, no PowerShell.
As the pool owner: invite a tester
On the hub, with a model the pool should be tested on:
make tester-invite MODEL=coder-32bThis mints a single-use, shard-only join token and a named API key, and writes them, the pool URL, and the hub's CA fingerprint to tester.env (mode 0600). Nothing secret is printed to the terminal. Send that file to the tester privately, not in a shared channel — each tester gets their own file.
| Control | What it does |
|---|---|
TESTER_SHARD_ONLY=0 | Allow the tester's node to run whole-model replicas too. Shard-only is the default so no model file is copied to their disk and the pool benchmark always exercises a shard. |
TESTER_POOL_HOST | The hostname/IP the tester should reach the hub on — its Tailscale address, typically. relay pins the hub's CA fingerprint rather than checking the hostname, so an IP is fine. |
TESTER_INVITE_DRY_RUN=1 | Print the commands tester-invite would run without running them. |
RELAY_ENROLL_TTL | Set on the hub before inviting if the default 15-minute join-token window is too short for the tester to get to the script. |
When the run finishes, it offers the archive on the tester's own machine over Tailscale for 30 minutes and prints a URL; pull it with curl -O. If the tester has no Tailscale, already serves something on it, or its daemon is down, the script says so and the tester sends the archive instead.
curl -O https://<their-machine>.<tailnet>.ts.net/When the test is done, revoke the tester's access — the key and the node both outlive the run on the hub side, even though the tester's own credential is deleted when their script finishes:
relay key rm KEY_ID # the command tester-invite printed
relay node ls # find the tester's node ID
relay node rm NODE_IDAs the tester: run it
You need a machine with a GPU (NVIDIA, AMD or Intel, current driver), about 15 GB free disk while the test runs (25 GB if your GPU has 11 GB or more, since a second, larger model gets tested), and 30-60 minutes — mostly downloading. Everything is given back when the run ends.
The source and releases open soon. Until then there is nothing to install.
The source and releases open soon. Until then there is nothing to install.
Connect to the pool owner's tailnet (or VPN) first, and run within the join token's window before it expires — 15 minutes by default, so an owner inviting a tester typically raises it to 48 hours first (RELAY_ENROLL_TTL=48h) to leave real room to get to the script.
Eight stages, resumable
The script prints eight numbered stages. If it stops, it says why — fix that and run the same command again; finished downloads and benchmarks are skipped, not redone.
| # | Stage |
|---|---|
| 1 | Preflight — platform check, required tools |
| 2 | Go toolchain and relay — built in a private cache, nothing shared with any system Go install |
| 3 | llama.cpp for your OS/architecture |
| 4 | GPU check and hardware report |
| 5 | Standalone GPU benchmark |
| 6 | Join the pool |
| 7 | Pool benchmark |
| 8 | Pack results and clean up |
Give up instead of finishing with bash scripts/tester.sh --cleanup — it stops the node and removes everything the script put on your machine.
What's in the archive, and what's left behind
The finished archive is named relay-tester-HOSTNAME-TIMESTAMP.tar.gz and contains the hardware report (GPU, driver, CPU, RAM, OS version), the benchmark numbers (standalone and pooled), and the logs. The API key and join token are stripped out before it's written.
When the run ends, the tester's machine leaves the pool and the script removes everything it created — the Go toolchain and its caches, llama.cpp, the relay it built, the downloaded models, the results folder, tester.env, and relay's own node files (credential, settings, cached model data) from ~/.config/relay (%APPDATA%\relay on Windows). It prints what it freed. Relay files that existed in that location before the test are left alone. The archive is the only thing left on disk afterward — delete it and the cloned repo yourself once it's been sent.