Introduction.
Relay pools VRAM across teammates' machines and serves it from one hub — an OpenAI-compatible /v1 alongside an Ollama-compatible /api. It is a host-owned virtual GPU accelerator, not a forge and not an agent platform.
A team with a few idle gaming GPUs does not need to rent cloud compute or pay per token to run real inference. relay turns those machines into one pool: one person runs the hub — it owns every model and serves the API — and everyone else runs relay up to lend VRAM whenever their card is free.
The hub is the only place model weights live. Nodes never download anything from the internet — the hub pushes a hash-verified copy of a GGUF into a node's cache before that node can run it, and a node's cache is a copy the hub granted, never a source of truth. Clients only ever talk to the hub, never to a node directly. The hub answers on one public address with two request surfaces behind the same bearer key: an OpenAI-compatible /v1 and an Ollama-compatible /api (see API).
What pooling buys you
Two different things happen depending on why a team pools GPUs, and they have opposite performance characteristics. Serving several people at once is replicas — a whole independent copy of a model on every device that has room, scaling linearly with machines. Running one model too big for any single card is sharding — the model's layers split across devices, which lets the pool host something no single card could, but generation is a relay: one device computes while the others wait their turn, so it does not get faster with more machines and is measurably slower than the same model on hardware that could hold it outright.
/v1 answers with an explicit 503 model_unavailable rather than silently falling back to CPU. What it is not
- Not a git forge. Relay serves inference; it has no repository hosting, no PR workflow, no issue tracker.
- Not an agent-graph engine and not an IDE harness. Relay does ship one first-party client:
relay chatis a terminal REPL that talks to the pool's/v1/chat/completionsand carries file and shell tools scoped to the directory it starts in — read, write, list, search, and a run-command tool gated by an interactive confirm. It is a subcommand of the samerelaybinary asrelay serve, and it exists so the pool is usable without writing a client first. What the daemon does not carry is the layer above that: no agent graph, no editor integration, no verify-loop orchestration. Point any OpenAI- or Ollama-compatible client at the hub and that layer is the client's concern.
License and status
Relay is Apache-2.0 and self-hosted: no account, no telemetry, no per-token cost. It is alpha, unaccredited software — every release is an automatic pre-release cut from main, on the ladder 0.1.0-alpha.N → -beta.N → -rc.N → 0.1.0, and nothing has reached 0.1.0 yet — and the project's own docs are written to be useful rather than reassuring about what is and is not yet protected (see Security).