gpu-arbiter — inference gateway

One front door for inference engines and GPU web apps:

• Apps get their own key, class and endpoint list. You create, regenerate, revoke or delete them in the UI or with the CLI.
• Endpoints are OpenAI-compatible engines (llama-swap, vLLM, llama.cpp, a remote API…), routed by the request's model, or web apps such as ComfyUI, published on their own host name.
• GPUs are shared between the endpoints that sit on them. Only one holds GPU memory at a time, following a schedule and per-class rules. The gateway frees and reclaims a GPU through llama-swap's API, by scaling a Kubernetes Deployment (any namespace, any cluster), or by starting and stopping a Docker container.

It runs in Kubernetes (Helm chart) or with docker compose (deploy/docker/).

Port	What
8080	API: OpenAI/Anthropic-compatible, per-app keys, routed by model
8081	Web: endpoint hosts (ComfyUI, an engine's dashboard), routed by Host
8090	Admin UI: GPUs, apps, endpoints, connections, rules, events

In this cluster: llama.sandou.space → API, gpu.sandou.space → UI, comfyui.sandou.space and llamaswap.sandou.space → web.

Configuration: four layers

1. The config file: endpoints, GPUs and optionally connections. In Kubernetes it is the chart's gateway: value (a ConfigMap); with docker, gateway.yaml. It's kept in git and shown read-only in the UI. arbiter/gwconfig.py documents every field.
2. The UI: endpoints and connections (Kubernetes clusters, Docker hosts) added there live in the database. They are validated against the whole configuration and applied live, with their secrets encrypted by DATA_KEY.
3. Environment: plain settings from the chart's env: or compose's .env.
4. Secrets file (SECRETS_FILE): rendered by the Vault Agent; it overrides the environment.

The config file never holds a secret. It refers to one as ${VAR} (or ${VAR:-default}), filled from layers 3–4.

Validate a file:

gpu-arbiter check-config gateway.yaml

Rules (UI → Rules)

• Classes:

  Class	Example apps	Max wait (hold_s)	Take the GPU after (preempt_after_s)	Parallel per endpoint
  realtime	Clara	120 s	0 s	no limit
  interactive	Qwen Code, OpenWebUI, browsers	600 s	120 s	no limit
  batch	MARC, n8n	20 s	900 s	1

  • Max wait is how long a request waits for the GPU before getting 503 with Retry-After.
  • Take the GPU after is how long another endpoint must have been idle before this class takes the GPU from it.

  A running job (requests in flight, or ComfyUI's queue) is never interrupted.

• Per GPU:

  • a pinned endpoint and hours: it's kept warm, and the others are "reserved" unless shared when idle is on;
  • how long each endpoint may sit idle before it's stopped (outside the pinned hours);
  • an override: pin an endpoint, switch the GPU off, or follow the schedule, for N hours.

• Waiting clients: a streaming request with no answer after 2 s gets its 200, then SSE comment lines until the engine answers. Clients ignore comments, so a handover looks like a slow first token. Opening a stopped web endpoint shows a page that refreshes itself; background requests from a tab left open never wake it.

Security

• App keys: 256-bit random, stored as SHA-256 hashes, shown once. Each is limited to its endpoints.
• Endpoint keys: stay inside the gateway. It swaps the app's key for the endpoint's own.
• Admin login:
  • the local admin, with a signed HttpOnly / SameSite=Strict cookie;
  • every change through the UI also requires an X-Requested-With header;
  • five failures from one address lock it out for a minute.
• Client address: X-Forwarded-For is believed only from TRUSTED_PROXIES (Traefik), so the throttle can't be dodged with forged headers.
• Web hosts: a basic-auth login sets a signed, HttpOnly, SameSite=Strict cookie bound to the host and endpoint. Firefox sends no basic credentials on websockets, so ComfyUI's live connection needs it. The cookie is stripped before requests reach the app.
• Kubernetes rights (in-cluster):
  • a Role per target namespace, limited to deployments/scale on the named Deployments, plus reading their pods;
  • a ValidatingAdmissionPolicy refuses any replica count above 1 from the gateway's account.
• External clusters: give their connection a token with the same narrow rights.
• Network: NetworkPolicies let only Traefik (and networkPolicy.apiFrom, for the API) reach the gateway. The engines' own namespaces admit only the gateway.
• Docker: use the socket proxy in deploy/docker/ (start, stop and inspect only), not the raw socket, which is root on the host.

Apps from a terminal

kubectl -n gpu-arbiter exec deploy/gpu-arbiter -- gpu-arbiter apps
kubectl -n gpu-arbiter exec deploy/gpu-arbiter -- gpu-arbiter app-create openwebui interactive --endpoints llamaswap
kubectl -n gpu-arbiter exec deploy/gpu-arbiter -- gpu-arbiter app-rotate openwebui     # prints the new key only
kubectl -n gpu-arbiter exec deploy/gpu-arbiter -- gpu-arbiter app-revoke legacy        # or app-enable / app-delete

With docker, use docker compose exec gateway gpu-arbiter ….

Kubernetes

• Vault kv/gpu-arbiter/config: adminPassword, sessionSecret (≥ 32 chars), dataKey (a Fernet key), legacyKey. The chart's vault.secrets maps each variable to its Vault path and key.
• Vault role gpu-arbiter: bound to service account gpu-arbiter in namespace gpu-arbiter. It reads the paths listed in vault.secrets.
• Pull secret: git-sandou-registry-secret in the namespace, because the registry refuses anonymous pulls.
• Hosts: those of endpoints from the config file are published automatically. Endpoints added in the UI need their host in ingress.extraWebHosts. For in-cluster control, their Deployment goes in rbac.extraTargets.

Docker

cd deploy/docker
cp .env.example .env
cp gateway.example.yaml gateway.yaml

Edit both files, then:

docker compose up -d

Release

CI (.forgejo/workflows/ci.yaml) runs on every push:

1. ruff and pytest;
2. the image, as git.sandou.space/pydehon/gpu-arbitrer:v<pyproject version> and :sha-<commit>;
3. the chart, to https://git.sandou.space/api/packages/pydehon/helm.

To release:

1. Bump version in pyproject.toml, and image.tag plus version in charts/chart.
2. Push.
3. Bump targetRevision in gpu-arbiter-app.yaml and apply it.

Develop

uv sync
uv run pytest -q

The tests use fake controls and a fake clock; nothing touches a cluster or a GPU.