An author's note

This project was built with LLMs. Not in the sense of prompting one to “make me a thing” but rather I had brought the problem to an LLM, proposed a rough idea for the solution and went back and forth until the design & plan was solid. I made the final choice on every decision.

Ollama Orchestrator

Internal-network orchestrator that lets a sleeping gaming PC serve Ollama on demand, fronted by an always-on host that wakes it via Wake-on-LAN and proxies traffic once it is ready.

        clients
           │  HTTP (Ollama API)
           ▼
 ┌───────────────────────┐         heartbeat poll          ┌──────────────────────┐
 │  always-on host       │ ──────────────────────────────► │  gaming PC           │
 │  ollama-orchestrator- │ ◄────────── /heartbeat ──────── │  ollama-heartbeat     │
 │  proxy                │                                 │  ollama-mode (CLI)    │
 │  (WoL + model cache)  │ ─── Wake-on-LAN magic packet ─► │  + reconciler         │
 └───────────────────────┘                                 │  ollama.service       │
                                                           └──────────────────────┘

Components

• protocol/ — shared wire contract (HeartbeatResponse, Mode, OllamaStatus, error envelope) used by both heartbeat and proxy so the JSON cannot desync.
• heartbeat/ (ollama-heartbeat) — runs on the gaming PC. Reads the mode file, polls local Ollama health, and serves GET /heartbeat. Read-only reporter (Axum); it does not touch ollama.service.
• cli/ (ollama-mode) — runs on the gaming PC.
  • ollama-mode set llm|gaming — records intent by atomically writing the mode file. Does not call systemd, so it needs no privileges.
  • ollama-mode reconcile — brings ollama.service into line with the recorded intent (idempotent). Run by systemd as the orchestrator user.
  • ollama-mode status — shows recorded intent vs. actual service state.
• proxy/ (ollama-orchestrator-proxy) — runs on the always-on host (Axum). A single state owner polls the heartbeat and maintains the backend state; request handlers read it lock-free and never mutate it.

How it behaves

Mode is the single source of truth. ollama-mode set only writes the mode file (atomically: temp + fsync + rename). A systemd .path unit watches the file and triggers ollama-mode reconcile, with a .timer as a reliable fallback. This removes the old dual-write (file + systemd) that could desync.

Model discovery while asleep. The proxy classifies requests. Discovery endpoints are answered while the backend is down and never trigger Wake-on-LAN:

• /api/tags, /api/version, /v1/models — served from a persisted cache, eagerly refreshed whenever the backend is Ready.
• /v1/models/{id} — derived from the cached /v1/models list (404 with an OpenAI-style error if the model is not in it).
• /api/show — served per model from the same persisted cache. Entries are enumerated from the tags list whenever the backend is Ready, so model details do not depend on a client having asked first; requests passing through while the backend is Ready are additionally cached write-through. Warming is batched (a few models per refresh tick) to keep the state owner responsive, so a long model list fills over a few minutes the first time the box is awake — a model not yet warmed still gets a 503. Entries for models the backend has stopped listing are pruned.
• /api/ps — synthesized: a sleeping box has nothing loaded, so the proxy answers 200 {"models":[]} truthfully without any cache.

Cache-served responses carry x-ollama-orchestrator-cache: hit (or synthetic), and the cache is persisted to disk so it survives a proxy restart / cold boot.

The "never wakes" guarantee also covers the detection gap: for up to three missed heartbeat polls (~15 s) after the box suspends, the proxy still believes it is Ready and forwards discovery requests upstream rather than serving them from cache. Those forwards fail. A failed discovery forward moves the state to Offline without a Wake-on-LAN and falls back to the cached answer, so the client still gets its 200. Only real work wakes the box. Before this, a single /api/tags or /api/show landing in that window powered the machine straight back on.

Inference (/api/generate, /api/chat, /api/embed, /v1/chat/completions, …) and everything else — deliberate management actions like /api/pull and unknown/future endpoints — trigger Wake-on-LAN when the backend is Offline, then return 503.

When not ready, the proxy returns 503 with Ollama-style JSON errors. The messages are written for end users, since chat frontends (e.g. Open WebUI) surface them directly in the conversation:

• {"error":"The AI server was asleep and is waking up now. Try again in about a minute."}
• {"error":"The AI server is starting up. Try again in a minute."}
• {"error":"The AI server is currently in gaming mode and not answering requests. Try again later."}
• {"error":"The model list is not available yet. Try again once the AI server has been online."} (empty cache)
• {"error":"Details for this model are not available yet. Try again once the AI server has been online."} (uncached /api/show)

A heartbeat that cannot read intent (mode: error, e.g. an unreadable mode file) is sticky: the proxy keeps its current state and never wakes — so a transient read error during gaming no longer powers the box on.

Auto-suspend (optional, gaming PC)

ollama-idle-suspend closes the power loop: the box suspends itself when genuinely idle, and the proxy wakes it via Wake-on-LAN on the next inference request. Shipped disabled; opt in with:

sudo systemctl enable --now ollama-orchestrator-idle-suspend.timer

A 5-minute timer counts consecutive idle checks and suspends after 3 in a row (~15 min sustained idle, tunable via IDLE_CHECKS_REQUIRED in heartbeat.env). Idle requires all of: recorded intent is llm (gaming mode is never auto-suspended), no Ollama model loaded (an empty /api/ps already implies a full keep_alive window of quiet), no established SSH connections, and no logged-in user session (greeters don't count). Any busy signal resets the streak.

Environment Variables

Proxy (always-on host)

• OLLAMA_HEARTBEAT_URL (required), e.g. http://192.168.1.42:8080/heartbeat
• OLLAMA_WOL_MAC (required), e.g. aa:bb:cc:dd:ee:ff
• OLLAMA_WOL_BROADCAST (optional, default 255.255.255.255:9)
• PROXY_BIND_ADDR (optional, default 0.0.0.0)
• PROXY_BIND_PORT (optional, default 8081)
• PROXY_CACHE_PATH (optional, default /var/lib/ollama-orchestrator-proxy/model-cache.json)
• PROXY_LOG_LEVEL (optional, default info)

The backend Ollama URL is derived from OLLAMA_HEARTBEAT_URL's host on port 11434.

Heartbeat + reconciler (gaming PC)

• MODE_FILE_PATH (optional, default /var/lib/ollama-orchestrator/mode)
• HEARTBEAT_BIND_ADDR (optional, default 0.0.0.0)
• HEARTBEAT_BIND_PORT (optional, default 8080)
• HEARTBEAT_LOG_LEVEL (optional, default info)

Build And Test

The workspace uses a shared protocol crate; build the whole thing at once. If cargo is not installed globally, use Nix (gcc provides the linker):

nix shell nixpkgs#cargo nixpkgs#rustc nixpkgs#gcc -c cargo fmt --all
nix shell nixpkgs#cargo nixpkgs#rustc nixpkgs#gcc -c cargo test --workspace
nix shell nixpkgs#cargo nixpkgs#rustc nixpkgs#gcc nixpkgs#clippy -c cargo clippy --workspace --all-targets

Or via the flake (builds + tests + clippy + fmt in a sandbox):

nix build .#heartbeat .#cli .#proxy
nix flake check
nix develop          # dev shell with the toolchain

Regenerate the lockfile after dependency changes (needed for --locked builds):

nix shell nixpkgs#cargo -c cargo generate-lockfile

Local Integration Validation

Start the mock heartbeat server (settable scenarios):

./scripts/mock-heartbeat.py

Run the proxy against it (use a tmp cache path so it can write without /var/lib):

OLLAMA_HEARTBEAT_URL=http://127.0.0.1:18080/heartbeat \
OLLAMA_WOL_MAC=aa:bb:cc:dd:ee:ff \
PROXY_BIND_ADDR=127.0.0.1 PROXY_BIND_PORT=8081 \
PROXY_CACHE_PATH=/tmp/oo-cache.json \
nix shell nixpkgs#cargo nixpkgs#rustc nixpkgs#gcc -c cargo run -p proxy

Switch mock states and probe:

curl 'http://127.0.0.1:18080/set?scenario=ready'      # then GET /api/tags warms+forwards
curl 'http://127.0.0.1:18080/set?scenario=shutdown'   # /api/tags now served from cache, no WoL
curl 'http://127.0.0.1:18080/set?scenario=gaming'
curl -i http://127.0.0.1:8081/api/tags                # discovery: cached, never wakes
curl -i http://127.0.0.1:8081/api/generate -d '{}'    # inference: 503 + WoL when offline

To observe WoL without root, point OLLAMA_WOL_BROADCAST at a local UDP listener.

Packaging

Canonical, distro-agnostic artifacts live in packaging/common/ and are consumed by both the NixOS module and the Arch package (single source of truth):

• systemd/ — units for heartbeat, proxy, and reconcile (.service + .path + .timer). Config comes from EnvironmentFile=/etc/ollama-orchestrator/*.env.
• sysusers.d/ — the ollama-orchestrator system user/group.
• tmpfiles.d/ — state dir + seeded mode file + proxy cache dir.
• ../packaging/polkit/ — group-based rule allowing ollama.service control.
• env/ — example env files.

NixOS

The root flake.nix (built with crane) exposes packages.{heartbeat,cli,proxy,default} and nixosModules.default. The module renders the env files from options, installs the shared units (repointing ExecStart at the store path), and applies the shared tmpfiles + polkit rule.

{
  inputs.ollama-orchestrator.url = "github:ollama-orchestrator/ollama-orchestrator";

  # Always-on host (proxy only):
  imports = [ inputs.ollama-orchestrator.nixosModules.default ];
  services.ollama-orchestrator = {
    enable = true;
    heartbeat.enable = false;
    proxy.heartbeatUrl = "http://192.168.1.42:8080/heartbeat";
    proxy.wolMac = "aa:bb:cc:dd:ee:ff";
  };

  # Gaming PC (heartbeat + reconciler only):
  # services.ollama-orchestrator = { enable = true; proxy.enable = false; };
}

See nix/flake-example.nix for a full two-host wiring.

Arch Linux

cd packaging/arch
makepkg -si

The package builds and ships all three binaries plus the shared units, sysusers, tmpfiles, polkit rule, and example env files. pacman's systemd hooks create the user and state dir from the shipped sysusers.d/tmpfiles.d (no imperative install script). Then, per host:

# Gaming PC
sudoedit /etc/ollama-orchestrator/heartbeat.env
sudo systemctl enable --now ollama-orchestrator-heartbeat.service
sudo systemctl enable --now ollama-orchestrator-reconcile.path ollama-orchestrator-reconcile.timer

# Always-on host
sudoedit /etc/ollama-orchestrator/proxy.env      # set OLLAMA_HEARTBEAT_URL + OLLAMA_WOL_MAC
sudo systemctl enable --now ollama-orchestrator-proxy.service

Uninstall

When an install goes sideways, scripts/uninstall.sh is the teardown button. It works regardless of how you installed (Arch, NixOS, or hand-rolled systemctl) because it only touches the units, state, config, and system user this project owns. It does not call your package manager — remove the package separately.

sudo ./scripts/uninstall.sh            # stop + disable all units, leave data alone
sudo ./scripts/uninstall.sh --purge    # also delete state, config, and the user/group
./scripts/uninstall.sh --dry-run       # preview every action, change nothing

What --purge deletes: /var/lib/ollama-orchestrator, /var/lib/ollama-orchestrator-proxy, /etc/ollama-orchestrator, and the ollama-orchestrator system user/group. It prompts before deleting (skip with --yes).

Arch

pacman -Rns ollama-orchestrator removes the package; its pre_remove hook stops and disables the units first. pacman does not remove the tmpfiles state dirs, the sysusers account, or the backed-up config (kept as .pacsave) — the post_remove hook prints the exact commands to purge those, or just run scripts/uninstall.sh --purge.

NixOS

Drop the services.ollama-orchestrator config (or set enable = false) and nixos-rebuild switch; the module's units, user, and tmpfiles entries go away declaratively. The state dirs under /var/lib/ollama-orchestrator* persist — delete them by hand if you want a clean slate.

Research Report

Upstream response-shape research and rationale for proxy 503 bodies:

• docs/ollama-response-research.md