Skip to content
Backplane

In early development · one real fleet

Every machine, accounted for.

Backplane is the fleet backplane: one place that knows what each machine is supposed to be, what it is actually doing, whether that is healthy for a machine of its kind, and what has been applied to it. And it is the one thing allowed to apply changes.

  • Declared in git
  • Observed, never executed
  • Changed by one controller
Backplane's dashboard. The columns, the health states and the vocabulary are the real ones; every machine name is invented, because this site names no host on the fleet.

Declared, observed, judged, changed

Four parts with hard lines between them. What a machine should be lives in git. What it is comes from an agent that only watches. Health is judged against what the machine promised. Change comes from one controller, over SSH.

Declared in git

Each machine is described in its own repository: a device.json for what it is, a backplane.json for which roles it gets. backplane reads them and never writes them back. Git stays the authority.

Observed by an agent that only watches

A small daemon on each machine polls out and reports what is really there: addresses, OS, disk, memory, GPU driver, and which roles were applied. It has no inbound port and never runs a command it is sent.

Healthy by its own standard

Every machine declares how available it is expected to be. A laptop asleep is not an outage; a build host that went quiet is. Incidents are announced once, and recovery once.

Changed only by the controller

Changes go over SSH with Ansible, run by one service. Every apply is preceded by a --check --diff run you can read, one run per machine at a time, with a drain for when the fleet must hold still.

One plane, every shape of machine

A build host, a laptop that sleeps, a single-board machine on a shelf, a Mac: different hardware, different operating systems, and no reason for them to be managed four different ways. They are seated in one plane, and each one is described by a file in its own repository.

What differs between them is not the mechanism. It is how available each is expected to be, and which roles it asks for.

Build hosts
Workstations
Laptops
Single-board
Macs
Backplane
  • Declare
  • Observe
  • Judge
  • Change
  • Record

Git is the authority

Each machine keeps two files in its own repository. One says what the machine is; the other says which roles it gets. backplane reads both, and writes neither.

Hand-kept descriptors drift: memory written four ways, an address typed as prose. So Backplane reads them leniently, pulls out what it can, and lints what it had to guess at instead of rejecting the file. The finding shows on the machine, where someone will fix it.

Then it lays what the agent saw beside what the file says. Where they disagree is the list worth reading: the address that changed, the OS that was upgraded and never written down.

device.json — what the machine is (an example)

{
  "name": "rack-07",
  "role": "build host",
  "os": { "distro": "Ubuntu 24.04", "arch": "x86_64" },
  "network": { "hostname": "rack-07" },
  "hardware": { "cpu": "16 cores", "ram_gb": 64, "gpu": "1x RTX" }
}

backplane.json — what it gets (an example)

{
  "availability": "always-on",
  "roles": [
    { "role": "basics" },
    { "role": "node-exporter" },
    { "role": "backplane_agent" },
    { "role": "build-runner", "vars": { "labels": ["build", "gpu"] } }
  ]
}

Quiet is not always down

A machine declares one of four availability classes, and silence means something different in each. Health is the worst of: last heard from against that class, alerts naming the machine, disk and memory, its last run, and its lint.

An incident is announced once, and its recovery once. A laptop that closes its lid every evening does not page anyone.

Availability classes and what silence means in each
Class For, say When it goes quiet
always-on a build host, the box backplane runs on An incident.
usually-on a workstation Worth a look, not a page.
sleeps a laptop Expected. When it wakes and checks in, work waiting for it is queued then.
off a machine in a drawer Not news.

One hand on the wheel

Every change goes over SSH with Ansible, and only the controller runs it. The web server, the dashboard and the agents never hold an SSH key or a secret.

The one thing the agent adds to plain push: when a machine that sleeps wakes up and checks in, Backplane sees it has changes waiting and queues the run then, instead of the push failing because the lid was shut.

The queue, as the dashboard shows it: a check, then the apply it approved. Machine names invented.
  • Check, then apply

    Every apply is preceded by a --check --diff run, and the diff is kept and shown.

  • One run per machine

    Two playbooks never converge the same host at once. Runs on different machines may run side by side.

  • Drain

    An operator can hold the whole fleet, or take one machine out while it is rebuilt. Work in flight finishes; queued work waits.

  • Secrets from ambry

    Role variables name secrets in ambry instead of carrying them. The SSH key is loaded into a per-run agent in memory and never written to disk.

  • Written down

    A finished run is recorded in backplane and appended to the machine's own CHANGELOG.md, naming the roles and the run.

What exists today

Most of it, now. Of eight phases, five are built, two are in progress and one is not built. It runs one fleet — ours — and converges machines from git. There is still nothing to download, no hosted version and no sign-up.

  1. Phase 0 · Built

    Bootstrap

    The design, the port block, and a building, testing skeleton of every component.

  2. Phase 1 · Built

    Declared inventory

    Every machine's descriptor read from its own git repository, normalized, and linted where it had to guess.

  3. Phase 2 · Built

    The agent

    The observe-only agent reporting from real machines, and the diff between what git declares and what it sees.

  4. Phase 3 · Built

    Health

    Availability classes, one rolled-up health per machine, and edge-triggered notifications that fire once.

  5. Phase 4 · Built

    The controller

    The run queue, leases, drain, check-then-apply, a per-run ssh-agent, and the changelog write-back. It has changed real machines.

  6. Phase 5 · In progress

    Roles and assignments

    A machine declares its roles, the catalog pins each at a commit, and a run is generated from both. The shared library is still filling, one reviewed role at a time.

  7. Phase 6 · Not built

    Cutover

    The machine inventory other services keep today moves here, and they read it from backplane instead.

  8. Phase 7 · In progress

    Phone app and this site

    The phone app reads the fleet on the same palette as the dashboard. This site is built and not yet served at its domain.

Built for one fleet first

backplane runs Spergs' own machines before it runs anyone else's. The about page says who makes it and why.