Concepts

Durability contract

Exactly which failures a swarm survives today — a daemon crash, a lost container, a machine reboot — and which it doesn't yet.

"Resumable" and "durable" mean different things here, and the stack is the first one, not yet the second:

  • Resumable: alineod, the agent's container, or the whole machine can go down, and every agent comes back to its last completed turn — without anyone rebuilding state by hand. Everything needed for that lives on the machine alineod runs on.
  • Durable: the swarm can be rebuilt on a different machine after the disk it ran from is gone. That needs a backup and restore path that isn't built yet.

Don't take either word further than this page says — if a capability isn't in the table below, it isn't built, regardless of what any other page implies.

What survives what, today

Layeralineod crashContainer loss or machine rebootDisk or host loss
Sandbox session (client.resume)yesyes, to the last sb.checkpoint() — snapshot + exec replayno — a snapshot is a docker commit on the sandbox host
Agent SDK (Alineo.reattach / resume)yesyes, to the last turn checkpoint — a fresh container from the setup snapshot, with the latest checkpoint restored into itno — checkpoints are files next to the SDK ledger
alineod swarmyes — see Crash recoveryyes, to the last completed turn, for specs with "checkpoint": true — the restore ladderno
alineod agent memoryyesyes — keyed by agent identity, not sandboxno — one SQLite file; back it up (scripts/backup.ts)

Three things are called a snapshot or a checkpoint in this stack, and only one of them carries an agent's conversation:

  • the setup snapshot — taken once per spec, so Alineo.start() skips reinstalling Pi and packages. A fresh container restored from it is empty: right packages, no conversation.
  • the turn checkpoint — a tarball of the agent's home directory (/root: Pi's session file and what the agent wrote there, package caches excluded), taken after every turn that finishes cleanly when the spec sets "checkpoint": true. alineod takes these; the newest three per sandbox are kept. This is what restores a conversation.
  • sb.checkpoint() — the Core SDK's named OpenSandbox snapshot of a whole container, used by client.resume for exec replay.

What a restored agent has

  • Its conversation, to the last completed turn. A turn that was running when the container went is not in the checkpoint. It is recorded as agent.turn_interrupted and settles with whatever text survived — never silently dropped, never silently repeated.
  • Its home directory as of that turn. Files written outside /root — /tmp, /workspace, anywhere else — are not in a turn checkpoint and come back as the setup snapshot left them. Its waitFor inputs (/inputs/*.txt, /inputs.json) are written again by alineod.
  • Its place in the tree. Parent and child edges are kept, so it can still be prompted, steered, and used as a spawn parent.

An agent whose spec doesn't set "checkpoint": true still comes back after a lost container — onto a fresh one, with a blank session. It keeps its place in the tree but not its conversation.

Definitions

  • Resumable — after any failure in the declared set (process crash, container loss, machine reboot), the system returns to the last committed point without a human reconstructing state by hand.
  • Durable — once the system acknowledges a step as committed, no failure in the declared set can undo it. Today's declared set stops at the machine: losing its disk is outside it.
  • Honest gap — work done after the last commit is either redone or explicitly reported as interrupted (agent.turn_interrupted, or lost when an agent can't be brought back at all) — never silently lost, never silently repeated.

Side effects are at-least-once, not exactly-once. A turn that ran git push or called an external API and was then interrupted may repeat that effect on retry. Nothing in the stack de-duplicates a side effect a model decided to take mid-turn — only alineod's own commands (spawn, pause, resume, stop, steer) are idempotent, via idempotencyKey.

Known gaps

  • Disk or host loss. Checkpoints, both ledgers and alineod's database all live on one machine. Restoring a swarm onto a new machine from a backup isn't built.
  • One instance per database. alineod is a single-node control plane — see Scaling.
  • Held credential bindings. A spec with approval: "hold" credential bindings can't be restored onto a fresh container yet; if its container is gone, the agent ends lost.
  • Non-root agents. A turn checkpoint captures /root. A spec that runs Pi as a different user isn't covered.

What this unlocks today

Resumable agents, to the last completed turn — including through a machine reboot. Not "durable": a swarm does not yet survive the loss of the disk it runs from. This page will move that into the table above once a fault test proves it, not before.