Crash recovery

alineod is designed to be crash-only: stopping it with kill -9 and starting it again is a supported way to restart it. Agents run in OpenSandbox containers, not as child processes of alineod, so they keep working while the daemon is down.

What happens on startup

Rebuild state from the ledger

The agents and handles tables are discarded and rebuilt by replaying the ledger. Two kinds of agent need reconnecting: any agent in provisioning, spawning, running, paused or blocked (still live), and any finished agent (done or failed) whose sandbox is still open — a turn ending does not close the sandbox, so it stays promptable and usable as a spawn parent. aborted and lost agents are skipped: stop/DELETE /runs/:id already closed their sandbox, and a lost agent is retried again on a later boot, not this one.

Reconnect every agent that has a sandbox

Each one goes down the restore ladder until a rung holds. Most stop at the first: Alineo.reattach() binds to the harness bridge already running in the container without restarting it, so an in-flight turn is not interrupted.

An agent that was mid-turn at crash time is recorded as agent.turn_interrupted, because the process reading its stream is gone. alineod then polls it until the turn finishes and records its result — anything waiting on that agent is still released.

A paused agent is reconnected without probing its bridge — a frozen container can't answer — and stays paused.

Retry spawns that hadn't forked yet

Agents recorded before their sandbox existed — a child held by waitFor, or a root still being provisioned — are retried from what was persisted in their agent.spawned event: the spec, waitFor list, prompt, and budget. A gather agent that was waiting on workers goes back to waiting and starts normally once they settle.

A child whose parent could not be brought back ends as lost.

Hand the memory store back

Before reconnecting anything, alineod opens its memory database, and every agent it reattaches or resumes is handed that same store. Memory is keyed by the agent's identity, not its sandbox, so nothing about it needs recovering, and a reconnected parent still forks its memory into the next child it spawns. See Agent memory.

Serve requests

The daemon starts listening once reconnection is done; spawn retries continue in the background.

The restore ladder

Each rung is tried only if the one before it failed:

Rung	What it does	When it's used
Reattach	Bind to the bridge already running in the container. Nothing restarts; a running turn keeps running.	The container is up and its bridge answers. Retried once after `ALINEOD_REATTACH_RETRY_DELAY_MS` (1 s) if the first attempt fails, to ride out a sandbox that is briefly reported in the wrong state.
Resume	Restart the bridge inside the same container. A turn that was running is dropped and recorded as `agent.turn_interrupted`.	The container is up but its bridge doesn't answer.
Restore	Provision a fresh container from the spec's setup snapshot, restore the agent's latest turn checkpoint into it, re-write its `waitFor` inputs, and start the bridge on the restored session.	The container is gone: deleted, reaped, or left `Exited` by a machine reboot.
`lost`	Record the agent as `lost`, with the error.	Every rung above failed for a still-live agent.

Restore is part of both reattach and resume, so it runs whichever rung finds the container missing. The agent's recorded sandboxId is updated to the new container (agent.provisioned again), so the next boot builds on it rather than restoring the same checkpoint twice.

An agent whose spec doesn't set "checkpoint": true has no turn checkpoint to restore. It still gets a fresh container, with a blank session — it keeps its place in the tree, but not its conversation.

A finished agent whose sandbox can't be brought back keeps its recorded outcome. It just can't be prompted or used as a spawn parent again; if its sandbox is gone for good, alineod records agent.released so no later boot tries again.

Surviving a machine reboot

After a reboot, Docker leaves every agent container Exited, and OpenSandbox has no way to start one again. To alineod that is the same case as a deleted container, so the restore rung covers it: start OpenSandbox and alineod again — under systemd or Docker's restart: policy that happens on boot — and every checkpointed agent comes back on a fresh container at its last completed turn. Turns that were running when the machine went down are recorded as interrupted.

What clients see

HTTP — requests in flight during the crash fail; retry them. Use idempotencyKey on spawns so a retry can't create a duplicate child.
Result long-polls — reissue GET /agents/:id/result?wait=…. A result that settled while alineod was down, or during catch-up, is returned immediately.
Event stream — reconnect with the last id you received in Last-Event-ID. Every persisted event since then is replayed before live events resume. Text deltas produced during the gap are not replayed.

curl -N localhost:4600/runs/r_3f9a1c20/events -H 'Last-Event-ID: 128'

Verifying it

apps/alineod/scripts/swarm-reattach-test.py drives a multi-agent run, and is used to kill and restart alineod mid-run and check that every agent — including a gather still waiting on its workers — completes. Restore has been checked the same way against a real deployment: checkpoint a turn, docker rm -f the agent's container, restart alineod, and confirm the same Pi session continues on the new container.

Limits

A turn that finished while alineod was down is recorded when alineod comes back — its result is the agent's final message, but the individual stream events from the gap are not in the ledger.
Restore depends on the turn checkpoints and the SDK ledger still being on disk. Losing alineod's data directory is outside what it survives — see the durability contract.

Crash recovery

On this page