# Crash recovery URL: /docs/alineod/concepts/crash-recovery How alineod reconnects to a live swarm after a restart — and restores agents whose containers are gone — without interrupting turns that are still running. alineod is designed to be **crash-only**: stopping it with `kill -9` and starting it again is a supported way to restart it. Agents run in OpenSandbox containers, not as child processes of alineod, so they keep working while the daemon is down. ## What happens on startup [#what-happens-on-startup] ### Rebuild state from the ledger [#rebuild-state-from-the-ledger] The `agents` and `handles` tables are discarded and rebuilt by replaying the ledger. Two kinds of agent need reconnecting: any agent in `provisioning`, `spawning`, `running`, `paused` or `blocked` (still live), and any **finished** agent (`done` or `failed`) whose sandbox is still open — a turn ending does not close the sandbox, so it stays promptable and usable as a spawn parent. `aborted` and `lost` agents are skipped: `stop`/`DELETE /runs/:id` already closed their sandbox, and a `lost` agent is retried again on a later boot, not this one. ### Reconnect every agent that has a sandbox [#reconnect-every-agent-that-has-a-sandbox] Each one goes down the [restore ladder](#the-restore-ladder) until a rung holds. Most stop at the first: [`Alineo.reattach()`](/docs/agent/api-reference/alineo#alineoreattach) binds to the harness bridge already running in the container **without restarting it**, so an in-flight turn is not interrupted. An agent that was mid-turn at crash time is recorded as `agent.turn_interrupted`, because the process reading its stream is gone. alineod then polls it until the turn finishes and records its result — anything waiting on that agent is still released. A paused agent is reconnected without probing its bridge — a frozen container can't answer — and stays paused. ### Retry spawns that hadn't forked yet [#retry-spawns-that-hadnt-forked-yet] Agents recorded before their sandbox existed — a child held by `waitFor`, or a root still being provisioned — are retried from what was persisted in their `agent.spawned` event: the spec, `waitFor` list, prompt, and budget. A gather agent that was waiting on workers goes back to waiting and starts normally once they settle. A child whose parent could not be brought back ends as `lost`. ### Hand the memory store back [#hand-the-memory-store-back] Before reconnecting anything, alineod opens its memory database, and every agent it reattaches or resumes is handed that same store. Memory is keyed by the agent's identity, not its sandbox, so nothing about it needs recovering, and a reconnected parent still forks its memory into the next child it spawns. See [Agent memory](/docs/alineod/concepts/memory). ### Serve requests [#serve-requests] The daemon starts listening once reconnection is done; spawn retries continue in the background. ## The restore ladder [#the-restore-ladder] Each rung is tried only if the one before it failed: | Rung | What it does | When it's used | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Reattach** | Bind to the bridge already running in the container. Nothing restarts; a running turn keeps running. | The container is up and its bridge answers. Retried once after `ALINEOD_REATTACH_RETRY_DELAY_MS` (1 s) if the first attempt fails, to ride out a sandbox that is briefly reported in the wrong state. | | **Resume** | Restart the bridge inside the same container. A turn that was running is dropped and recorded as `agent.turn_interrupted`. | The container is up but its bridge doesn't answer. | | **Restore** | Provision a fresh container from the spec's setup snapshot, restore the agent's latest [turn checkpoint](/docs/alineod/concepts/durability-contract#what-survives-what-today) into it, re-write its `waitFor` inputs, and start the bridge on the restored session. | The container is gone: deleted, reaped, or left `Exited` by a machine reboot. | | **`lost`** | Record the agent as `lost`, with the error. | Every rung above failed for a still-live agent. | Restore is part of both reattach and resume, so it runs whichever rung finds the container missing. The agent's recorded `sandboxId` is updated to the new container (`agent.provisioned` again), so the next boot builds on it rather than restoring the same checkpoint twice. An agent whose spec doesn't set `"checkpoint": true` has no turn checkpoint to restore. It still gets a fresh container, with a blank session — it keeps its place in the tree, but not its conversation. A **finished** agent whose sandbox can't be brought back keeps its recorded outcome. It just can't be prompted or used as a spawn parent again; if its sandbox is gone for good, alineod records `agent.released` so no later boot tries again. ## Surviving a machine reboot [#surviving-a-machine-reboot] After a reboot, Docker leaves every agent container `Exited`, and OpenSandbox has no way to start one again. To alineod that is the same case as a deleted container, so the restore rung covers it: start OpenSandbox and alineod again — under systemd or Docker's `restart:` policy that happens on boot — and every checkpointed agent comes back on a fresh container at its last completed turn. Turns that were running when the machine went down are recorded as interrupted. ## What clients see [#what-clients-see] * **HTTP** — requests in flight during the crash fail; retry them. Use `idempotencyKey` on spawns so a retry can't create a duplicate child. * **Result long-polls** — reissue `GET /agents/:id/result?wait=…`. A result that settled while alineod was down, or during catch-up, is returned immediately. * **Event stream** — reconnect with the last `id` you received in `Last-Event-ID`. Every persisted event since then is replayed before live events resume. Text deltas produced during the gap are not replayed. ```bash curl -N localhost:4600/runs/r_3f9a1c20/events -H 'Last-Event-ID: 128' ``` ## Verifying it [#verifying-it] `apps/alineod/scripts/swarm-reattach-test.py` drives a multi-agent run, and is used to kill and restart alineod mid-run and check that every agent — including a gather still waiting on its workers — completes. Restore has been checked the same way against a real deployment: checkpoint a turn, `docker rm -f` the agent's container, restart alineod, and confirm the same Pi session continues on the new container. ## Limits [#limits] * A turn that finished while alineod was down is recorded when alineod comes back — its result is the agent's final message, but the individual stream events from the gap are not in the ledger. * Restore depends on the turn checkpoints and the SDK ledger still being on disk. Losing alineod's data directory is outside what it survives — see [the durability contract](/docs/alineod/concepts/durability-contract).