Apps: durable, self-healing workloads

A sandbox is ephemeral — you spin it up, run something, tear it down, and a daemon restart drops it. An app is the opposite: a named workload the daemon manages over time. It keeps a healthy instance running, restarts it on failure, health-checks it, and — the headline — re-creates it from spec after a daemon restart or host reboot.

Use crucible run for throwaway work; use crucible app for a server you want to stay up.

crucible app create web --image nginx:alpine -p 8080:80 \
  --restart always --health http:80:/
crucible app ls
# NAME  DESIRED  PHASE    HEALTH   RESTARTS  INSTANCE
# web   running  running  healthy  0         sbx_9f2ac1

What an app is

An app is desired state the daemon converges toward. It owns at most one running instance at a time — an ordinary sandbox, booted from the app's image with its published ports, network policy, and entrypoint. The app's name is a stable handle; its instance id changes each time the instance is (re-)created.

Desired state is persisted in a small control-plane store (separate from the ephemeral sandbox registry), so it outlives the daemon.

Survives a restart

This is the point of apps. When the daemon restarts (an upgrade, a crash, a host reboot), the old instances are gone — but each app's spec is still in the store, so the reconciler boots a fresh instance from it. Your app comes back.

It is re-created, not live-re-attached: the new instance is a cold boot from the image (~a couple of seconds), and in-VM memory from before the restart is gone. That is exactly right for a stateless server (nginx, an API, a worker) — the 95% case. (Surviving with in-VM memory intact is later trajectory work.)

crucible app create web --image nginx:alpine -p 8080:80 --restart always
curl localhost:8080          # served by the instance
sudo systemctl restart crucible
curl localhost:8080          # served again — a fresh instance, re-created from spec

Forks stay ephemeral by design: fork fan-out is short-lived exploration, so surviving a restart is irrelevant to it.

Self-healing

The daemon keeps the app healthy along two axes.

Restart policy governs what happens when the instance dies:

Restarts use exponential backoff (1s, doubling, capped at 60s) so a broken instance isn't hot-looped, and a crash-loop guard: after several fast failures the app enters a crashlooping phase (surfaced in status) and is retried at the capped interval — the same shape as Kubernetes' CrashLoopBackOff. An instance that runs healthy past a window resets the failure count, so a one-off crash hours later restarts normally rather than counting as a loop.

Two restart levels, don't confuse them: the guest supervisor restarts a crashed process inside a live instance; the daemon reconciler (this) boots a replacement when the whole instance is gone or unhealthy.

Health checks are the liveness signal the daemon probes:

--health http:80:/       # GET / on guest port 80, expect 2xx
--health tcp:5432        # TCP connect to guest port 5432 succeeds

An instance that fails its health check past the threshold is destroyed and restarted (subject to the backoff above). A start-period grace window means slow starters aren't killed while warming up. Without a health check, "process alive" is the liveness signal. (An exec health check — running a command in the guest — and seeding checks from an image's HEALTHCHECK are a follow-up.)

The crucible app commands

create flags: --image (required), --pull, --restart, --health, -p/--publish (repeatable), --net-allow (repeatable), --vcpus, --memory, --disk, --stopped (create without starting an instance).

logs/exec/shell resolve the app's current instance for you, so you never juggle the instance id.

Status fields

app get / app ls surface the observed status the reconciler maintains:

• phase — pending (booting / backing off), running, crashlooping, stopped
• health — healthy, unhealthy, unknown (no check, or in the start period)
• restarts — how many times the daemon has restarted the instance
• instance_id — the sandbox currently backing the app (empty when none)

From the API / SDKs

Apps are the REST /apps routes (see api.md) and are first-class in the Go SDK:

cr.CreateApp(ctx, api.CreateAppRequest{AppSpec: api.AppSpec{
    Name:    "web",
    Image:   &api.ImageRef{OCI: "nginx:alpine"},
    Publish: []api.PortMapping{{HostPort: 8080, GuestPort: 80}},
    Restart: wire.RestartPolicy{Policy: wire.RestartAlways},
    Health:  &api.HealthCheck{Type: "http", Path: "/", Port: 80},
}})

app := cr.App("web")
res, _ := app.Exec(ctx, wire.ExecRequest{Cmd: []string{"nginx", "-t"}}, os.Stdout, os.Stderr)

MCP agents get create_app / list_apps / get_app / delete_app tools (see mcp.md), under the same operator guardrails as sandbox creation.