Resonate: durable promises, and the server we embed

Resonate is an open-source durable-execution system built around one idea: durable promises. It is the engine behind Loam's durable execution (platform part 4). This post covers how it works, why we chose it, how Loam uses it, and where its edges are.

What it isCopy Anchor Link

A durable promise is a promise whose state lives on a server instead of in a process's memory. It has an id, a state (pending, resolved, rejected, canceled or timed out), a value, a timeout and tags. Anyone who knows the id can create it, await it or settle it, from any process, in any language, at any time.

Durable functions are built on top. A function runs through a context, and every call it makes through that context is recorded as a durable promise:

• ctx.run(f, args) runs f and records its result. On replay the recorded result is returned and f is not called again.
• ctx.rpc(name, args) asks another worker, possibly in another language, to run a function, and returns a durable promise for the result.
• ctx.sleep(duration) is a promise that a timer resolves.
• ctx.promise() creates a promise that something outside the program settles: a webhook, or a human pressing Approve.

When the process running a function dies, the server notices (the task's lease lapses) and hands the function to another worker, which runs it again from the top. Every recorded call returns immediately with its stored value, and execution continues from the first call that has no result. That is the same replay model Temporal uses, expressed as promises rather than an event history.

Inside the serverCopy Anchor Link

The server holds three kinds of state: promises, tasks and schedules.

• Promises are grouped by origin. A workflow is a tree of promises under one root, and the server commits every transition of one origin atomically. Guarantees are linearizable per origin and independent across origins.
• Tasks deliver work. When a promise needs a worker, the server creates a task and sends it through a transport: HTTP poll (a worker holds a server-sent-events stream open, the SDK default), HTTP push (the server calls a serverless function), or others. A worker acquires the task with a fenced lease (a version number compared-and-set on acquire). Delivery is best effort; a lost message is recovered by the task's retry timeout, 30 seconds by default.
• Schedules create promises from a template on a cron expression. The promise id includes the tick's timestamp and the insert is idempotent, so a tick fires once even if the schedule is processed twice.

The current server is written in Rust and assembled from plugins: storage engines (SQLite, Postgres, MySQL, a blob store for object storage, and others), transports (HTTP push and poll, Google Pub/Sub) and gateways (HTTP, a web console, metrics). The monorepo also carries a server written in Zig that keeps all state in S3, and one written as a single SQL file on Postgres.

A specified protocolCopy Anchor Link

What set Resonate apart for us is how carefully the protocol is specified. The repository has an executable abstract machine in Lean 4, a TLA+ model, a catalogue of properties, and a trace checker that replays a real server's traffic against the model. Each storage engine is held to an engine differential and a port differential against that model, and live servers are checked for linearizability with a port of the model to porcupine. For a component that will hold agents' run state, that is the evidence we want.

Why we chose itCopy Anchor Link

The plugin architecture decided it. A spike linked the Rust server into an axum application using Resonate's public build, start and stop calls, ran the official Python fan-out example against it (in crash mode, only the failed branch re-ran), and survived kill -9 of the whole binary in the middle of a human-in-the-loop workflow. No upstream change was needed.

How Loam uses itCopy Anchor Link

• Linked in, not beside. The crate operon-durable builds a registry of only the plugins Loam carries (SQLite, MySQL, the HTTP gateway, the poll and push transports) and starts the server in Loam's own runtime. It never calls Resonate's run, which installs a global tracing subscriber and waits for signals. Configuration comes from Loam's flags, never from resonate.toml. Resonate's option to abort the process on a handler panic is pinned off.
• The standard protocol on 127.0.0.1:8001, Resonate's default port, so the official SDKs work with no change but a URL. The listener refuses non-loopback addresses until Loam's auth plan exists, and the push transport is off by default, because on an unauthenticated server any caller could make Loam send HTTP requests to an arbitrary address.
• Loam's own workflows through the Rust SDK, in process. The SDK accepts a custom network. Loam's implementation calls the server directly and delivers tasks to an in-process worker plugin, so Loam's long operations (bulk import first) replay with the SDK's semantics without touching the listener.
• Storage: SQLite on a single node; TiDB through the MySQL plugin in clusters today; a TiKV store for the blob server is being validated as the cluster target.
• Conformance: Loam's CI runs Resonate's own linearizability tooling against the embedded server and requires the checker to report linearizable.

The forkCopy Anchor Link

The server crates are not on crates.io, so Loam depends on a pinned revision of dina-kar/resonate. The fork branch is upstream plus a small set of commits, each also proposed upstream:

• Dependency hygiene: newer rusqlite and sqlx, and reqwest on rustls instead of OpenSSL. Upstream at the pinned revision failed our cargo deny policy with eight RustSec advisories; this cleared seven, with no source change. The one left (the Marvin timing issue in rsa, reached through sqlx's MySQL driver) has no fix and does not apply to TLS connections, and is ignored with that rationale.
• TiDB support in the MySQL plugin: TiDB's error numbers and a pessimistic-transaction pin, so commit conflicts retry instead of answering 500.
• Feature gates: Google Cloud's ID-token auth behind a feature, and the Rust SDK without default TLS features, since Loam talks to the server in process.

Loam never reads, copies or links Resonate's ScyllaDB storage plugin or its NATS pieces, which have a BUSL-1.1 lineage.

LimitsCopy Anchor Link

• No tenants. Groups, schedules and searches are global to one server. Loam plans to isolate tenants by running one server instance per namespace over the namespace's own store; today the embedded server is one instance. Serving per-tenant routes behind Loam's dispatcher needs a small upstream change (the router constructor is private today).
• No retention. The protocol has no delete for settled promises, and no backend prunes them, so state grows without bound. Pruning is subtle: a late retry of a pruned id runs the work again.
• Versions must match. Server 0.10.x refuses the PyPI Python SDK 0.7.x; the monorepo SDK 0.8.1 works.
• Young, vendor-led, moving fast. The protocol evolves quickly. Pinning a revision and running the conformance suite on every bump is how we live with that.
• Not a transaction across systems. A step and a Loam write are not atomic. Steps must be idempotent; Loam uses each step's promise id as the idempotency key of its write.

The next post in this series covers the other system Loam leans on for state that does not live in the bucket: TiKV.