What we're building, and how.
What Loam is, how each part works, and the open-source projects it stands on. Every post says what is available today and what is still being built.
Series
The Loam platform
One post per pillar: the vision, the engine, the reactive database, durable execution, functions, jobs, SQL and self-hosting.
- Loam: the AI-native cloud in your bucket8 min
- The engine: hybrid retrieval, streams, Iceberg and the wire APIs8 min
- Loam Live: a reactive database on TiKV9 min
- Durable execution: Resonate, embedded9 min
- Functions billed on CPU time7 min
- Jobs on Loam: Celery, BullMQ, PySpark and Flink SQL7 min
- Postgres and MySQL on your bucket7 min
- Self-hosting Loam with GitOps6 min
What Loam is built on
Deep dives on the open-source projects under Loam: how each works, why we chose it, how we use it, its license and its limits.
- Resonate: durable promises, and the server we embed6 min
- TiKV and PD: Raft, Percolator, keyspaces and our client fork7 min
- RustFS: the object store under a self-hosted Loam4 min
- Apache DataFusion and Arrow: the query engine under every Loam read5 min
- Lance: the columnar format that holds Loam’s documents and vectors5 min
- Tantivy and Quickwit’s splits: full-text search on object storage5 min
- Apache Iceberg, iceberg-rust and Lakekeeper: tables any engine can read5 min
- Dapr: the API surface, and secrets from every cloud5 min
- workerd, wasmtime and gVisor: three ways to run code you did not write5 min
- OpenFGA: relationship-based authorization for Loam4 min
- Beside Loam, not inside it: Sail, RisingWave, Neon and WeSQL6 min
- The small crates that carry a lot: SlateDB, foyer, openraft, redb, qdrant-edge and connect-rust5 min
Loam: the AI-native cloud in your bucket
An AI application today runs five or six stateful systems. Loam puts retrieval, application data, durable workflows and jobs on one platform, with object storage as the ground it all stands on.
The engine: hybrid retrieval, streams, Iceberg and the wire APIs
How Loam's retrieval engine turns a bucket into vector, full-text and graph search behind the Qdrant, Elasticsearch and Flight SQL protocols, and where streams and Iceberg tables fit.
Loam Live: a reactive database on TiKVLoam Live: a reactive database on TiKV
Queries that stay live and push new results, mutations that are transactions, server functions, a protobuf sync API, and a change feed that makes application data searchable. How it works, and what is built.
Durable execution: Resonate, embeddedDurable execution: Resonate, embedded
Loam links the Resonate server into its own binary, so the official Resonate SDKs get durable promises, retries, schedules and human approvals from the system that holds the agent's memory. Here is how, and the patterns we build on it.
Functions billed on CPU timeFunctions billed on CPU time
An agent that computes for 10 ms and waits 20 s on a model should pay for 10 ms. Our proposal for Loam Functions, with workerd, wasmtime and gVisor tiers and Resonate for long waits.
Jobs on Loam: Celery, BullMQ, PySpark and Flink SQLJobs on Loam: Celery, BullMQ, PySpark and Flink SQL
Our proposal for running the job frameworks teams already use on Loam with a configuration change, on one Rust queue core over TiKV, with durable steps from Resonate when you want them.
Postgres and MySQL on your bucketPostgres and MySQL on your bucket
Real applications need real Postgres and MySQL. What Loam will serve over the Postgres wire itself, what we learned running Neon and WeSQL on RustFS, and why one is under evaluation and the other is not adopted.
Self-hosting Loam with GitOpsSelf-hosting Loam with GitOps
How the whole stack is meant to deploy from one Git repository, from a laptop cluster to your own data centre, on RustFS, the TiKV operator, a Loam operator and Argo CD. And the line between what is open source and what only the managed cloud runs.
Resonate: durable promises, and the server we embedResonate: durable promises, and the server we embed
What Resonate is, how durable promises and replay work, why Loam chose it over Temporal and the alternatives, how we link its Rust server into our binary through a fork, and its license and limits.
TiKV and PD: Raft, Percolator, keyspaces and our client forkTiKV and PD: Raft, Percolator, keyspaces and our client fork
How TiKV replicates and transacts, what PD does, why API v2 keyspaces made TiKV Loam's transactional store, what its change feed can and cannot do for us, and why we carry a fork of the Rust client.
RustFS: the object store under a self-hosted LoamRustFS: the object store under a self-hosted Loam
Loam's source of truth is a bucket, so the bucket has to keep its promises. What RustFS is, why it replaced MinIO as Loam's default self-hosted store, what Loam needs from any S3 store, and what we still have to verify.
Apache DataFusion and Arrow: the query engine under every Loam readApache DataFusion and Arrow: the query engine under every Loam read
Every Loam read, whether a Qdrant query, an Elasticsearch search or SQL, compiles to a DataFusion plan over Arrow batches. How DataFusion works, what Loam adds to it, why we embed it, and the version lockstep that comes with it.
Lance: the columnar format that holds Loam’s documents and vectorsLance: the columnar format that holds Loam’s documents and vectors
How Lance lays out fragments, versions and indexes for fast random access on object storage, why Loam stores collections in it instead of Parquet, the detached versions we commit, and the churn we pin against.
Tantivy and Quickwit’s splits: full-text search on object storageTantivy and Quickwit’s splits: full-text search on object storage
How Tantivy indexes and scores text, how Quickwit's split format makes an index readable from a bucket with one ranged GET, why Loam vendors Quickwit's code instead of running Quickwit, and how we keep BM25 scores independent of the split layout.
Apache Iceberg, iceberg-rust and Lakekeeper: tables any engine can readApache Iceberg, iceberg-rust and Lakekeeper: tables any engine can read
How Iceberg tables commit on object storage, what iceberg-rust can and cannot write today, why Lakekeeper is Loam's catalog, and what Loam plans to add on top for AI analytics. All of it planned.
Dapr: the API surface, and secrets from every cloudDapr: the API surface, and secrets from every cloud
What Dapr is and how its sidecar and components work, why Loam plans to serve a subset of Dapr's API from a Rust server instead of running a sidecar per function, and how tenant secrets would flow through Dapr's secrets building block.
workerd, wasmtime and gVisor: three ways to run code you did not writeworkerd, wasmtime and gVisor: three ways to run code you did not write
The three isolation technologies behind Loam's proposed function tiers. How V8 isolates, WebAssembly and a user-space kernel each contain code, what each costs, how each can be metered, and why workerd never runs two tenants in one process.
OpenFGA: relationship-based authorization for LoamOpenFGA: relationship-based authorization for Loam
How OpenFGA's Zanzibar-style model answers "may this user do this to that", why Loam plans it as its fine-grained authorizer beside built-in roles, the tenant fence we add to Lakekeeper's model, and why tuple writes go through an outbox.
Beside Loam, not inside it: Sail, RisingWave, Neon and WeSQLBeside Loam, not inside it: Sail, RisingWave, Neon and WeSQL
Four engines Loam plans to run next to itself as separate processes. How each works, what it would do for Loam, its license and its limits, including the one we evaluated and did not adopt.
The small crates that carry a lot: SlateDB, foyer, openraft, redb, qdrant-edge and connect-rustThe small crates that carry a lot: SlateDB, foyer, openraft, redb, qdrant-edge and connect-rust
Six Rust libraries inside Loam that get less attention than DataFusion or Lance but hold up the primary-key index, the cache, the metastore, the HNSW hot tier and every protobuf API. What each does, how Loam uses it, and its limits.
Crash gates: how we test kill -9 at every stepCrash gates: how we test kill -9 at every step
Named failpoints, a store fault matrix and a seeded linearizability simulation. This is what "no acknowledged write is ever lost" means in Loam's test suite.
One manifest, two formats: Lance + TantivyOne manifest, two formats: Lance + Tantivy
How Loam keeps a Lance dataset and a set of Tantivy splits mutually consistent on S3, with one compare-and-swap as the only commit point.
Why a retrieval engine should live on object storageWhy a retrieval engine should live on object storage
Most of an AI app's search data is cold most of the time. Here is why Loam keeps every byte in your bucket and treats compute as a cache.