The engine: hybrid retrieval, streams, Iceberg and the wire APIs

The retrieval engine is the oldest part of Loam and the part that exists most completely. It replaces the vector database, the search cluster and, for the narrow query shape GraphRAG needs, the graph database, with one Rust binary over one bucket. This post covers its data model, how a write and a read move through it, the protocols it speaks, and the pieces that are still designs: graph expansion, public streams and Iceberg tables.

Five kinds of objectsCopy Anchor Link

A namespace holds five kinds of objects. Three are live today.

The idea that ties them together is that the log is the spine. Every write, whatever protocol it arrives on, lands in a stream first. A write to a collection goes to that collection's implicit stream. Collections, graphs and tables are materializations of streams, kept up to date by links that record their applied offset atomically with every commit. That gives exactly-once materialization, and it gives every write a consistency token that means something in any read on any object.

The write pathCopy Anchor Link

A collection write is admitted only while the collection's unapplied backlog is under a budget (a million records or 128 MiB by default). Past it, the write gets HTTP 429 or gRPC RESOURCE_EXHAUSTED with a Retry-After. That backpressure is what keeps strong reads cheap: the backlog a read may need to merge always fits in memory.

Each collection is one Lance dataset for documents and vectors, plus a set of Tantivy splits for full text and typed-field indexes. A chain of immutable manifests binds them, and a compare-and-swap on the manifest pointer is the only commit point. We wrote about that in One manifest, two formats. Upserts and deletes go through a primary-key index on SlateDB, which maps a key to where its current row lives, and through per-split delete bitmaps. The "What Loam is built on" series covers each of those projects.

The read pathCopy Anchor Link

Every read compiles to an Apache DataFusion plan, whichever protocol it came in on. Loam adds its own physical operators:

A hybrid query is one plan: filter bitmap → (vector search ‖ BM25) → fusion → fetch. Nothing crosses a network hop between retrieval and fusion, and the planner sees the whole query.

Consistency. By default a read is strong: it sees every write acknowledged before it began. It does that by reading the durable state at the manifest's applied offset and merging the log tail after it. A read that carries a consistency token is guaranteed to see that write, on any node. eventual skips the tail for lower latency.

Scores that do not depend on layout. BM25 statistics are computed over every live document in every split plus the tail, not per split, and deleted or shadowed documents never count. So a document's score is the same before and after a split merge, and on a hot node or a cold one.

The hot tierCopy Anchor Link

Stateless compute over object storage is only fast with caching. Each query node owns the collections that rendezvous hashing assigns to it, and keeps four layers of derived state for them:

• H0: manifests, split footers and file metadata, in RAM.
• H1: byte ranges of durable objects in a RAM and NVMe cache (foyer). Objects are immutable, so the cache never needs invalidation.
• H2: acceleration structures, today HNSW graphs built with qdrant-edge, Qdrant's index code packaged as a library. The graphs are built from a manifest and stored back in the bucket, so another node can load them instead of rebuilding.
• H3: the in-memory tail of records already in the log but not yet indexed.

A differential test runs the same queries with the hot tier on and off and requires identical results.

The wire APIsCopy Anchor Link

Loam takes over the role each system plays in an AI stack, not every feature of its protocol. The footprint is deliberately narrow.

There is no Neo4j Bolt or Cypher surface and no ClickHouse surface. Graphs are reached through native expansion and SQL table functions, and analytics through Iceberg and Flight SQL. The Elasticsearch surface is scoped by external conformance suites, not by feature parity: no Kibana, no Painless, no full Query DSL.

The same retrieval is reachable from SQL, with the function names Spice uses:

SELECT id, _score
FROM rrf(
  vector_search('memories', [0.12, 0.34, 0.56], 'embedding', 100),
  text_search('memories', 'quarterly revenue', 'body', 100)
)
LIMIT 10;

Auth. None of these listeners has authentication yet. One auth plan will cover all of them at once. Until then the new gateways bind to 127.0.0.1 by default, and the durable and Live listeners refuse any non-loopback address.

What is still designCopy Anchor Link

Graph expansion for GraphRAGCopy Anchor Link

GraphRAG, LightRAG and Cognee retrieve with a narrow shape: seed by vector or BM25, expand one or two hops, rerank. Running a separate graph database for that one step adds a silo and an ETL path. Loam's design keeps expansion in the query engine: graphs are mapped over collections and tables (a vertex label maps to a keyed source, an edge type to a source with (src_key, dst_key) columns), adjacency is stored as CSR and CSC sidecars per segment, and an ExpandExec operator does filtered one-to-two-hop expansion inside the same plan as the seed query. Graph algorithms such as Leiden, PageRank and weakly connected components become table functions. Adapters for LightRAG and the LlamaIndex property graph are planned. None of it is built.

Streams as a productCopy Anchor Link

The log exists and carries every write. What is planned is exposing streams directly: a native stream API over HTTP and gRPC with idempotent producers, streaming subscribe and named consumers; OTLP log ingest; and later a Kafka wire-protocol gateway, modelled on WarpStream's leaderless design, so RisingWave, Flink, Spark, Kafka Connect and Debezium can read and write Loam streams unchanged. Segments already use the Kafka RecordBatch v2 layout internally, and a test checks that the kafka-protocol crate decodes Loam's batches and that Loam decodes its batches.

Iceberg tablesCopy Anchor Link

Tables for AI analytics (LLM calls and costs, traces, evals) will be Apache Iceberg tables in your bucket, catalogued in Lakekeeper, so DuckDB, Trino, Spark, ClickHouse and Snowflake read them without Loam. Loam adds keyed tables (one live row per key, through deletion vectors), a hot tier for Iceberg (a file index and sorted projections on NVMe) and a real-time tail, so a query sees rows seconds after they are written while external engines see snapshots at commit cadence. The Iceberg deep dive covers what that needs upstream.

Where this standsCopy Anchor Link

We have not published performance numbers, and will not until the exit benchmarks (BEIR relevance against Elasticsearch BM25, recall against Qdrant, the framework suites) run against a release. The next post moves off the bucket, to the part of Loam that needs real transactions: the reactive database on TiKV.