One manifest, two formats: Lance + Tantivy

A Loam collection is a document set with dense vectors, sparse vectors, full text and filters. We store it in two open formats, because no single format does all of that well:

• Lance holds each document's source JSON, system columns and vector columns. It gives us columnar storage, vector indexes (IVF-PQ and friends) and versioning.
• Tantivy holds the full-text index and typed fields, packaged as Quickwit-style split bundles: one immutable, single-segment index per batch, with a hotcache footer so a reader opens a split with one ranged GET.

Two formats raise one hard question: how does a reader always see a Lance version and a set of splits that belong together? This post covers the answer that shipped with collection storage.

The manifest is the only linearization pointCopy Anchor Link

Every collection has a chain of immutable collection manifests at collections/<cid>/manifests/<version>-<ulid>.pb. A manifest records:

• the Lance version to read,
• the split references (each with its footer range, row-id ranges and delete-bitmap path),
• the vector index segments,
• the applied offsets per partition of the collection's log,
• the paths of this commit's primary-key delta and dead letters.

The metastore holds one pointer per collection. A commit writes all of its new objects first, then compare-and-swaps the pointer from version v to v+1. A reader that loads manifest v gets a Lance version, split set and delete bitmaps that were written to be read together. Nothing else is a commit point.

Detached Lance versionsCopy Anchor Link

Lance's normal commit path writes 1.manifest, 2.manifest and so on, and a writer that finds a newer version rebases onto it. That's a problem for us. A zombie writer (one whose lease expired but which is still running), or a writer that crashed mid-commit, could have its version absorbed by the next commit or block it entirely.

So a Loam collection's Lance dataset has exactly one mainline version: the empty version 1. Every later commit is a detached version (_versions/d<id>.manifest), built from exactly the parent manifest's Lance version. Detached versions have unique paths and never rebase. A crashed or fenced writer's version is simply never referenced by a manifest, and garbage collection removes it later.

This has a consequence for GC. Lance's own cleanup only lists mainline manifests, so it would delete files that only detached versions reference. Loam never runs Lance cleanup. Its own GC computes reachability from the Lance manifests of the retained manifest chain.

Stable row ids join the two formatsCopy Anchor Link

Lance and Tantivy need a shared key. We use Lance's stable row id: every Tantivy document stores it in a _rowid fast field, and each split reference records which row-id ranges map to which doc ids. Documents enter a split in ascending row-id order, so row id → (split, doc id) is a binary search.

Stable row ids survive Lance compaction. When Lance rewrites fragments, the splits and the primary-key index don't change.

Typed fields live only in Tantivy, where they are indexed and stored as fast fields for filters, sorts and aggregations. Lance keeps _source, system columns and vectors. Changing a field's mapping never rewrites Lance.

Deletes and upserts without rewriting splitsCopy Anchor Link

Splits are immutable. When a document is updated or deleted, the primary-key index (SlateDB, an LSM on object storage) says which row, and therefore which split and doc, held the old version. The commit writes a new roaring delete bitmap for that split, text/deletes/<split>/<ulid>.bitmap, and the new manifest points to it. Merges drop deleted docs later.

The primary-key index is derived state. It is written after the manifest CAS and carries a watermark of the manifest it reflects. If a worker crashes between the CAS and the PK write, the next worker replays the per-commit PK deltas newer than the watermark, or rebuilds the index from Lance.

Sparse vectors ride on the splitsCopy Anchor Link

Sparse vectors (Qdrant's named sparse vectors, with its idf modifier) are stored twice:

• in Lance, as a Struct<indices, values> column, which is the source of truth;
• in each split, as a postings field with one term per index plus a fast field holding the weights.

A query unions the postings of its indices, masks deleted and filtered documents, and scores the candidates exactly. Splits already handle writes, deletes, merges and the tail, so sparse search needed a scorer, not a new storage engine.

One commit, step by stepCopy Anchor Link

The link worker for a collection runs this loop for each batch it reads from the log:

1. Load manifest v. Reuse the PK index only if its watermark matches.
2. Decode the records and resolve every key through the PK index. Records that don't decode or that break the schema become dead letters.
3. Write new rows and deletes as a detached Lance version built from v's Lance version.
4. Write the new split, the changed delete bitmaps, the PK delta, the dead letters, and manifest v+1. All of these are create-only objects at new ULID paths.
5. CAS the pointer v → v+1, fenced by the worker's lease.
6. Update the PK index and its watermark.

A crash at any step leaves either the old manifest or the new one: never half of each. That is exactly what the crash gates check.

The full format reference is in §03 Storage formats, and the rulings behind it are in the decision log. The query engine, which runs vector, BM25 and sparse retrieval over these snapshots and fuses the results, has since shipped and is available.