Quickstart
Build the operon binary, write records to a stream and watch a link materialize them.
Operon has no release yet, so you build it from source. The M0 binary runs every role in one process and serves a native HTTP API for namespaces, streams and links.
Prerequisites
- A Rust toolchain. The repository pins the version in
rust-toolchain.toml, andrustupinstalls it on first build. curl, to call the API.
Build and start a node
git clone https://github.com/dina-kar/operon.git
cd operon
cargo run --release -p operon -- devoperon dev keeps the metastore and a local bucket in .operon/ and listens on 127.0.0.1:8080. Check that it's ready:
curl -i localhost:8080/ready/ready answers 200 once the metastore has a leader.
Create a namespace and a stream
curl -X POST localhost:8080/v1/namespaces -d '{"name": "acme"}'
# {"id":1}
curl -X POST localhost:8080/v1/namespaces/acme/streams \
-d '{"name": "events", "partitions": 2}'
# {"id":1}Names are unique within their scope. Creating one that exists answers 409 with the existing id:
{"error":"already_exists","id":1,"message":"rejected: namespace already exists: 1"}Run a link
Declare a link before writing, so you can watch it catch up. M0 has one link target, counter, which parses each record's value as an integer and sums it per key. Records whose value isn't an integer are counted as skipped.
curl -X POST localhost:8080/v1/namespaces/acme/links \
-d '{"name": "totals", "source": "events"}'
# {"id":1}With no target, a link writes to a counter named after itself.
Write records
Keys and values are base64. Here YQ== is a, Yg== is b, NQ== is 5, MTA= is 10 and LTI= is -2.
curl -X POST localhost:8080/v1/namespaces/acme/streams/events/partitions/0/records \
-d '{"records": [{"key": "YQ==", "value": "NQ=="}, {"key": "Yg==", "value": "MTA="}]}'
# {"base_offset":0,"last_offset":1,"token":[{"offset":1,"partition":0,"stream":1}]}
curl -X POST localhost:8080/v1/namespaces/acme/streams/events/partitions/1/records \
-d '{"records": [{"key": "YQ==", "value": "LTI="}]}'
# {"base_offset":0,"last_offset":0,"token":[{"offset":0,"partition":1,"stream":1}]}The reply is sent once the records are durable in the bucket. token is the consistency token for the write.
Read them back
curl "localhost:8080/v1/namespaces/acme/streams/events/partitions/0/records?offset=0"{
"records": [
{"offset":0,"key":"YQ==","value":"NQ==","headers":[],"timestamp_ms":1790272455527},
{"offset":1,"key":"Yg==","value":"MTA=","headers":[],"timestamp_ms":1790272455527}
],
"next_offset": 2,
"high_watermark": 2,
"log_start_offset": 0
}To wait for new records instead of polling, add max_wait_ms (up to 60000). The request returns as soon as records arrive or the wait runs out.
Check the link
curl localhost:8080/v1/namespaces/acme/links/totals{
"id": 1,
"name": "totals",
"source": "events",
"target": {"kind": "counter", "name": "totals"},
"options": {},
"version": 1,
"applied": [{"partition": 0, "offset": 2}, {"partition": 1, "offset": 1}],
"counters": {"a": 3, "b": 10},
"skipped": 0
}applied is, per partition, the next offset the link will read: every record before it is reflected in counters. The link committed its result and those offsets in one step, so it never counts a record twice, even across crashes.
Look inside the bucket
Everything Operon stored is under .operon/bucket:
.operon/bucket/wal/standard/1/01M3A8VH33B8HFH62CW49PP73X.wal
.operon/bucket/wal/standard/1/01M3A8VHBQRJE9F5MDQBQZC823.wal
.operon/bucket/ns/1/links/1/data/01M3A8VKJ48QCJG8Z7RY5NWKWH.cnt
.operon/bucket/ns/1/links/1/manifests/00000000000000000001-01M3A8VKJ431D5AE6JQ836GXV6.manThe WAL objects hold your records. A background segmenter later rewrites them into per-partition segments under ns/1/streams/. The link's data and manifest files are immutable; the metastore holds a pointer to the current manifest.
Use a real bucket
operon standalone runs the same process against object storage:
cargo run --release -p operon -- standalone --bucket s3://my-bucket/operon--bucket takes an s3://, gs:// or file:// URL. The metastore's local database stays in --data-dir.