§33 Loams Flow Connectors: Registry, Capabilities and the Catalog

Status: Proposed · 2026-10-01. Source: the owner's drafts "Précis (CDMP, Java stack)", "Top 200 connectors" and "Rollout" (chatdump.md lines 551–628), read as the design of Loams Flow's connector registry, and the owner's direction of 2026-10-01 that event ingestion runs on Apache Iggy and Apache Fluss (§32 D331–D333). This document makes decisions D352–D359 and asks Q348–Q359. It depends on §32 for the envelope (D334), the bridges (D336), Flow routes (D337–D339) and the fabric/ workspace (D343). No code is written by this document.

Names follow §32's owner rulings of 2026-10-01: packages under loams (@loams/* on npm), Go modules under loams.dev/..., CloudEvents types io.loams.dev.<domain>.<name>.v1. Java is deferred: the Camel connector layer stays, as an unmodified Camel runtime driven by generated YAML routes, but any Loams-written Java (a Java SDK, a Kestra plugin, custom Camel processors) waits until the owner lifts the deferral.

Markers are §32's: (verify), (estimate), (read 2026-10-01). "The précis" is the draft's "Précis (CDMP, Java stack)"; "CDMP" there is the draft's name for the canonical event contract, which in Loams is CloudEvents 1.0 (D334).

1. SummaryCopy Anchor Link

2. Goals and non-goalsCopy Anchor Link

2.1 GoalsCopy Anchor Link

1. Breadth without writing hundreds of adapters (the précis' first line): count unique connectors, reuse Camel's 300-plus components, Iggy's plugins and Debezium, and own only the hot path.
2. Honest capabilities: a user can see, before running a route, whether a connector is a source or sink, streaming or batch, transactional or not, and which auth it takes.
3. One envelope (D355) and one bulk path (D356).
4. Testable: every declared capability has a contract test; the starred set has end-to-end tests against real services in CI.

2.2 Non-goalsCopy Anchor Link

• No Loams-written connector where a maintained Apache-2.0 one exists in Iggy, Camel or Debezium.
• No JVM in the engine or in loams-fabric: Camel and Debezium run in their own pods.
• No orchestration layer of Loams's own: schedules, retries and batch flows around connectors are Resonate workflows (D210); Kestra users keep Kestra.
• Exactly-once claims only where the external system deduplicates.
• Remote ingress waits for the auth plan. Webhook and OTLP receivers are on ingest's loopback listener (D111), so in CN1 remote providers and collectors cannot reach them, except through a tunnel or reverse proxy the operator runs on the same host (documented as development-only). With the unified auth plan (Q30) they are exposed through the edge (Envoy, D184) with TLS and per-route credentials.

3. Reconciliation with the précisCopy Anchor Link

4. The capability schema (D353)Copy Anchor Link

apiVersion: loams.flow/v1
kind: Connector
id: kafka                          # [a-z0-9-]{1,48}, unique
name: Apache Kafka
specVersion: 1.0.0                 # of this manifest; bumps on capability changes
category: messaging                # messaging | relational | nosql | search | vector | graph | warehouse | lakehouse
                                   # | object-storage | format | cdc | integration | saas | observability | infra | identity | ai | protocol
priority: P1                       # P1 = CN1 (★), P2 = CN2, P3 = CN3
starred: true
status: preview                    # planned | preview | stable | deprecated
runtime:
  kind: native                     # native | iggy | camel | debezium | openapi
  ref: loams_flow::connectors::kafka   # crate path | Iggy plugin name | Camel URI scheme | Debezium connector class | OpenAPI spec URL
  version: "rdkafka 0.39 / librdkafka 2.x (verify)"
licence:
  component: Apache-2.0
  dependencies: { librdkafka: BSD-2-Clause }
capabilities:
  source:
    streaming: true
    batch: false
    cdc: false
    webhook: false
    resumable: true                # restarts from a committed position
    position: kafka-offsets        # what the source checkpoints
  sink:
    streaming: true
    batch: true
    transactional: false
    upsert: false
    delete: false
    idempotent: true               # the sink itself dedupes retries (Kafka idempotent producer)
  delivery: { source: at_least_once, sink: at_least_once }
  ordering: per_partition          # none | per_key | per_partition | total
  formats: [cloudevents-binary, cloudevents-structured, json, avro, protobuf, bytes]
  schema: { registry: optional, evolution: backward }
  bulk: { arrow: false, max_batch_rows: 65536 }
  backpressure: pull               # pull | push-with-ack | push-rate-limited
auth: [none, sasl-plain, sasl-scram-256, sasl-scram-512, mtls, aws-msk-iam]
config:                            # JSON Schema 2020-12 for an instance's settings
  $ref: schemas/kafka.config.json
secrets: [sasl.password, tls.key_pem]   # resolved through Dapr secret stores (D189); never stored in the instance
envelope:
  emits: io.loams.dev.flow.kafka.record.v1
  consumes: "*"
limits: { max_record_bytes: 16777216 }
conformance: [contract, roundtrip, kill-restart, dup-check]
docs: docs/guides/connectors/kafka.md

Rules:

1. Refusal. ValidateRoute refuses a route whose from connector lacks the source mode it uses (for example cdc on a polling source), whose to connector lacks upsert when the route upserts, or whose delivery asks for more than the connector declares. The error names the connector, the capability and the manifest version.
2. Runtime-specific truth. The same system can have different capabilities per runtime (Camel's producer/consumer support, Iggy's source/sink plugins, Kestra's task/trigger split differ); the manifest describes the runtime Loams ships, and Appendix A shows the others.
3. Tests per capability. conformance lists the suites the connector passes: contract (each declared capability exercised), roundtrip (sink → source returns the events, where both exist), kill-restart (no loss after killing the runtime mid-batch), dup-check (duplicates bounded by the declared delivery), bulk (Arrow path at 1 M rows), cdc (inserts, updates, deletes, DDL, slot or binlog resume).
4. Versioning. A capability removed or narrowed bumps specVersion's major; routes pinned to the old major keep running until migrated, and ListConnectors shows both.

The protobuf form (proto/loams/flow/v1/connector.proto, FL3/CN1) mirrors the YAML one to one; the YAML is the source and CI checks that both agree.

5. Runtimes (D354)Copy Anchor Link

Why these and not others:

6. Envelope and delivery (D355)Copy Anchor Link

• Source events. One CloudEvent per record, row change or file. id is derived from the source position so a re-read produces the same id: Kafka "<topic>/<partition>/<offset>"; Debezium the connector's source.lsn/binlog position plus table and key; S3 "<bucket>/<key>@<etag>"; batch reads (ADBC, Postgres/MySQL batch, Iceberg, Parquet objects) "<run-id>/<partition>/<first-row>-<last-row>" for batch-of-rows events, where run-id is the first 16 hex digits of SHA-256(instance ‖ query or object ‖ as-of position: snapshot id, exported snapshot or cursor value), so a retry or a re-read of the same run at the same as-of position yields the same ids, and a later run yields new ones (datacontenttype: application/vnd.apache.arrow.stream). subject is the table, key or path. Extensions: loamsconnector, loamsinstance, and for CDC loamsop (c, u, d, r) and loamslsn.
• Pass-through. Input that already is a valid CloudEvent keeps its type, source and id (Kafka records with ce_ headers, Debezium CloudEvents, structured CloudEvents webhooks); the connector adds only loamsconnector and loamsinstance. Routes therefore filter on loamsconnector or on the producer's type, never on the io.loams.dev.flow. prefix alone; a manifest that passes events through declares envelope.passthrough: true, and its contract test checks both the pass-through and the wrapped case.
• Sinks. A sink receives CloudEvents (or Arrow batches for bulk sinks) and maps them per its manifest; HTTP sinks send ce-* headers and Idempotency-Key: <ce_id>; database sinks upsert by the declared key when upsert: true.
• Positions. A native source commits its position after Iggy (or ingest) acknowledges the batch; Iggy plugins commit Iggy offsets after the sink acknowledges; Debezium commits its offsets after the HTTP sink's 2xx. Every runtime is therefore at-least-once, and D334's dedup key or the target's PK makes it effectively once where the target dedupes.

7. CDC (D357)Copy Anchor Link

Postgres / MySQL / SQL Server / Oracle / Db2 / MongoDB
   │ logical replication · binlog · CDC tables · change streams
   ▼
Debezium Server 3.7 (CloudEvents converter; HTTP sink)  ──or──  Iggy postgres_source (CDC mode)
   │ POST /v1/namespaces/{ns}/fabric/topics/{topic}/events   (structured or batched CloudEvents)
   ▼
loams-fabric ingest (dedup by source + id)  →  Iggy topic <db>.<schema>.<table>
   │ fluss_sink
   ▼
Fluss PK table <table>_current (Versioned on loamslsn; deletes on loamsop = d)   +   Fluss Log table <table>_history
   │ tiering
   ▼
Iceberg (current + history)  →  House: SELECT … FINAL, SCD2 view over history (valid_from = ts, valid_to = next ts per key)

• Loams Postgres (§28) and WeSQL (§29) use Loams's own change bridges (D154) into collections; a route can forward those changes to the Fabric through the iggy link target (§32 D336).
• Snapshots use Debezium's incremental snapshots (signal table), which emit loamsop = r.
• Monitoring: replication-slot lag in bytes and seconds, Debezium's MilliSecondsBehindSource, and an alert when retained WAL passes a threshold (default 10 GiB, (estimate)), because an abandoned slot fills the source's disk.
• Schema changes flow as Debezium schema-change events into the DLQ-free <db>.schema_changes topic; the route either evolves the Fluss table (additive) or pauses with an error (incompatible), as links do (§09 §4).

8. The ★ set (D358)Copy Anchor Link

Rollout: CN1 the 21 above; CN2 every P2 row of Appendix A through stock Camel components (loams-connect) or Iggy plugins, each with a manifest and contract tests; CN3 the P3 rows, most through OpenAPI-generated native connectors. CN2 and CN3 are not planned yet.

9. CN1 exit gateCopy Anchor Link

• Every ★ connector has a manifest that validates, a licence that passes D359, and its conformance suites green in CI against real services (containers for Kafka, Postgres, MySQL, Debezium, RustFS as S3, Lakekeeper, Elasticsearch, ClickHouse, Redis, LocalStack-free Kinesis through floci (D60), and recorded fixtures for Snowflake and BigQuery plus a nightly job against real accounts when credentials exist).
• End to end: Postgres CDC through Debezium into a Fluss PK table and a Loams collection, with inserts, updates, deletes and a Debezium restart, equals the source table; a 10 M-row Snowflake (fixture) or Postgres read through ADBC lands as Arrow batches in a Fluss Log table in under the budget measured in CN1 Task 0.
• loams-fabric connectors list|describe|validate and the generated catalog page docs/guides/connectors/index.md.

10. Where it lives (open core)Copy Anchor Link

All of §33 is open source (D220): self-hosters need connectors. loams-connect (YAML route templates and configuration for the unmodified Camel runtime) lives in this repository under connect/ with a path-filtered CI job, or in its own repository (Q353). loam-platform adds the managed fleet: per-tenant connector pods, autoscaling, plan limits on connector count and throughput, the hosted secrets UI, connector metering (through the usage hooks, §27).

11. RisksCopy Anchor Link

12. Open questionsCopy Anchor Link

13. SourcesCopy Anchor Link

All read on 2026-10-01: github.com/apache/camel tag camel-4.22.1 (components/ and the camel-aws, camel-azure, camel-google, camel-debezium, camel-ai, camel-salesforce module lists; components/camel-iggy/src/main/docs/iggy-component.adoc; components/camel-clickhouse/src/main/docs/clickhouse-component.adoc); github.com/kestra-io plugin repositories (193 plugin-* repositories; module lists of plugin-jdbc, plugin-aws, plugin-gcp, plugin-azure, plugin-notifications, plugin-serdes, plugin-fs); github.com/kestra-io/kestra v2.0.4 README; github.com/debezium/debezium v3.7.0.Final and github.com/debezium/debezium-server (module list); github.com/apache/iggy core/connectors/{sdk,sinks,sources} and postgres_source/README.md; crates.io: adbc_core and adbc_driver_manager 0.24.0 (arrow ≥ 58, < 60), rdkafka 0.39.0, rskafka 0.6.0, sqlparser 0.63.0; github.com/adbc-drivers/{snowflake,bigquery} (Apache-2.0); github.com/supabase/etl (Apache-2.0, a Rust Postgres replication library, a candidate for Q351); github.com/airbytehq/airbyte (licence NOASSERTION, ELv2 platform); github.com/redpanda-data/connect (no single SPDX licence). Loams: §02 §7.4, §09, §21, §24 (D184, D189), §26 (D210), §27, §28, §29, §32, docs/open-core.md.

Appendix A. The connector matrixCopy Anchor Link

Columns: Source / Sink — the connector reads from / writes to the system. Streaming — continuous, low-latency delivery. Batch — bulk or scheduled transfer. CDC — row-level change capture. Webhook — receives (as a source) or sends (as a sink) webhooks. Auth — the methods the Loams manifest will declare: none, basic (user and password), key (API key or token), oauth2, jwt, hmac (signed payloads), mtls, sasl (PLAIN/SCRAM), iam (AWS), sa (GCP service account or workload identity), aad (Entra ID), ssh, kerb (Kerberos), cs (connection string). Camel — the Camel 4.22.1 component(s) that cover it, or ·. Kestra — the Kestra plugin (and sub-module) that covers it, or ·. Priority — P1 = ★ (CN1), P2 = CN2 (stock Camel or Iggy plugins), P3 = CN3 (long tail, OpenAPI-generated). Y = yes, · = no.

The Camel and Kestra columns were filled from the component and plugin lists read on 2026-10-01; the capability cells are the planned manifest values and are (verify) until each connector's contract tests pass (D353 rule 3). CN1 Task 2 generates this table from the registry and fails CI if they disagree.

A.1 Messaging and streaming (20)Copy Anchor Link

A.2 Relational (16)Copy Anchor Link

A.3 NoSQL, search, vector and graph (20)Copy Anchor Link

A.4 Warehouse, lakehouse, OLAP and compute (20)Copy Anchor Link

A.5 Object storage and files (10)Copy Anchor Link

A.6 Formats (6)Copy Anchor Link

Formats are codecs used by other connectors; "Source" and "Sink" mean decode and encode.

A.7 CDC (6)Copy Anchor Link

A.8 Integration and orchestration (6)Copy Anchor Link

A.9 CDP, marketing, analytics and ads (22)Copy Anchor Link

A.10 CRM, sales and support (12)Copy Anchor Link

A.11 Commerce and payments (8)Copy Anchor Link

A.12 Developer and collaboration (14)Copy Anchor Link

A.13 Observability and incident (10)Copy Anchor Link

A.14 Infrastructure and runtime (8)Copy Anchor Link

A.15 Identity (4)Copy Anchor Link

A.16 AI (10)Copy Anchor Link

AI connectors are mostly sinks used as enrichment steps (embedding, classification, extraction) inside a route, with the result written back into the event.

A.17 Generic protocols (8)Copy Anchor Link

Totals: 200 connectors; 21 ★ (P1), and the P2 and P3 counts are computed by CN1 Task 2's generator.