§29 WeSQL as Loams's MySQL-on-the-Bucket OLTP Engine

Status: Proposed · 2026-10-01. This document comes from the owner's question, "will wesql become usable with proper transaction support, refer tidb (go) or starrocks (java), implement all starrocks features on wesql, both use rocksdb+iceberg on s3", and from the direction the owner approved in answer: verify the transaction model, then close WeSQL's gaps in four milestones (W1–W4), keep analytics on Iceberg beside it, and do not port StarRocks. Everything here is a proposal: decisions D273–D280 and open questions Q271–Q279. It extends §23 (D148, D154, D156, Q50, Q51) and builds on §28 §7.2 (Arm A). It changes no code in Loams; the code work is in the fork ostrium-labs/wesql and, for W2, in loams-safekeeper.

Markers, as in §23 and §28:

• (verify) means not checked against a primary source, or checked only by reading code that was not run. The PR that depends on it checks it first.
• (estimate) means computed or reasoned, not measured.
• (spike) means measured in the 2026-09-29 spike of §23 §9.
• Paths of the form wesql/… point into ostrium-labs/wesql branch 8.0 at eef34f452 (2026-09-29, after the four merged fork PRs). Paths of the form mysql/… point into Oracle mysql-8.0.46 (976e5da9). tidb/… points into pingcap/tidb master at 8936d7b (2026-09-25). starrocks/… points into StarRocks/starrocks main read through the GitHub API on 2026-10-01.
• Nothing in this document is posted to wesql, pingcap or StarRocks. Upstream work needs the owner's explicit go-ahead (§23 §2.2).

1. SummaryCopy Anchor Link

2. Goals and non-goalsCopy Anchor Link

2.1 GoalsCopy Anchor Link

1. A MySQL engine for apps that need one, with real transactions, foreign keys, and a durability and failover story that holds when the local volume is lost.
2. Storage entirely on the bucket. Snapshots, SmartEngine's extents and the log live in RustFS, and no committed transaction depends on a local disk surviving (§6).
3. Reuse Loams's pieces, build only the join. Arm A's acceptors (§28 §7.2), the TiKV metadata and x/ records (§23 §6.2), the binlog bridge (D154), Iceberg and Lakekeeper (§08).
4. An honest record of what SmartEngine, TiDB and StarRocks are (§3), because the owner's question rested on a mixed-up picture of the three.

2.2 Non-goalsCopy Anchor Link

• Porting StarRocks or TiDB code into WeSQL (§8.3).
• Multi-primary or distributed transactions in WeSQL. It is one writer at a time (D278).
• Making WeSQL the default MySQL answer. TiDB is better on every axis except the bucket-only storage (§9).
• Linking WeSQL into Loams, or patching it in a way that makes Loams's code GPL (D11, D148, §10).
• Upstream contributions without the owner's go-ahead.

3. Correcting the recordCopy Anchor Link

Each claim in the question, checked against primary sources on 2026-10-01.

What the corrected picture means. Iceberg is where WeSQL, TiDB and StarRocks meet only if something writes Iceberg. None of the three does that for OLTP data. Loams does it for collections and tables (§08), which is why D279 puts the bridge in Loams.

4. Transactions in SmartEngine, verified in the sourceCopy Anchor Link

Read on 2026-10-01 from wesql/storage/smartengine/ and wesql/mysql-test/suite/smartengine/. The spike's observation (commit, rollback and FOR UPDATE work, §23 §9.2) holds. This section says exactly what is behind it. Nothing below was run beyond the spike and the recorded MTR results; where a conclusion depends on running code, it says (verify).

4.1 What is thereCopy Anchor Link

4.2 What is not thereCopy Anchor Link

The two engines the owner pointed at, for comparison. TiDB has optimistic and pessimistic transactions over TiKV's Percolator-style 2PC, snapshot isolation and a RR that is snapshot isolation, no SERIALIZABLE (it accepts the syntax and treats it as RR; verify against docs.pingcap.com/tidb/stable/transaction-isolation-levels), foreign keys from v6.6.0 (GA in v8.5.0), and distributed transactions. StarRocks has no OLTP transactions to speak of: it is an OLAP engine with load transactions (docs.starrocks.io; verify the current wording). Neither is a reference for SmartEngine's locking. TiDB is the reference for foreign keys (§5), because it implements them above a store that, like SmartEngine, has point locks and no gap locks.

4.3 What is proposed (D274)Copy Anchor Link

• Document the matrix above in the product docs and in WeSQL's deployment notes, and gate every claim in them on a test.
• T1, smartengine_lock_wait_timeout default. One second is a bench default, not an OLTP default. The deployment sets it (for example 50 s, as InnoDB) and sets smartengine_deadlock_detect = ON.
• T2, savepoints. The engine already has SetSavePoint/RollbackToSavePoint per statement. Implementing ROLLBACK TO SAVEPOINT with writes on top is plausible (estimate: small, one PR), and ORMs use savepoints for nested transactions (Django atomic(), Hibernate). It is the first item after W1 (Q276).
• T3, SERIALIZABLE. Not planned. Apps that need it use explicit FOR UPDATE, and the matrix says so.
• T4, user XA. Not planned; Loams needs the internal 2PC only.
• W1 depends on one transaction property: a locking read that does not validate the snapshot (a current read). The engine has it: TryLock skips ValidateSnapshot when validate_snapshot is false or no snapshot exists (core/transactions/transaction_impl.cc ~line 590). Whether ha_smartengine can request it for an arbitrary key without going through a row read needs a short spike (verify, §11 W1).

5. W1: foreign keys (D275)Copy Anchor Link

5.1 Why not inside SmartEngineCopy Anchor Link

The fork's PR 2 listed what an in-engine implementation needs: the capability flag, checks in write_row/update_row/delete_row, recursive actions from inside the handler with a depth limit, and binlog behaviour for cascades. That is InnoDB's design, and InnoDB's code is not in this tree. Two facts favour the SQL layer instead:

• MySQL 8.0 already prelocks foreign-key tables. process_table_fks (mysql/sql/sql_base.cc ~line 4394) adds the parent and child tables of a DML statement to its prelocking set (for NO ACTION and RESTRICT rules it opens the other side for checks, for CASCADE, SET NULL and SET DEFAULT it opens it for writes). So the other table is already open and metadata-locked when the statement runs. Cascades from a handler would need that machinery anyway.
• The data dictionary stores foreign keys for any engine that sets HTON_SUPPORTS_FOREIGN_KEYS (mysql/sql/sql_table.cc ~line 8253), and PR 2 checked that the SQL layer's DDL paths need no engine help.

So the engine gets the capability flag, and the enforcement sits in the handler wrappers (handler::ha_write_row, ha_update_row, ha_delete_row in sql/handler.cc), which every DML path goes through (INSERT … ON DUPLICATE KEY UPDATE, REPLACE, LOAD DATA, multi-table UPDATE and DELETE). The code lives in the fork's overlay (sql/wesql_fk.{h,cc}) with a small hook in handler.cc in the patch, as wesql_ddl_compat does today. It is active only for tables whose engine carries a new WeSQL flag, and only when wesql_enforce_foreign_keys = ON. With the option OFF, PR 93's strip-with-warning behaviour is unchanged (and serverless_honor_innodb_engine from PR 2 remains the escape hatch for apps that cannot wait).

5.2 The reference: how TiDB does itCopy Anchor Link

Read from tidb/docs/design/2022-06-22-foreign-key.md and tidb/pkg/executor/foreign_key.go; the user docs say the feature arrived in v6.6.0 and is GA from v8.5.0 (docs.pingcap.com/tidb/stable/foreign-key/).

5.3 The W1 designCopy Anchor Link

1. Capability. SmartEngine sets HTON_SUPPORTS_FOREIGN_KEYS only when wesql_enforce_foreign_keys = ON at server start (read-only variable). DDL then keeps the FOREIGN KEY clause instead of stripping it (PR 93's strip_foreign_keys is skipped).
2. Checks are current reads. InnoDB's checks read the latest committed parent, not the transaction's snapshot, and so must W1's: a child insert under RR whose parent was created after the snapshot must succeed. The probe therefore takes the lock with snapshot validation off (§4.3), so a parent that changed after the snapshot is not a 1213 error. Which handler call carries that choice is the W1 spike's first question.
3. Locks.
   • Child insert or update of FK columns: a shared lock on the matched parent's primary-key row, taken before the child row is written. If the FK references a unique secondary key, the probe resolves the key to the parent's primary key and locks that row, so the lock is always on the primary-key row (the row that a delete or an update of the referenced columns locks exclusively first).
   • Parent delete, or update of referenced columns: the exclusive lock on the parent row that the statement takes anyway, then a current-read scan of the child FK index prefix, then the referential action.
   • Why that is enough. Child insert and parent delete both go through the parent's primary-key row lock, so they serialize. Two parent deletes of different rows do not interact. Phantom children are impossible once the parent row is locked, because a new child must first take a shared lock on that row. No range lock is needed (there are none to take).
   • What it does not give: nothing here makes SmartEngine serializable for other predicates (§4.2); it makes the FK invariant hold.
4. Timing. Checks are immediate, per row (InnoDB semantics, not deferred to commit), so an application sees errors 1452 and 1451 at the offending statement, as MySQL clients expect.
5. Referential actions run as nested operations on the child table through the same handler wrappers, so the cascaded row changes are ordinary row operations: they take locks, run FK checks of their own, and appear in the binlog. Depth is limited to 15 as in InnoDB (ER_FK_DEPTH_EXCEEDED), with cycle detection through a per-statement set of (table, primary key).
6. The binlog contract (the part InnoDB does differently). With InnoDB and row-based logging, cascaded changes are not logged as rows; a replica re-derives them by enforcing the same FKs. That breaks the binlog bridge (D154, D279): a consumer that sees only the logged rows would never see cascaded deletes, and its collection and Iceberg table would diverge from the database. So in W1:
   • cascaded rows are logged as row events inside the parent statement's transaction;
   • applier threads (replica, fork PR 3's replay, the W3 replica) do not enforce or cascade: they run with FK enforcement off, since the logged rows already contain everything. This is a deliberate difference from InnoDB. It means a WeSQL binlog applied by InnoDB would double-apply cascades; Loams does not do that (§23 §14, no mixed replication).
7. DDL.
   • ADD FOREIGN KEY on an existing table validates under a metadata lock with the check query above (a COPY-style path), failing with 1452/3780 as MySQL does. Concurrent DML is blocked for the duration, which is the cost of not having TiDB's online schema states (estimate: minutes for ten million rows; the deployment target is app metadata schemas, §5.6).
   • Dropping or renaming a referenced table or column, truncating a parent, and type changes of FK columns follow MySQL's existing dictionary rules (3730, 1553, 3780).
   • foreign_key_checks = 0 skips checks and cascades for the session, for dumps and migrations.
8. Bulk load. LOAD DATA checks per row like any statement. Loads that need speed use foreign_key_checks = 0 and a validating ALTER afterwards.

5.4 Risks specific to W1Copy Anchor Link

5.5 Interim: serverless_honor_innodb_engineCopy Anchor Link

PR 2 (merged) keeps an explicit ENGINE=InnoDB table on InnoDB, with real FKs, at the cost that InnoDB tables are copied whole into each snapshot and live on the local volume (PR 2's trade-off notes). It is the supported path for small relational metadata schemas until W1, and after W1 for anything W1 does not cover. It does not help W2: an InnoDB table is not on the bucket table by table.

5.6 ScopeCopy Anchor Link

W1 is for application metadata schemas (Forgejo, Matomo-style apps, ORMs): thousands to millions of rows, modest write rates. It is not for bulk data with FK-heavy fan-out. That is the same scope PR 2 gave the InnoDB option.

6. W2: durable commits in a Loams WAL quorum (D276, D277)Copy Anchor Link

6.1 The gapCopy Anchor Link

Q50: a commit after the last snapshot is lost when the local volume goes. Fork PR 3 fixed the crash case: recovery restores the last consistent snapshot, downloads the archived binlog and replays the window after the snapshot (Consistent_recovery, the temporary wesql_snapshot_replay channel). It cannot recover what was never uploaded. The binlog archive thread uploads slices about once a second (§23 §9.2), so on volume loss the most recent second or so of acknowledged commits is gone, and the spike lost a row written 20 s before the kill, which PR 3's analysis traced to the missing replay and not to the upload cadence. After PR 3 the remaining loss window is the archive slice interval (estimate: up to a few seconds). W2 closes it to zero.

6.2 Which log is shipped (D276)Copy Anchor Link

The binlog is shipped. SmartEngine's WAL stays local and keeps its job: process-crash recovery with the volume intact. In this mode smartengine_flush_log_at_trx_commit can be 0 or 2 (estimate: it removes one local fsync from the commit path), but that is an optimization behind its own test, not the default (§6.3, recovery of prepares). W2 runs with gtid_mode = ON (with enforce_gtid_consistency), for the replay channel of fork PR 3 and for the W3 and W4 consumers. mysql.gtid_executed is not the recovery marker. That gtid_executed is on SmartEngine's supported system tables (se_is_supported_system_table) only says which engine stores the table; it does not say when rows are written. With binary logging on, MySQL 8.0 writes a transaction's GTID to the table at commit only if the engine persists it itself (THD::se_persists_gtid(), sql/handler.cc ~line 1577), and only InnoDB does (Clone_persist_gtid, storage/innobase/clone/clone0repl.cc); for every other engine the GTIDs reach the table when the binlog rotates or the server shuts down (Gtid_state::save_gtids_of_last_binlog_into_table; dev.mysql.com, "mysql.gtid_executed Table"). The fork's patch moves the table to SmartEngine and does not change that path. So after a power loss SmartEngine can hold a transaction whose GTID is not in the table, and replaying "every GTID not in gtid_executed" would apply it twice. W2's marker is defined in §6.3.

6.3 The commit pathCopy Anchor Link

MySQL's ordered commit has a flush stage (write the transactions' binlog caches to the file), a sync stage (fsync the file, per sync_binlog) and a commit stage (engines commit). The engine's prepare (se_prepare) comes before all three, and process_flush_stage_queue, sync_binlog_file and process_commit_stage_queue are the functions in mysql/sql/binlog.cc (lines 8515, 8716, 8571).

 client COMMIT
   │ SE prepare (named transaction; local; not yet visible)
   ▼
 flush stage      leader writes the group's binlog caches to the local binlog file
   ▼
 sync stage       NEW: leader sends the group's bytes (whole transactions, contiguous) to the
                  quorum client; waits until a majority of acceptors has them durable
                  (optionally also fsyncs locally, sync_binlog=1, off by default in this mode)
   ▼
 commit stage     SE commit for every transaction in the group; ack to clients

• One round trip per group, not per transaction. The leader of the group already batches, so the RPC rate follows group commits.
• The append unit is a flush-stage batch of whole transactions. Acceptors therefore hold transaction-aligned prefixes.
• LSN is a virtual stream offset: the cumulative number of binlog bytes since the stream's origin, across files. The (file, position) of the MySQL world maps to it through the Rotate events that are part of the stream; the client keeps the map for each file's start.
• Failure of the quorum: the sync stage cannot complete. The group's transactions are prepared in the engine but not committed, and are not acknowledged. After a timeout the server goes read-only (super_read_only = ON) and reports the failure; it does not acknowledge a commit that is not on a majority.
• Reconciliation at start. Before serving, the server runs the equivalent of sync-safekeepers: it takes a new term (a vote round), learns the majority's highest flushed offset, and then:
  • local binlog ahead of the quorum (a crash between the local write and the quorum ack): the extra tail holds only transactions that are prepared in the engine and were never acknowledged. If it is the same lineage (the server holds the highest term), the tail is re-proposed; if another primary was elected meanwhile, the tail is truncated and MySQL's own crash recovery rolls the prepared transactions back, because they are not in the binlog;
  • local binlog behind the quorum, or empty (volume lost): the missing bytes come from the acceptors (and the bucket, below) into the replay window of fork PR 3, and Consistent_recovery applies them before the server opens for clients.
• The applied-offset marker. Each commit writes one extra key, wesql_wal_applied = (term, end offset) (the stream offset just past the transaction's XID event, with the term that appended it), into the transaction's own SmartEngine write batch, in a reserved system subtable, at se_commit. The marker and the data are therefore in one engine WAL record: after any crash, power loss included, both survive or neither does. W2 requires binlog_order_commits = ON, so engine commits run in stream order and the engine WAL holds them in that order; a lost WAL tail drops a suffix of commits, never a commit in the middle. The committed engine state after a crash is therefore exactly the binlog prefix up to the surviving marker M. The marker travels in the engine snapshots, so a restore from the bucket brings its own M; it must agree with the binlog position PR 3 records for the snapshot, and recovery fails closed if they disagree. When W2 is first enabled on a database, a fresh snapshot is taken and its position is the initial M.
• Recovery uses the marker, not prepares and not mysql.gtid_executed. In W2 mode, XA recovery rolls back every recovered engine prepare, whether or not its XID is in the binlog, and the replay channel of fork PR 3 then applies every binlog transaction in (M, quorum end] before the server opens for clients. Rolling back is safe because a prepare is not visible and was not acknowledged unless it is on the quorum, in which case replay re-applies it from the binlog. commit_by_xid is not used: a lazily flushed engine WAL (flush_log_at_trx_commit 0 or 2) can keep a later transaction's prepare and lose an earlier one's (prepares are not in binlog order within a group), so committing recovered prepares would leave gaps below the highest committed offset. The rollback keeps the state an exact prefix. Replayed transactions commit through the same path and write the marker, so a crash during replay restarts from the new M: replay is idempotent at transaction granularity without replica_exec_mode = IDEMPOTENT, which would hide real errors. Recovered prepares whose XIDs are not on the quorum (the local tail above) are rolled back in the same step. W2 moves the meaning of "in the binlog" from "in the local file" to "on a quorum".
• Rejected alternative: make SmartEngine persist GTIDs at commit as InnoDB does (SE_GTID_PERSIST and a GTID in the commit record). It is more patch in the SQL layer's GTID code, and a GTID set still has to be compared with the binlog transaction by transaction. One offset per commit says the same thing for an ordered stream.
• Required acceptance tests (§11 W2, 2b and 2c): with flush_log_at_trx_commit 0, 1 and 2, a simulated power loss that discards the engine WAL's unsynced tail, including the case where SmartEngine's data survives but mysql.gtid_executed lacks its GTIDs (no binlog rotation and no clean shutdown since the commits). Recovery must lose no acknowledged row and apply no row twice.

6.4 The protocol boundary and licensing (D277)Copy Anchor Link

Rules that keep D11 and D148 intact:

• The client copies no Loams source. It is written from the protocol specification. Apache-2.0 code cannot be combined into a GPL-2.0-only work (the FSF treats Apache-2.0 as compatible with GPL-3.0 and not with GPL-2.0), so nothing from loams-safekeeper is pasted or translated line by line into the fork. Where Neon's walproposer C code (PostgreSQL License, permissive and GPL-compatible) would help, it is read as a specification, not copied, to keep provenance simple (verify with the owner's counsel, Q271).
• The acceptors do not link anything GPL. They speak the protocol over a socket. A separate program communicating over a socket is not a combined work under the usual reading of the GPL (verify, Q271).
• WeSQL is still never linked into Loams, so Loams's code stays Apache-2.0 (D11). The fork's source is published as GPL-2.0 requires, from ostrium-labs/wesql.
• Dependencies of the client are limited to what the MySQL tree already carries (OpenSSL, zstd, protobuf if used), all GPL-2.0-compatible.

Transport and authorization (required, not optional). Term fencing orders writers; it does not say who may write. So:

• Mutual TLS. The client verifies the acceptor's certificate chain against a configured CA and matches the endpoint's identity (the configured name or SAN); it refuses the connection if either check fails. Acceptors require a client certificate. No plaintext mode, including on a cluster network (the rule §23 §6.3 sets for the MySQL hop).
• Per-stream authorization. The client certificate's identity (or a short-lived token bound to it) names the (ns, db) stream it may propose to. The acceptor checks that binding before it processes START_WAL_PUSH, votes or appends for that stream, and refuses other streams. The existing global --auth-token and --trusted-network options of loams-wal (§28 §7.2) do not bind a caller to a stream and are not enough for this kind (Q279).
• Read side. The replica and the bridge authenticate the same way, with read-only rights on the stream.
• Credentials come from the unified auth plan (Q30), rotate without a restart, and are never in the binlog or the metastore (§23 §6.2).

6.5 The bucket copy and trimCopy Anchor Link

The acceptors offload committed bytes to the bucket (D269: pgwal/… objects for Postgres; for mysql-binlog a sibling prefix mysqlbinlog/<ns>/<db>/<begin>-<end>.lwal with the same segment conventions). Two logs then exist in the bucket: the acceptors' offload, and the fork's own binlog archive slices (binlog_archive). W2 keeps the fork's archive (it feeds PR 3's recovery and the existing replica path) and lets the acceptors' offload serve as the tail between the last archived slice and the quorum's end. Trim follows D269's rule: a journal segment is recycled once every timeline in it has passed min(backup_lsn, remote_consistent_lsn, commit_lsn), where for WeSQL remote_consistent_lsn is the fork's archive position, reported over the same feedback message the pageserver uses. Whether to keep two archives or let the acceptors' offload replace the fork's is Q272.

6.6 Cost (estimate)Copy Anchor Link

The commit adds one quorum round trip to the path that today is a local binlog fsync. §28 §7.2's Arm A is designed to be at or better than the safekeepers' commit latency (which §28 §6.1 estimates at 1–3 ms p50 across AZs, an estimate, not a measurement); the Arm A gate runs are PRs #165 and #166, open when this was written, so this document treats Arm A's numbers as pending (verify). Arm B (TiKV on the commit path) failed the laptop gate in §28 §7.1 and is not an option here. W2's target is a p99 commit within the same budget as Loams Postgres (the owner's < 5 ms p99 target, which is a server-hardware number, §28 §7.1). MySQL group commit amortizes the round trip across concurrent clients, so a single-client INSERT loop pays it in full and a 16-client one does not.

7. W3: failover on the same quorum (D278)Copy Anchor Link

Upstream removed Raft HA on 2026-08-22 (§23 §4.2); the single-node WeSQL has no leader election. W3 rebuilds failover without bringing consensus back into MySQL: the quorum is the acceptors', the fencing is theirs, and MySQL stays single-writer.

7.1 RolesCopy Anchor Link

7.2 FencingCopy Anchor Link

• Terms are the fence. An acceptor refuses an append from a term lower than the one it has promised (§28 §7.2, D264). The term is made durable in the acceptor's TiKV metadata before the vote is acknowledged (D268). A deposed primary's next sync stage therefore fails, its group is not acknowledged, and it sets super_read_only and exits.
• No acknowledged commit can be lost to a split brain. A commit needs a majority at its term, and a new primary's election needs a majority at a higher term, so the two majorities intersect (the Paxos argument that walproposer already relies on, §28 §6.1).
• Stale reads from a deposed primary are not covered by terms. The primary keeps a lease: it sends a heartbeat (an empty append) every T/3, and if no majority has answered for T it sets itself read-only. The control plane waits T plus a margin after the old primary stops answering before it routes to a new one. T is a deployment setting (estimate: 5–10 s; failover time is bounded below by it unless the old primary is known dead).
• The router follows the record. Wire routing (§23 §6.3, D153) reads x/<ns>/<db>; after the compare-and-set the router sends new connections to the new primary and existing ones to the old one fail. Proposed 2026-10-01 (D320, §31 §9.1): where vtgate fronts WeSQL, the repoint is vtctldclient TabletExternallyReparented <new-primary-tablet> for the shard, beside the x/ record change.

7.3 PromotionCopy Anchor Link

1. Decide. The control plane marks the primary suspect (missed lease renewals in TiKV, a failed health check, an operator action) and picks the replica with the most applied offset.
2. Fence and elect. The candidate runs a vote round at term n + 1 with a majority of acceptors and learns the highest flushed offset (the sync-safekeepers step of §6.3). From here the old primary cannot commit.
3. Catch up. The candidate applies the binlog up to that offset: what its relay log already holds, plus the tail read from the acceptors (START_REPLICATION). The replica read path queues events into the relay log with the function PR 3 already uses (queue_event_from_objstore).
4. Open. read_only and super_read_only go OFF, the status variable Wesql_ready_for_write (fork PR 4) turns ON, and the candidate starts proposing at term n + 1.
5. Repoint. The x/ record changes primary and term by compare-and-set. The saga's steps are idempotent per term, so a crashed control plane re-runs from the record.

7.4 Replica bootstrap and steady stateCopy Anchor Link

• Cold replica (no local volume): restore the newest consistent snapshot from the bucket (the same code as PR 3's recovery), read the archived binlog slices after it, then switch to the acceptors' tail. Initialization from a clean bucket took 75–90 s in the spike (§23 §9.2, spike), with replay time on top (estimate: proportional to the binlog since the last snapshot; snapshot frequency is the lever).
• Warm replica keeps applying continuously, so promotion is election plus a short catch-up (estimate: seconds, bounded by the lease T).
• Snapshots continue to be taken by the primary only (Consistent_archive); replicas do not write to the bucket.
• Replica reads are eventually consistent. A client needing read-your-writes reads the primary. A consistency token (D76) carrying the binlog offset can let a replica wait until it has applied that offset (Q277).

7.5 What W3 does not doCopy Anchor Link

No multi-primary, no automatic split of read and write at the SQL layer (that is a router job, later), and no cross-region failover (acceptors are one region's AZs, as in §28).

8. W4: analytics through Iceberg, not a StarRocks port (D279)Copy Anchor Link

8.1 The bridge (extends D154)Copy Anchor Link

D154 already specifies that the row binlog becomes DocOps on a collection's implicit stream with an idempotent producer. W4 adds the Iceberg half and says how the pieces connect:

 WeSQL ──binlog (ROW, FULL)──► bridge task ──► changelog stream per table (+I, -U, +U, -D; D21)
                                                  ├─► link ──► collection (search; D154's DocOps)
                                                  └─► link ──► keyed Iceberg table in Lakekeeper (RustFS)
                                                                 ▲                      ▲
                                                                 │                      │
                                                    StarRocks external catalog   Loams analytics (DataFusion)
                                                    (sidecar, reads in place)    Trino, DuckDB, Spark, …

• Source. Before W2, the bridge reads from the primary as a replica (COM_BINLOG_DUMP, D154's N6). After W2, it reads from the acceptors (the same START_REPLICATION as W3's replica), so the bridge no longer loads the primary.
• Sequence and exactly-once. Producer id = (database, bridge), sequence = the virtual stream offset of §6.3 plus the ordinal inside the transaction. This is D154's (file index, position) made global, and it is the same contract as D129.
• Mapping. Each table maps to a keyed table by primary key. Insert, update and delete become changelog records; the link applies them as upserts and deletes to the Iceberg table with merge-on-read deletion vectors (§03, §08 §1). Cascaded deletes appear because W1 logs them (§5.3, 6); without that property, this bridge would silently diverge, which is a reason to build W1 before relying on the Iceberg copy for tables with FK actions.
• DDL. Binlog query events carry DDL. The bridge applies additive schema changes (new nullable column) to the Iceberg schema and halts the table's link on anything else, for the operator, as a changelog from Debezium-style tools usually does.
• Initial copy, with an exact boundary. START TRANSACTION WITH CONSISTENT SNAPSHOT alone does not say which binlog offset matches the snapshot: a commit can land between the snapshot and a later read of the position. The bridge therefore takes the boundary the way mysqldump --single-transaction --source-data does (dev.mysql.com, mysqldump): under a brief FLUSH TABLES WITH READ LOCK (which blocks commits through the global commit lock; verify that it blocks SmartEngine commits), it runs START TRANSACTION WITH CONSISTENT SNAPSHOT (REPEATABLE READ, the only level that takes a snapshot, §4.1), reads SHOW MASTER STATUS and @@global.gtid_executed (8.0.46 names; SHOW BINARY LOG STATUS is a later release), then UNLOCK TABLES. The copy reads the snapshot; streaming starts at the recorded position, mapped to the stream offset of §6.3, and the GTID set is the cross-check. Producer sequences drop any overlap. The W4 test commits continuously during this handshake and compares checksums.
• Freshness. Seconds (estimate: the link tick plus an Iceberg commit), which is analytics freshness, not transactional.

8.2 The readersCopy Anchor Link

• Loams analytics (§08) reads the table in place, with the hot tier of §04.
• StarRocks, if a deployment wants it, runs as a sidecar: CREATE EXTERNAL CATALOG … iceberg against Lakekeeper's REST catalog on RustFS, and queries the table in place (§3). Loams never links or embeds it. StarRocks is Apache-2.0 (GitHub license API), so the sidecar carries no license concern; whether the Iceberg catalog's deletion-vector reads match Loams's writes is a conformance test (verify, §11 W4).
• Any other Iceberg engine reads the same table (§08 §8).

8.3 Why "implement all StarRocks features on WeSQL" is rejectedCopy Anchor Link

1. Different engines. StarRocks' speed comes from a vectorized C++ executor over its own columnar segments, with a cost-based optimizer and materialized views in a Java frontend. WeSQL's executor is MySQL's row-at-a-time executor over an LSM. Features such as vectorized joins, runtime filters, colocate groups and MPP exchange are not additions to MySQL's executor; they are another executor. That is a rewrite, not a port, in two languages and two licenses of code.
2. Different storage. StarRocks' features assume columnar segments with per-column indexes and delete vectors. SmartEngine's rows are LSM key-value pairs; the layouts do not match, and OLTP and OLAP access patterns pull the LSM's compaction in opposite directions.
3. License. WeSQL is GPL-2.0-only; StarRocks is Apache-2.0. Apache-2.0 code can be taken into a GPL-3.0 work, but not into a GPL-2.0-only one (§6.4). Porting StarRocks code into the fork would need either a separate Apache component (which is the sidecar again) or a re-implementation.
4. Maintenance. StarRocks ships frequent releases. A port would diverge from upstream on day one, and Loams would own the divergence, on top of the MySQL and SmartEngine forks already owned.
5. The cheaper way exists and is already in the design. Iceberg is the contract. OLTP writes stay in WeSQL, analytics reads come from an Iceberg table that StarRocks, Loams, DuckDB or Trino scan in place. That gives the owner the combination in the question (a row engine on the bucket and a columnar engine reading Iceberg on the bucket) with Loams writing the Iceberg half.

9. The alternative: TiDB on our TiKV (D280)Copy Anchor Link

TiDB on Loams's TiKV already has what W1–W3 would add: distributed transactions (Percolator over TiKV), foreign keys (§5.2), and HA (TiKV's Raft and PD, TiDB servers are stateless). It speaks the MySQL protocol, and a TiCDC changefeed is the standard binlog-like bridge. Honestly stated:

The rule (D280). Choose WeSQL only if the requirement is OLTP storage that lives entirely on the bucket: the same property that D1 gives the retrieval engine, a stateless engine over S3 with disks as caches, for apps whose data is modest and whose operators want one durable place. Choose TiDB for anything that needs scale-out writes, more than one write node, distributed transactions across data, or mature HA, and accept that its data sits on TiKV disks.

D260 is in the way, and this document does not remove it. D260 (2026-09-29, owner) says "no TiDB anywhere in the engine or Loams cloud," and that it supersedes D123 (§23's "D123 is paused" is the weaker earlier statement). §23 §14 says WeSQL does not revive D123. If the owner wants TiDB for MySQL, that is a decision on D260 and Q260 (MySQL wire access), made there. W1–W4 are the cost of not reversing it for the one case where bucket-only storage is the requirement. The owner may reasonably conclude that the four milestones cost more than reversing D260 for a single use case; that is Q273.

10. Licensing (D277)Copy Anchor Link

D11 (no copyleft dependencies in Loams) and D148 (WeSQL is a separate process) are unchanged. The two questions that need a lawyer rather than an engineer are in Q271.

11. Milestones and acceptance testsCopy Anchor Link

Each milestone is a small stack of PRs, in the fork (F) and in this repository (L). PR counts are estimates.

Ordering. W1 has no dependency on the Loams side and can start now. W2 waits for the Arm A gate. W3 follows W2. W4's first half can start from D154's N6 before W2.

12. RisksCopy Anchor Link

13. Open questionsCopy Anchor Link

14. Contradictions with earlier decisions, and how they are resolvedCopy Anchor Link

15. SourcesCopy Anchor Link

Read on 2026-10-01.

• WeSQL fork (ostrium-labs/wesql, branch 8.0, eef34f452): README.md; storage/smartengine/handler/{ha_smartengine.cc, se_hton.cc, se_hton.h}; storage/smartengine/plugin/{se_plugin.cc, se_system_vars.cc}; storage/smartengine/transaction/{se_transaction.cc, se_transaction_impl.cc}; storage/smartengine/core/transactions/{transaction_impl.cc, transaction_lock_mgr.h}; storage/smartengine/core/LICENSE; sql/wesql_ddl_compat.cc; docs/mysql-8.0.46-ddl-compat-design.md; patches/mysql-server-8.0.46.patch; mysql-test/suite/smartengine/ (r/unsupported_tx_isolations.result, r/lock_rows_not_exist.result, r/rollback_savepoint.result, r/gap_lock_issue254.result, disabled_smartengine.def, include/locking_issues_case4.inc); merged fork PRs 1–4 (gh pr list -R ostrium-labs/wesql --state merged): #1 path-style, #2 honour InnoDB, #3 replay after the snapshot, #4 readiness.
• MySQL (mysql-8.0.46, 976e5da9): sql/binlog.cc (ordered_commit, process_flush_stage_queue, sync_binlog_file, process_commit_stage_queue), sql/sql_base.cc (process_table_fks), sql/sql_table.cc (HTON_SUPPORTS_FOREIGN_KEYS).
• TiDB (pingcap/tidb at 8936d7b): docs/design/2022-06-22-foreign-key.md; pkg/executor/foreign_key.go (FKCheckExec, FKCascadeExec); pkg/meta/model/table.go (FKInfo, ReferredFKInfo); docs.pingcap.com/tidb/stable/foreign-key/, /tidb/stable/tidb-architecture/, /tidbcloud/tidb-x-architecture/.
• StarRocks: docs.starrocks.io (Architecture; Primary Key table; Catalog overview and the Iceberg catalog page, found by search); StarRocks/starrocks (fe/, be/src/storage/{kv_store.h, tablet_meta_manager.h, persistent_index.cpp, persistent_index.h, local_primary_key_recover.h}; GitHub license and languages APIs).
• Loams: §23 (D148–D157, Q50, Q51, §4.2, §9.2); §28 §6.1, §6.9, §7.1, §7.2 (D263–D271, Q261–Q264); §08, §09, §03, §21 §6.3, §20 §12 (D129); D1, D2, D11, D21, D76, D126, D130, D260; Q260.