Follow one record.
Understand the system.

RockBeeClient exposes put, get, and delete for nonempty opaque byte keys and values. User keys are limited to 65,534 bytes; the complete wire frame must also fit its 1 MiB bound. client.table("products") creates a JSON-table handle sharing the same connection, admission limits, and shutdown lifecycle. It uses composition, not inheritance.

use rockbee_client::RockBeeClient;
use serde_json::json;

let client = RockBeeClient::connect("127.0.0.1:7100").await?;

// Byte values: the application chooses the encoding.
client.put(b"cart:customer-42", b"product-42:2", None).await?;
let cart = client.get(b"cart:customer-42").await?;

// JSON table: generate a UUID, or choose your own ID.
let products = client.table("products");
let id = products.put(&json!({
    "name": "Trail Backpack", "price": 79
})).await?;
let product = products.get(&id).await?;
// Either GET can lag a preceding durable PUT.

put_key(id, document, ttl) replaces a document under a supplied ID. JSON objects can have different fields. Raw keys and table IDs live in separate physical namespaces.

KV API · Table API

A PUT starts eagerly and returns a completion handle. Dropping an admitted handle does not cancel the write. Awaiting success confirms that the writer acknowledged SlateDB durability. Shutdown drains admitted operations, including dropped handles, before disconnecting.

The client and gateway do not automatically replay uncertain mutations. Request IDs correlate responses on a connection; they are not durable deduplication keys. By default, client clones share 128 active and 128 pending slots, with a 5-second end-to-end request deadline.

Client contract · Mutation outcomes

proto/rockbee.proto defines the versioned envelope, requests, responses, and correlation IDs. Prost generates Rust types; the generated file is checked into the repository and included by the protocol crate. Ordinary Cargo builds use that file. Regeneration is an explicit developer command.

4-byte big-endian length | Protobuf Envelope
                          version = 1
                          request_id = positive integer
                          oneof: KV / Table / Status / Response

Maximum frame: 1 MiB, including the length prefix.

This is a custom TCP protocol. Multiplexed requests use positive IDs; zero is reserved for sequential mode. The shared RPC layer bounds connections and in-flight work. Its frame read/write timeouts are separate from client and query execution deadlines.

cargo run -p rockbee-protocol --example generate_protocol

Wire schema · Code generation

Validated mutations take the one writer route exactly once. GET and SQL requests go to readers in round-robin order, with reader failover inside one routing budget. A successful missing-value response is final; the gateway does not ask another reader to search for a newer value.

Readers use SlateDB’s read-only DbReader, replay WAL data, and refresh their view. The checked-in poll interval is 10 seconds. That is a polling setting, not a visibility guarantee. Lower intervals trade extra object-store requests for more frequent refresh.

The gateway has no separate KV cache. Each writer or reader has SlateDB block and metadata caches; SQL adds a reader-local Parquet file cache. Cold requests can fetch durable objects, while warm requests may use the working set already present.

Router · Reader · Reader configuration

Rockbee keeps opaque byte keys and JSON tables in the same SlateDB database, under disjoint physical key prefixes. A table name is an ASCII identifier of 1–64 bytes; a document ID is UTF-8 and may be up to 1,024 bytes. A table value must be a JSON object. A table PUT writes the document and a durable catalogue entry in one SlateDB batch, so the exporter can discover the table after a restart.

raw key       = [0, ...]
table prefix  = [1, table_name_length, ...table_name_bytes]
catalogue key = [2, ...table_name_bytes]

Export objects are deliberately outside the SlateDB data paths. The table name is hex encoded, preserving distinct case-sensitive names even when the object store is a local filesystem.

<prefix>/_rockbee_tables/<hex-table-name>/
  segments/<uuid>.parquet
  manifests/<20-digit-revision>.json

table limits · table key encoding · durable table PUT · object layout

A writer-owned background worker checks registered tables every 500 ms. A table’s first export, a compaction, or a detected WAL gap builds a new base generation. This work is independent of the mutation acknowledgement: the client receives success only after SlateDB durability, not after Parquet publication.

1. Flush the writer memtable and create a detached SlateDB checkpoint with a 24-hour lease.
2. Scan that pinned checkpoint for the current rows of one table, writing bounded Parquet batches.
3. Record the checkpoint’s WAL high-water mark, then consume the committed WAL tail through EOF.
4. Publish the complete generation only after all of its segments exist.

The manifest cursor is exported_wal_file_id. It advances only after a whole WAL file has reached EOF; uploaded partial batches are not made visible on their own. A missing/truncated WAL causes a fresh pinned snapshot rather than skipping history.

export constants · worker lifecycle · snapshot and tail reconciliation · WAL cursor rule

Each Parquet row contains the document ID and JSON value plus the ordering and visibility metadata needed to reconstruct current table state: seq, absolute expiry, tombstone, WAL file ID, and row ordinal. Segment writes use create-only object-store operations. Only afterwards can the worker publish a new create-only manifest revision.

{
  "format_version": 1,
  "revision": 42,
  "writer_epoch": 9,
  "exported_wal_file_id": 1234,
  "segments": [{ "path": "…/segments/uuid.parquet", "bytes": 1048576 }]
}

Publication checks the active SlateDB writer epoch, requires the expected preceding revision, and rejects a cursor that would regress. The result is a complete immutable generation or the preceding published generation; a failed upload or publication does not make partial state queryable.

manifest schema · segment upload and publication · writer-epoch fence

A reader reloads the table manifest for every SQL query. It checks that every referenced segment stays inside that table’s object prefix, downloads any cache miss into a reader-owned temporary directory, verifies its manifest size, and gives DuckDB those local file paths. DuckDB does not directly read S3 or SlateDB for a query.

Before your SQL predicates run, Rockbee builds the selected table view by selecting the newest row for each id. The ordering is exact:

row_number() OVER (
  PARTITION BY id
  ORDER BY seq DESC, wal_file_id DESC, row_ordinal DESC
)

Only rank 1 remains; then its tombstone and expiry are evaluated. This ordering matters: an older live value cannot reappear when the newest record is a delete or has expired.

manifest reload and execution · local download cache · deduplication and visibility view

A table delete is a durable SlateDB tombstone. The exporter preserves it as a Parquet row with tombstone: true; queries remove it only after deciding that it is the newest row for that ID. This keeps a historical pre-delete value from resurfacing.

TTL is stored as an absolute Unix-millisecond expiry. ttl_millis = 0 means no expiry. At the exact expiry instant a point read is invisible, and query SQL uses the equivalent condition expires_at_unix_ms > now. A later non-expiring PUT replaces an earlier expiry because it is the newest record.

ttl_millis == 0  // Ttl::NoExpiry
ttl_millis > 0   // Ttl::ExpireAtMillis(now + ttl_millis)

table TTL and delete durability · wall-clock TTL semantics · tombstone export

Each query gets an in-memory DuckDB connection and a private spill directory. External access, persistent secrets, and extension auto-loading are disabled. SQL must be one SELECT; DuckDB parses it into a syntax tree, and Rockbee permits only the selected unqualified table and lexical CTEs. Expressions, joins over CTEs, and aggregates still run in DuckDB. Allowed table functions are range, generate_series, unnest, json_each, and json_tree; file scans, other tables, and catalogue access are rejected.

The reader configuration can change timeout, DuckDB memory (1–16,384 MB), and concurrency (1–128). It does not make query state durable or turn the cache into an authority.

reader defaults and configurable limits · DuckDB isolation and result bounds · SQL source restrictions

On writer restart, Rockbee detects registered tables and restarts export even without another mutation. The exporter retries transient object-store failures, but a newer SlateDB writer epoch fences the old exporter. On normal shutdown it cancels and joins the exporter before closing the database; owned checkpoint cleanup is attempted on completion, failure, and cancellation.

Compaction creates a new snapshot generation after at least 128 delta segments and 60 seconds. It replaces what the current manifest references, preserving the same visibility rules. Manifest revisions and unreferenced Parquet objects are currently retained; automatic garbage collection is not implemented.

SQL is eventually consistent with durable writes. A registered table with no first manifest returns a warming error. A lagging reader may not yet see the catalogue entry and can treat the table as empty. The returned exported_wal_file_id describes the export generation used; it is not a write receipt or a cross-reader read barrier.

restart resumes export · writer shutdown · compaction and WAL-gap recovery · warming behavior

The selected table exposes id VARCHAR and value JSON. Your SELECT shapes each result row. SQL reads the published projection, so this example result applies after the product is visible in that generation.

let result = products.query(
    "SELECT id, value->>'name' AS name,
            TRY_CAST(value->>'price' AS DOUBLE) AS price
     FROM products
     WHERE TRY_CAST(value->>'price' AS DOUBLE) < 100
     ORDER BY price"
).await?;

for row in result.rows {
    println!("{row}");
}

{"id":"product-42","name":"Trail Backpack","price":79.0}

The client returns JSON row values plus exported_wal_file_id. Results are bounded and materialized; this API does not return a streaming database cursor. The watermark describes the exported view, not confirmation that a particular recent write is present.

Query client · Result representation

Writer and reader processes share the same backend, bucket or directory, and prefix. Current backend constructors cover S3, LocalStack, and local files. Native GCS and Azure backend configuration is not implemented. The gateway is configured with one writer and a list of readers.

# Reader-side SQL configuration
[query]
timeout_ms = 4000
memory_limit_mb = 256
max_concurrent = 2

[reader]
poll_interval_ms = 10000

Cache capacities are not preallocated memory or total process RSS ceilings. Query memory is a DuckDB setting per query; concurrent queries and other process work consume additional resources.

Reader config · Writer config · Backend construction

rockbee-cdc tails durable SlateDB WAL changes into JSON lines. It flushes output before advancing a locked, source-bound cursor. Crashes may repeat events, so consumers need at-least-once handling. TTL expiry does not emit a delete event; engine-private table and catalogue keys are filtered from this raw KV stream.

make local starts an owned writer, reader, and gateway on dynamic ports with a unique storage prefix. It prints connection details. Fixed development configs use gateway 7100, writer 7101, and reader 7102. Start the writer before the reader.

--check-config reports effective configuration without proving connectivity. A status probe reports endpoint role and local RPC counters; it does not prove storage health or reader freshness.

Reader request failover is implemented. Automatic writer failover and write sharding are not. The public transport is plain TCP without built-in authentication or TLS; deployments must supply the required network boundary. SQL path restrictions are not a tenant-authentication system.

CDC contract · Development & operations

The repository contains durability, lost-acknowledgement, reader convergence, SQL update/delete/expiry, and publication recovery tests. Local fixtures establish those behaviors under their test conditions; they do not establish production throughput or linear scaling.

Storage behavior tests · Query tests · Gateway failure tests