diff --git a/PLAN.md b/PLAN.md index 3b39bb5..1247488 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,11 +42,11 @@ encryption, retention, backup, and deployment policy remain host concerns. ## Slice 4 — authoritative reconnect and convergence - [ ] **User value:** offline and concurrent work reconnects without duplicate mutation, silent loss, stale authorization, or ambiguous conflict. -- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease plus a preferred-owner hint so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Broader conflict decisions, backpressure, upgrade mid-queue, and auth isolation remain. +- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Each activation serializes uploads with one in flight, processes at most two acknowledged commands, exposes the retained durable count when backpressured, and resumes the next bounded run only through the visible retry action. Broader conflict decisions, upgrade mid-queue, and auth isolation remain. - **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics. - **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy. - **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`. -- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_converges_without_losing_local_intent -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit with acknowledged removal, then a divergent canonical update producing explicit conflict while the local command and prior committed snapshot/cursor remain intact. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. The completed slice proof must additionally cover broader partial reject/conflict decisions, backpressure, upgrade mid-queue, and multi-user isolation. +- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_converges_without_losing_local_intent -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit with acknowledged removal, then a divergent canonical update producing explicit conflict while the local command and prior committed snapshot/cursor remain intact. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. The completed slice proof must additionally cover broader partial reject/conflict decisions, upgrade mid-queue, and multi-user isolation. ## Slice 5 — local-first multiplayer Kanban milestone diff --git a/examples/kanban/static/sync.js b/examples/kanban/static/sync.js index 9802067..3ea6d8c 100644 --- a/examples/kanban/static/sync.js +++ b/examples/kanban/static/sync.js @@ -8,9 +8,14 @@ const LEASE_MS = 5000; const LEASE_POLL_MS = 100; const LEASE_KEY = "uploaderLease"; let database; +let uploadLimit; let retryTimer; let leaseTimer; let synchronizing = false; +let uploadsThisRun = 0; +let uploadedTotal = 0; +let inFlightUploads = 0; +let maxObservedInFlight = 0; let stopped = false; class UploadError extends Error { @@ -168,6 +173,30 @@ async function upload(command) { throw new Error("sync retry limit exhausted"); } +function beginUpload() { + inFlightUploads += 1; + maxObservedInFlight = Math.max(maxObservedInFlight, inFlightUploads); + root.setAttribute("data-sync-in-flight", String(inFlightUploads)); + root.setAttribute("data-sync-max-observed-in-flight", String(maxObservedInFlight)); +} + +function finishUpload() { + inFlightUploads -= 1; + root.setAttribute("data-sync-in-flight", String(inFlightUploads)); +} + +async function continuePendingWork() { + const commands = await pendingCommands(database); + root.setAttribute("data-sync-pending-count", String(commands.length)); + if (commands.length === 0) return; + if (uploadsThisRun >= uploadLimit) { + root.setAttribute("data-sync-manual-retry", "available"); + setPhase("backpressured", `Upload limit ${uploadLimit} reached; ${commands.length} durable command${commands.length === 1 ? " remains" : "s remain"} queued. Retry now to continue.`); + return; + } + setTimeout(() => synchronize(validatePending(commands[0])).catch(failPermanently), 0); +} + async function synchronize(command) { if (synchronizing) return; synchronizing = true; @@ -188,7 +217,13 @@ async function synchronize(command) { root.removeAttribute("data-sync-manual-retry"); setOnline(navigator.onLine); try { - const acknowledgement = await upload(command); + beginUpload(); + let acknowledgement; + try { + acknowledgement = await upload(command); + } finally { + finishUpload(); + } setOnline(true); root.setAttribute("data-sync-upload-sequence", String(acknowledgement.serverSequence)); setPhase("awaiting-ack", `Command ${command.id} uploaded; awaiting canonical acknowledgement.`); @@ -203,8 +238,13 @@ async function synchronize(command) { source.addEventListener("acknowledgement", async (event) => { const canonical = JSON.parse(event.data); if (canonical.commandId !== command.id) return; + source.close(); root.setAttribute("data-sync-pending-before-ack", String((await pendingCommands(database)).length)); await removeAcknowledged(database, command.id); + uploadsThisRun += 1; + uploadedTotal += 1; + root.setAttribute("data-sync-uploaded-this-run", String(uploadsThisRun)); + root.setAttribute("data-sync-uploaded-total", String(uploadedTotal)); root.setAttribute("data-sync-pending-count", String((await pendingCommands(database)).length)); root.setAttribute("data-sync-ack-sequence", String(canonical.serverSequence)); root.setAttribute("data-sync-canonical-column", canonical.canonicalColumn); @@ -214,7 +254,7 @@ async function synchronize(command) { clearTimeout(leaseTimer); await releaseUploaderLease(database); root.setAttribute("data-sync-leader", "false"); - source.close(); + await continuePendingWork(); }); source.addEventListener("snapshot-required", async (event) => { const missing = JSON.parse(event.data); @@ -271,20 +311,32 @@ async function runLeaseLoop(command) { async function start() { if (!root) return; + uploadLimit = Number.parseInt(root.getAttribute("data-sync-upload-limit"), 10); + if (!Number.isSafeInteger(uploadLimit) || uploadLimit < 1) throw new Error("data-sync-upload-limit must be a positive integer"); database = await openLog(); const commands = await pendingCommands(database); + root.setAttribute("data-sync-uploaded-this-run", "0"); + root.setAttribute("data-sync-uploaded-total", "0"); + root.setAttribute("data-sync-in-flight", "0"); + root.setAttribute("data-sync-max-observed-in-flight", "0"); root.setAttribute("data-sync-pending-count", String(commands.length)); if (commands.length === 0) { setPhase("idle", "No pending commands."); return; } const command = validatePending(commands[0]); - root.addEventListener("click", (event) => { + root.addEventListener("click", async (event) => { if (!event.target.closest("[data-sync-retry]")) return; - synchronize(command).catch(failPermanently); + uploadsThisRun = 0; + root.setAttribute("data-sync-uploaded-this-run", "0"); + root.removeAttribute("data-sync-manual-retry"); + const [next] = await pendingCommands(database); + if (next) synchronize(validatePending(next)).catch(failPermanently); }); - window.addEventListener("online", () => { - if (root.getAttribute("data-sync-phase") === "offline") synchronize(command).catch(failPermanently); + window.addEventListener("online", async () => { + if (root.getAttribute("data-sync-phase") !== "offline") return; + const [next] = await pendingCommands(database); + if (next) synchronize(validatePending(next)).catch(failPermanently); }); await runLeaseLoop(command); } diff --git a/examples/kanban/templates/sync_shell.heml b/examples/kanban/templates/sync_shell.heml index 8241eb4..5a7ab65 100644 --- a/examples/kanban/templates/sync_shell.heml +++ b/examples/kanban/templates/sync_shell.heml @@ -6,7 +6,7 @@ hemx Kanban sync -
+

Sync status

Waiting for pending commands.

diff --git a/examples/kanban/tests/browser_e2e.rs b/examples/kanban/tests/browser_e2e.rs index 8367b12..134ab6d 100644 --- a/examples/kanban/tests/browser_e2e.rs +++ b/examples/kanban/tests/browser_e2e.rs @@ -369,6 +369,114 @@ async fn pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack( result.and(quit) } +#[tokio::test] +async fn upload_backpressure_keeps_pending_work_visible_and_recoverable() -> WebDriverResult<()> { + // test req: sync/017 + let app_port = available_port(); + let app_addr = format!("127.0.0.1:{app_port}"); + let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example")); + app_command.env("HEMX_KANBAN_ADDR", &app_addr); + let _app = TestProcess::start(app_command, "hemx-kanban", &app_addr, STARTUP_TIMEOUT) + .expect("start ready hemx-kanban"); + + let webdriver_port = available_port(); + let webdriver_addr = format!("127.0.0.1:{webdriver_port}"); + let mut webdriver = Command::new("geckodriver"); + webdriver.arg("--port").arg(webdriver_port.to_string()); + let _webdriver = TestProcess::start(webdriver, "geckodriver", &webdriver_addr, STARTUP_TIMEOUT) + .expect("start ready geckodriver"); + let mut caps = DesiredCapabilities::firefox(); + caps.set_headless()?; + let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?; + + let result = async { + driver.goto(&format!("http://{app_addr}/")).await?; + let seeded = driver + .execute_async( + r#" + const done = arguments[arguments.length - 1]; + const open = indexedDB.open('hemx-kanban-v1', 1); + open.onupgradeneeded = () => { + const database = open.result; + if (!database.objectStoreNames.contains('commands')) database.createObjectStore('commands', { keyPath: 'id' }); + if (!database.objectStoreNames.contains('meta')) database.createObjectStore('meta'); + }; + open.onsuccess = () => { + const tx = open.result.transaction('commands', 'readwrite'); + const commands = tx.objectStore('commands'); + for (let causal = 1; causal <= 3; causal += 1) { + commands.add({ + id: `pressure:${causal}`, schemaVersion: 1, actor: 'pressure', session: 'pressure-session', + causal, kind: 'reorder_card', cardId: String(causal), eventKind: 'click', key: null, + }); + } + tx.oncomplete = () => done({ seeded: true }); + tx.onabort = () => done({ error: tx.error && tx.error.name }); + }; + "#, + Vec::new(), + ) + .await? + .json() + .clone(); + assert_eq!(seeded["seeded"], true, "failed to seed pending work: {seeded}"); + + driver.goto(&format!("http://{app_addr}/sync-demo")).await?; + wait_until( + &driver, + "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'backpressured'", + ) + .await?; + let bounded = driver + .execute( + "const root = document.querySelector('[data-kanban-sync]'); return { phase: root.getAttribute('data-sync-phase'), pending: root.getAttribute('data-sync-pending-count'), limit: root.getAttribute('data-sync-upload-limit'), uploaded: root.getAttribute('data-sync-uploaded-this-run'), total: root.getAttribute('data-sync-uploaded-total'), maxInFlight: root.getAttribute('data-sync-max-observed-in-flight'), sequence: root.getAttribute('data-sync-ack-sequence'), manual: root.getAttribute('data-sync-manual-retry'), status: root.querySelector('[role=status]').textContent }", + Vec::new(), + ) + .await? + .json() + .clone(); + assert_eq!(bounded["phase"], "backpressured"); + assert_eq!(bounded["pending"], "1"); + assert_eq!(bounded["limit"], "2"); + assert_eq!(bounded["uploaded"], "2"); + assert_eq!(bounded["total"], "2"); + assert_eq!(bounded["maxInFlight"], "1"); + assert_eq!(bounded["sequence"], "2"); + assert_eq!(bounded["manual"], "available"); + assert_eq!( + bounded["status"], + "Upload limit 2 reached; 1 durable command remains queued. Retry now to continue." + ); + assert_eq!(command_count(&driver).await?, 1); + + driver.find(By::Css("[data-sync-retry]")).await?.click().await?; + wait_until( + &driver, + "const root = document.querySelector('[data-kanban-sync]'); return root?.getAttribute('data-sync-phase') === 'acknowledged' && root?.getAttribute('data-sync-pending-count') === '0'", + ) + .await?; + let recovered = driver + .execute( + "const root = document.querySelector('[data-kanban-sync]'); return { pending: root.getAttribute('data-sync-pending-count'), uploaded: root.getAttribute('data-sync-uploaded-this-run'), total: root.getAttribute('data-sync-uploaded-total'), maxInFlight: root.getAttribute('data-sync-max-observed-in-flight'), sequence: root.getAttribute('data-sync-ack-sequence'), status: root.querySelector('[role=status]').textContent }", + Vec::new(), + ) + .await? + .json() + .clone(); + assert_eq!(recovered["pending"], "0"); + assert_eq!(recovered["uploaded"], "1"); + assert_eq!(recovered["total"], "3"); + assert_eq!(recovered["maxInFlight"], "1"); + assert_eq!(recovered["sequence"], "3"); + assert_eq!(recovered["status"], "Command pressure:3 acknowledged in done."); + assert_eq!(command_count(&driver).await?, 0); + Ok(()) + } + .await; + let quit = driver.quit().await; + result.and(quit) +} + #[tokio::test] async fn two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application( ) -> WebDriverResult<()> {