diff --git a/PLAN.md b/PLAN.md index 03dc397..90f6e69 100644 --- a/PLAN.md +++ b/PLAN.md @@ -46,7 +46,7 @@ encryption, retention, backup, and deployment policy remain host concerns. - **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics. - **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy. - **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`. -- **Proof:** `cargo test -p hemx-wasm --test browser flat_patch_persists_offline_then_uploads_with_same_operation_identity -- --exact` proves `sync/002` and `sync/003` through generated real WASM: `SyncEffect::send_patch` travels beside ordinary DOM effects, the strict flat scalar schema is accepted, the offline interaction performs zero patch requests while remaining durable, and reconnect uploads with identical opaque operation/idempotency identity before matching acknowledgement removes it. `cargo test -p hemx-wasm --test browser kanban_public_api_offline_sync_journey_converges_without_duplicate_replay -- --exact` proves `sync/023` through the public generated client handler and real WASM: offline durable mutation, unavailable-origin reload projection, reconnect upload, canonical convergence, duplicate idempotency without a second event, conflicting duplicate rejection, invalid-command rejection, and an empty local queue. `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_and_user_conflict_resolution_preserve_suffix -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit, then a divergent canonical update producing explicit conflict with a two-command queue; the user’s canonical-state action removes only the conflicted head, retains and resumes the suffix, and reaches canonical convergence with an empty queue. `cargo test -p hemx-kanban-example --test browser_e2e keep_local_retry_preserves_conflicted_command_and_suffix_order -- --exact` proves the keep-local action derives one resolution identity; a permanent rejection preserves order and re-enables the choice; three later transport failures still preserve the original conflicted head plus suffix and expose manual recovery; final acceptance removes only the head and converges the untouched suffix without duplication or loss. `cargo test -p hemx-kanban-example --test browser_e2e redacted_sync_diagnostics_are_bounded_and_leak_no_sensitive_material -- --exact` proves a 15-second-old queue is bucketed rather than timestamped, cursor and acknowledgement latency advance after recovery, diagnostics stay under a fixed small readout, and command id/payload/actor/session/cookie material never reaches the diagnostics surface; the conflict and keep-local tests additionally assert conflict/rejection counters at their actual failure boundaries. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. `cargo test -p hemx-kanban-example --test browser_e2e mixed_queue_removes_accepted_prefix_and_retains_rejected_tail -- --exact` proves an accepted prefix is canonically applied and removed once before a permanent rejection stops processing, exposes its typed HTTP/server cause, disables blind retry, and leaves both the rejected command and untouched suffix durable. `cargo test -p hemx-kanban-example --test browser_e2e schema_upgrade_preserves_queued_order_and_local_intent -- --exact` proves a three-command v1 queue migrates atomically to the explicit-target v2 schema, remains byte-for-intent ordered after interrupted upload, then receives canonical sequences 1–3 in original order and drains without loss. `cargo test -p hemx-kanban-example --test browser_e2e account_partition_hides_replay_and_export_until_owner_returns -- --exact` proves a beta editor and alpha viewer enumerate zero commands and issue no replay for an alpha owner queue, signed-out startup cannot open a partition, no foreign id/export surface leaks, and only the returning alpha owner can export the intact command then resume it exactly once. `cargo test -p hemx-kanban-example --test browser_e2e canonical_snapshot_and_history_are_tenant_scoped -- --exact` proves alpha and beta sessions receive only their own canonical cards and acknowledgement events with independent sequence 1 cursors, while signed-out snapshot/history reads receive 401 and expose no stream. `cargo test -p hemx-kanban-example --test browser_e2e identical_sync_inputs_reconcile_deterministically -- --exact` proves `sync/022` through a real browser and live sync endpoints: one accepted server result and canonical snapshot plus an ordered command sequence produce byte-equivalent structured reconciliation records on independent cloned inputs, do not mutate inputs, and select the explicit server-authoritative model. Per-requirement audit: every `sync/001-023` row now has concrete browser proof. Slice 4 remains open pending a strict citation/proof audit of its cross-cutting `operations/001-005`, `security/002-005`, and `performance/004-005` requirements; the next slice must close those requirements from existing proof or identify the smallest real missing behavior. +- **Proof:** `cargo test -p hemx-wasm --test browser flat_patch_persists_offline_then_uploads_with_same_operation_identity -- --exact` proves `sync/002` and `sync/003` through generated real WASM: `SyncEffect::send_patch` travels beside ordinary DOM effects, the strict flat scalar schema is accepted, the offline interaction performs zero patch requests while remaining durable, and reconnect uploads with identical opaque operation/idempotency identity before matching acknowledgement removes it. `cargo test -p hemx-wasm --test browser kanban_public_api_offline_sync_journey_converges_without_duplicate_replay -- --exact` proves `sync/023` through the public generated client handler and real WASM: offline durable mutation, unavailable-origin reload projection, reconnect upload, canonical convergence, duplicate idempotency without a second event, conflicting duplicate rejection, invalid-command rejection, and an empty local queue. `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_and_user_conflict_resolution_preserve_suffix -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit, then a divergent canonical update producing explicit conflict with a two-command queue; the user’s canonical-state action removes only the conflicted head, retains and resumes the suffix, and reaches canonical convergence with an empty queue. `cargo test -p hemx-kanban-example --test browser_e2e keep_local_retry_preserves_conflicted_command_and_suffix_order -- --exact` proves the keep-local action derives one resolution identity; a permanent rejection preserves order and re-enables the choice; three later transport failures still preserve the original conflicted head plus suffix and expose manual recovery; final acceptance removes only the head and converges the untouched suffix without duplication or loss. `cargo test -p hemx-kanban-example --test browser_e2e redacted_sync_diagnostics_are_bounded_and_leak_no_sensitive_material -- --exact` proves a 15-second-old queue is bucketed rather than timestamped, cursor and acknowledgement latency advance after recovery, diagnostics stay under a fixed small readout, and command id/payload/actor/session/cookie material never reaches the diagnostics surface; the conflict and keep-local tests additionally assert conflict/rejection counters at their actual failure boundaries. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. `cargo test -p hemx-kanban-example --test browser_e2e mixed_queue_removes_accepted_prefix_and_retains_rejected_tail -- --exact` proves an accepted prefix is canonically applied and removed once before a permanent rejection stops processing, exposes its typed HTTP/server cause, disables blind retry, and leaves both the rejected command and untouched suffix durable. `cargo test -p hemx-kanban-example --test browser_e2e schema_upgrade_preserves_queued_order_and_local_intent -- --exact` proves a three-command v1 queue migrates atomically to the explicit-target v2 schema, remains byte-for-intent ordered after interrupted upload, then receives canonical sequences 1–3 in original order and drains without loss. `cargo test -p hemx-kanban-example --test browser_e2e account_partition_hides_replay_and_export_until_owner_returns -- --exact` proves a beta editor and alpha viewer enumerate zero commands and issue no replay for an alpha owner queue, signed-out startup cannot open a partition, no foreign id/export surface leaks, and only the returning alpha owner can export the intact command then resume it exactly once. `cargo test -p hemx-kanban-example --test browser_e2e canonical_snapshot_and_history_are_tenant_scoped -- --exact` proves alpha and beta sessions receive only their own canonical cards and acknowledgement events with independent sequence 1 cursors, while signed-out snapshot/history reads receive 401 and expose no stream. `cargo test -p hemx-kanban-example --test browser_e2e identical_sync_inputs_reconcile_deterministically -- --exact` proves `sync/022` through a real browser and live sync endpoints: one accepted server result and canonical snapshot plus an ordered command sequence produce byte-equivalent structured reconciliation records on independent cloned inputs, do not mutate inputs, and select the explicit server-authoritative model. `cargo test -p hemx-kanban-example --test browser_e2e sync_requests_timeout_and_cancel_on_pagehide -- --exact` proves the sync request boundary enforces a named finite timeout and reports `TimeoutError`, while page teardown cancels a pending request with `AbortError`, closes the acknowledgement stream, and clears retry/lease timers. Per-requirement audit: every `sync/001-023` row now has concrete browser proof; existing live-browser proofs also cover `operations/001-002`, `operations/005`, `security/002`, `security/004`, `performance/004`, and `performance/005`, while this slice closes the sync request/stream leg of `operations/003`. Slice 4 remains open on `operations/004` bounded liveness/slow-consumer proof, the non-sync handler/background-replay legs of `operations/003`, `security/003` host enforcement plus request media/size boundaries, and `security/005` adversarial decoder coverage. The next coherent slice is `security/005`: prove malformed effect/state/event/command/sync inputs—including unknown versions, invalid kinds, oversized lengths, truncation, and trailing data—are rejected before any partial application. ## Slice 5 — local-first multiplayer Kanban milestone diff --git a/examples/kanban/static/sync.js b/examples/kanban/static/sync.js index dfcf376..32e47f9 100644 --- a/examples/kanban/static/sync.js +++ b/examples/kanban/static/sync.js @@ -7,6 +7,7 @@ const LEGACY_COMMAND_SCHEMA = 1; const MIGRATION_KEY = "commandSchemaMigration"; const MAX_ATTEMPTS = 3; const BACKOFF_MS = [25, 50]; +const REQUEST_TIMEOUT_MS = 1_000; const root = document.querySelector("[data-kanban-sync]"); const TAB_ID = sessionStorage.getItem("hemx-kanban-sync-tab-id") || crypto.randomUUID(); const LEASE_MS = 5000; @@ -16,6 +17,8 @@ let accountPartition; let uploadLimit; let retryTimer; let leaseTimer; +let acknowledgementSource; +const activeRequests = new Set(); let synchronizing = false; let uploadsThisRun = 0; let uploadedTotal = 0; @@ -39,6 +42,30 @@ class UploadError extends Error { } } +// req: operations/003 +export async function fetchWithTimeout( + input, + init = {}, + fetchImplementation = fetch, + timeoutMs = REQUEST_TIMEOUT_MS, +) { + if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 1) { + throw new TypeError("sync request timeout must be a positive integer"); + } + const controller = new AbortController(); + const timeout = setTimeout( + () => controller.abort(new DOMException(`sync request timed out after ${timeoutMs} ms`, "TimeoutError")), + timeoutMs, + ); + activeRequests.add(controller); + try { + return await fetchImplementation(input, { ...init, signal: controller.signal }); + } finally { + clearTimeout(timeout); + activeRequests.delete(controller); + } +} + function requestResult(request) { return new Promise((resolve, reject) => { request.addEventListener("success", () => resolve(request.result), { once: true }); @@ -322,7 +349,7 @@ async function upload(command) { setPhase(attempt === 1 ? "uploading" : "retrying", `Uploading ${command.id} (attempt ${attempt} of ${MAX_ATTEMPTS}).`); try { const query = new URLSearchParams({ command_id: command.id, card_id: command.cardId, column: command.targetColumn }); - const response = await fetch(`/sync/commands?${query}`, { method: "POST" }); + const response = await fetchWithTimeout(`/sync/commands?${query}`, { method: "POST" }); if (response.status === 503 && attempt < MAX_ATTEMPTS) { const base = BACKOFF_MS[attempt - 1]; const delay = base + Math.floor(Math.random() * base); @@ -413,6 +440,7 @@ async function synchronize(command) { const reconnect = command.session || command.actor || "kanban"; const source = new EventSource(`/sync/acknowledgements?after=0&reconnect=${encodeURIComponent(reconnect)}`); + acknowledgementSource = source; let opens = 0; source.addEventListener("open", () => { opens += 1; @@ -422,6 +450,7 @@ async function synchronize(command) { const canonical = JSON.parse(event.data); if (canonical.commandId !== command.id) return; source.close(); + if (acknowledgementSource === source) acknowledgementSource = undefined; root.setAttribute("data-sync-pending-before-ack", String((await pendingCommands(database)).length)); const queueCommandId = command.queueCommandId || command.id; await removePendingCommand(database, queueCommandId); @@ -454,7 +483,7 @@ async function synchronize(command) { }); source.addEventListener("snapshot-required", async (event) => { const missing = JSON.parse(event.data); - const response = await fetch(missing.snapshotUrl); + const response = await fetchWithTimeout(missing.snapshotUrl); if (!response.ok) throw new Error(`snapshot failed with ${response.status}`); const snapshot = await response.json(); const queued = await pendingCommands(database); @@ -497,6 +526,7 @@ async function synchronize(command) { } synchronizing = false; source.close(); + if (acknowledgementSource === source) acknowledgementSource = undefined; clearTimeout(leaseTimer); await releaseUploaderLease(database); root.setAttribute("data-sync-leader", "false"); @@ -564,7 +594,8 @@ async function runLeaseLoop(command) { async function start() { if (!root) return; - const contextResponse = await fetch("/sync/context", { credentials: "same-origin", cache: "no-store" }); + root.setAttribute("data-sync-request-timeout-ms", String(REQUEST_TIMEOUT_MS)); + const contextResponse = await fetchWithTimeout("/sync/context", { credentials: "same-origin", cache: "no-store" }); if (!contextResponse.ok) throw new Error(`account context failed with ${contextResponse.status}`); const context = await contextResponse.json(); if (!context || typeof context.accountPartition !== "string" || !context.accountPartition) { @@ -633,6 +664,12 @@ async function start() { window.addEventListener("pagehide", () => { stopped = true; clearTimeout(leaseTimer); + clearTimeout(retryTimer); + acknowledgementSource?.close(); + acknowledgementSource = undefined; + for (const controller of activeRequests) { + controller.abort(new DOMException("sync cancelled because page is hidden", "AbortError")); + } if (database) releaseUploaderLease(database).catch(() => {}); }); diff --git a/examples/kanban/tests/browser_e2e.rs b/examples/kanban/tests/browser_e2e.rs index 7076a63..e2c8478 100644 --- a/examples/kanban/tests/browser_e2e.rs +++ b/examples/kanban/tests/browser_e2e.rs @@ -599,7 +599,7 @@ async fn account_partition_hides_replay_and_export_until_owner_returns() -> WebD #[tokio::test] async fn canonical_snapshot_and_history_are_tenant_scoped() -> WebDriverResult<()> { - // test req: sync/007 req: security/004 req: auth/005 + // test req: sync/007 req: security/004 req: auth/005 req: performance/004 let app_port = available_port(); let app_addr = format!("127.0.0.1:{app_port}"); let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example")); @@ -1708,7 +1708,7 @@ async fn missing_history_rebase_and_user_conflict_resolution_preserve_suffix() - #[tokio::test] async fn redacted_sync_diagnostics_are_bounded_and_leak_no_sensitive_material( ) -> WebDriverResult<()> { - // test req: operations/001 req: security/002 req: sync/016 req: sync/021 + // test req: operations/001 req: operations/005 req: security/002 req: sync/016 req: sync/021 let app_port = available_port(); let app_addr = format!("127.0.0.1:{app_port}"); let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example")); @@ -2077,6 +2077,81 @@ async fn keep_local_retry_preserves_conflicted_command_and_suffix_order() -> Web result.and(quit) } +#[tokio::test] +async fn sync_requests_timeout_and_cancel_on_pagehide() -> WebDriverResult<()> { + // test req: operations/003 + let app_port = available_port(); + let app_addr = format!("127.0.0.1:{app_port}"); + let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example")); + app_command.env("HEMX_KANBAN_ADDR", &app_addr); + let _app = TestProcess::start(app_command, "hemx-kanban", &app_addr, STARTUP_TIMEOUT) + .expect("start hemx-kanban"); + + let webdriver_port = available_port(); + let webdriver_addr = format!("127.0.0.1:{webdriver_port}"); + let mut webdriver = Command::new("geckodriver"); + webdriver.arg("--port").arg(webdriver_port.to_string()); + let _webdriver = TestProcess::start(webdriver, "geckodriver", &webdriver_addr, STARTUP_TIMEOUT) + .expect("start ready geckodriver"); + let mut caps = DesiredCapabilities::firefox(); + caps.set_headless()?; + let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?; + + let result = async { + driver.goto(&format!("http://{app_addr}/sync-demo")).await?; + wait_until( + &driver, + "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'idle'", + ) + .await?; + let proof = driver + .execute_async( + r#" + const done = arguments[arguments.length - 1]; + (async () => { + const { fetchWithTimeout } = await import('/sync.js'); + const pendingFetch = (_input, init) => new Promise((_resolve, reject) => { + init.signal.addEventListener('abort', () => reject(init.signal.reason), { once: true }); + }); + const started = performance.now(); + let timeout; + try { + await fetchWithTimeout('/never-timeout', {}, pendingFetch, 40); + } catch (error) { + timeout = { name: error.name, message: error.message, elapsedMs: performance.now() - started }; + } + const cancellationPromise = fetchWithTimeout('/never-pagehide', {}, pendingFetch, 10_000) + .then(() => ({ resolved: true })) + .catch((error) => ({ name: error.name, message: error.message })); + window.dispatchEvent(new PageTransitionEvent('pagehide')); + done({ timeout, cancellation: await cancellationPromise }); + })().catch((error) => done({ error: String(error), stack: error?.stack })); + "#, + Vec::new(), + ) + .await? + .json() + .clone(); + assert!(proof["error"].is_null(), "bounded request failed: {proof}"); + assert_eq!(proof["timeout"]["name"], "TimeoutError", "{proof}"); + assert!(proof["timeout"]["message"] + .as_str() + .is_some_and(|message| message.contains("40 ms"))); + assert!(proof["timeout"]["elapsedMs"] + .as_f64() + .is_some_and(|elapsed| (35.0..1_000.0).contains(&elapsed))); + assert_eq!(proof["cancellation"]["name"], "AbortError", "{proof}"); + assert_eq!( + proof["cancellation"]["message"], + "sync cancelled because page is hidden" + ); + Ok(()) + } + .await; + let quit = driver.quit().await; + result.and(quit) +} + #[tokio::test] async fn identical_sync_inputs_reconcile_deterministically() -> WebDriverResult<()> { // test req: sync/022