feat(kanban): recover after retry exhaustion

req: sync/004\nreq: sync/010\nreq: sync/011\nreq: sync/014\nreq: sync/016

PLAN.md

@@ -42,11 +42,11 @@ encryption, retention, backup, and deployment policy remain host concerns.
 ## Slice 4 — authoritative reconnect and convergence
 
 - [ ] **User value:** offline and concurrent work reconnects without duplicate mutation, silent loss, stale authorization, or ambiguous conflict.
-- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, survives one injected 503 through bounded exponential backoff with randomized jitter, uploads through the idempotent command endpoint, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; a canonical payload conflict is not retried and remains durable with a visible reason. Exhausted-retry/offline recovery, broader conflict decisions, multi-tab leadership, and auth isolation remain.
+- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. Broader conflict decisions, multi-tab leadership, backpressure, and auth isolation remain.
 - **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics.
 - **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy.
 - **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`.
-- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. The completed slice proof must additionally cover exhausted-retry/offline recovery, partial reject/conflict decisions, missing-history snapshots, two tabs, backpressure, upgrade mid-queue, and multi-user isolation.
+- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. The completed slice proof must additionally cover broader partial reject/conflict decisions, missing-history snapshots, two tabs, backpressure, upgrade mid-queue, and multi-user isolation.
 
 ## Slice 5 — local-first multiplayer Kanban milestone
 

examples/kanban/src/main.rs

@@ -15,7 +15,7 @@ use hemx_kanban_example::ui::board::{self as board};
 use hemx_kanban_example::ui::board_card as card_board;
 use hemx_kanban_example::ui::{self, board as board_ui};
 use serde::{Deserialize, Serialize};
-use std::collections::{BTreeMap, BTreeSet};
+use std::collections::BTreeMap;
 use std::convert::Infallible;
 use std::fs::{self, OpenOptions};
 use std::io::Write;
@@ -41,8 +41,8 @@ struct SyncState {
     next_sequence: u64,
     acknowledgements: BTreeMap<CommandId, SyncAcknowledgement>,
     reconnects: BTreeMap<String, u64>,
-    fail_first_upload: bool,
-    transient_failures: BTreeSet<CommandId>,
+    transient_failure_limit: u8,
+    transient_failures: BTreeMap<CommandId, u8>,
 }
 
 #[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd)]
@@ -294,7 +294,10 @@ async fn main() {
             next_sequence: 1,
             ..SyncState::default()
         });
-    sync.fail_first_upload = std::env::var_os("HEMX_KANBAN_FAIL_FIRST_SYNC").is_some();
+    sync.transient_failure_limit = std::env::var("HEMX_KANBAN_SYNC_FAILURES")
+        .ok()
+        .and_then(|value| value.parse::<u8>().ok())
+        .unwrap_or_else(|| u8::from(std::env::var_os("HEMX_KANBAN_FAIL_FIRST_SYNC").is_some()));
     let mut board = initial_board();
     for acknowledgement in sync.acknowledgements.values() {
         if let Some(card) = board
@@ -433,9 +436,17 @@ async fn sync_command(
         }
         return Ok(Json(existing.clone()));
     }
-    if sync.fail_first_upload && sync.transient_failures.insert(command_id.clone()) {
+    let transient_failure_limit = sync.transient_failure_limit;
+    if transient_failure_limit > 0 {
+        let failures = sync
+            .transient_failures
+            .entry(command_id.clone())
+            .or_default();
+        if *failures < transient_failure_limit {
+            *failures += 1;
             return Err(SyncRejection::Transient);
         }
+    }
 
     let mut board = state.board.lock().unwrap();
     let card_index = board

examples/kanban/static/sync.js

@@ -3,6 +3,8 @@ const COMMANDS = "commands";
 const MAX_ATTEMPTS = 3;
 const BACKOFF_MS = [25, 50];
 const root = document.querySelector("[data-kanban-sync]");
+let database;
+let retryTimer;
 
 class UploadError extends Error {
   constructor(status, retryable) {
@@ -64,6 +66,18 @@ function validatePending(command) {
   return command;
 }
 
+function setOnline(online) {
+  root.setAttribute("data-sync-connection", online ? "online" : "offline");
+}
+
+function scheduleManualRetry(command, error) {
+  clearTimeout(retryTimer);
+  root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error));
+  root.setAttribute("data-sync-manual-retry", "available");
+  setPhase("offline", "Sync is offline after bounded retries; the durable command remains queued. Retry now when ready.");
+  root.dispatchEvent(new CustomEvent("kanban:sync-exhausted", { detail: { commandId: command.id, attempts: MAX_ATTEMPTS } }));
+}
+
 async function upload(command) {
   root.setAttribute("data-sync-max-attempts", String(MAX_ATTEMPTS));
   for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) {
@@ -97,17 +111,13 @@ async function upload(command) {
   throw new Error("sync retry limit exhausted");
 }
 
-async function start() {
-  if (!root) return;
-  const database = await openLog();
-  const commands = await pendingCommands(database);
-  root.setAttribute("data-sync-pending-count", String(commands.length));
-  if (commands.length === 0) {
-    setPhase("idle", "No pending commands.");
-    return;
-  }
-  const command = validatePending(commands[0]);
+async function synchronize(command) {
+  root.removeAttribute("data-sync-error");
+  root.removeAttribute("data-sync-manual-retry");
+  setOnline(navigator.onLine);
+  try {
     const acknowledgement = await upload(command);
+    setOnline(true);
     root.setAttribute("data-sync-upload-sequence", String(acknowledgement.serverSequence));
     setPhase("awaiting-ack", `Command ${command.id} uploaded; awaiting canonical acknowledgement.`);
 
@@ -130,10 +140,39 @@ async function start() {
       root.dispatchEvent(new CustomEvent("kanban:sync-acknowledged", { detail: canonical }));
       source.close();
     });
+  } catch (error) {
+    setOnline(false);
+    if (error instanceof UploadError && !error.retryable) throw error;
+    scheduleManualRetry(command, error);
+  }
+}
+
+async function start() {
+  if (!root) return;
+  database = await openLog();
+  const commands = await pendingCommands(database);
+  root.setAttribute("data-sync-pending-count", String(commands.length));
+  if (commands.length === 0) {
+    setPhase("idle", "No pending commands.");
+    return;
+  }
+  const command = validatePending(commands[0]);
+  root.addEventListener("click", (event) => {
+    if (!event.target.closest("[data-sync-retry]")) return;
+    synchronize(command).catch(failPermanently);
+  });
+  window.addEventListener("online", () => {
+    if (root.getAttribute("data-sync-phase") === "offline") synchronize(command).catch(failPermanently);
+  });
+  await synchronize(command);
+}
+
+function failPermanently(error) {
+  root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error));
+  setPhase("failed", "Sync failed; the durable command remains queued.");
 }
 
 start().catch((error) => {
   if (!root) return;
-  root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error));
-  setPhase("failed", "Sync failed; the durable command remains queued.");
+  failPermanently(error);
 });

examples/kanban/templates/sync_shell.heml

@@ -9,6 +9,7 @@
   <section data-kanban-sync data-sync-version="1" aria-labelledby="sync-title">
     <h2 id="sync-title">Sync status</h2>
     <p role="status" aria-live="polite">Waiting for pending commands.</p>
+    <button type="button" data-sync-retry>Retry sync now</button>
   </section>
   <script +src="self.runtime_src" defer></script>
   <script src="/sync.js" defer></script>

examples/kanban/tests/browser_e2e.rs

@@ -369,6 +369,131 @@ async fn pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack(
     result.and(quit)
 }
 
+#[tokio::test]
+async fn exhausted_offline_retries_keep_command_until_later_reconnect() -> WebDriverResult<()> {
+    // test req: sync/004 req: sync/010 req: sync/011 req: sync/014 req: sync/016
+    let app_port = available_port();
+    let app_addr = format!("127.0.0.1:{app_port}");
+    let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example"));
+    app_command
+        .env("HEMX_KANBAN_ADDR", &app_addr)
+        .env("HEMX_KANBAN_SYNC_FAILURES", "3");
+    let _app = TestProcess::start(app_command, "hemx-kanban", &app_addr, STARTUP_TIMEOUT)
+        .expect("start ready hemx-kanban");
+
+    let webdriver_port = available_port();
+    let webdriver_addr = format!("127.0.0.1:{webdriver_port}");
+    let mut webdriver = Command::new("geckodriver");
+    webdriver.arg("--port").arg(webdriver_port.to_string());
+    let _webdriver = TestProcess::start(webdriver, "geckodriver", &webdriver_addr, STARTUP_TIMEOUT)
+        .expect("start ready geckodriver");
+    let mut caps = DesiredCapabilities::firefox();
+    caps.set_headless()?;
+    let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?;
+
+    let result = async {
+        driver.goto(&format!("http://{app_addr}/")).await?;
+        let seeded = driver
+            .execute_async(
+                r#"
+                const done = arguments[arguments.length - 1];
+                const open = indexedDB.open('hemx-kanban-v1', 1);
+                open.onupgradeneeded = () => {
+                  const database = open.result;
+                  if (!database.objectStoreNames.contains('commands')) database.createObjectStore('commands', { keyPath: 'id' });
+                  if (!database.objectStoreNames.contains('meta')) database.createObjectStore('meta');
+                };
+                open.onsuccess = () => {
+                  const tx = open.result.transaction('commands', 'readwrite');
+                  tx.objectStore('commands').add({
+                    id: 'offline-actor:1', schemaVersion: 1, actor: 'offline-actor', session: 'offline-session',
+                    causal: 1, kind: 'reorder_card', cardId: '2', eventKind: 'click', key: null,
+                  });
+                  tx.oncomplete = () => done({ seeded: true });
+                  tx.onabort = () => done({ error: tx.error && tx.error.name });
+                };
+                "#,
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(seeded["seeded"], true, "failed to seed offline command: {seeded}");
+
+        driver.goto(&format!("http://{app_addr}/sync-demo")).await?;
+        wait_until(
+            &driver,
+            "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'offline'",
+        )
+        .await?;
+        let exhausted = driver
+            .execute(
+                "const root = document.querySelector('[data-kanban-sync]'); return { phase: root.getAttribute('data-sync-phase'), connection: root.getAttribute('data-sync-connection'), pending: root.getAttribute('data-sync-pending-count'), attempts: root.getAttribute('data-sync-attempts'), maxAttempts: root.getAttribute('data-sync-max-attempts'), manual: root.getAttribute('data-sync-manual-retry'), error: root.getAttribute('data-sync-error'), status: root.querySelector('[role=status]').textContent, retry: root.querySelector('[data-sync-retry]').textContent }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(exhausted["phase"], "offline");
+        assert_eq!(exhausted["connection"], "offline");
+        assert_eq!(exhausted["pending"], "1");
+        assert_eq!(exhausted["attempts"], "3");
+        assert_eq!(exhausted["maxAttempts"], "3");
+        assert_eq!(exhausted["manual"], "available");
+        assert_eq!(exhausted["error"], "sync upload failed with 503");
+        assert_eq!(
+            exhausted["status"],
+            "Sync is offline after bounded retries; the durable command remains queued. Retry now when ready."
+        );
+        assert_eq!(exhausted["retry"], "Retry sync now");
+        let queued = command_count(&driver).await?;
+        assert_eq!(queued, 1);
+
+        driver
+            .find(By::Css("[data-sync-retry]"))
+            .await?
+            .click()
+            .await?;
+        wait_until(
+            &driver,
+            "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'acknowledged'",
+        )
+        .await?;
+        let converged = driver
+            .execute(
+                "const root = document.querySelector('[data-kanban-sync]'); return { connection: root.getAttribute('data-sync-connection'), pending: root.getAttribute('data-sync-pending-count'), attempts: root.getAttribute('data-sync-attempts'), sequence: root.getAttribute('data-sync-ack-sequence'), column: root.getAttribute('data-sync-canonical-column'), status: root.querySelector('[role=status]').textContent, error: root.getAttribute('data-sync-error') }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(converged["connection"], "online");
+        assert_eq!(converged["pending"], "0");
+        assert_eq!(converged["attempts"], "1");
+        assert_eq!(converged["sequence"], "1");
+        assert_eq!(converged["column"], "done");
+        assert_eq!(converged["status"], "Command offline-actor:1 acknowledged in done.");
+        assert!(converged["error"].is_null());
+        assert_eq!(command_count(&driver).await?, 0);
+
+        driver.goto(&format!("http://{app_addr}/")).await?;
+        let canonical = driver
+            .execute(
+                "return [...document.querySelectorAll('section.column')].map((column) => ({ title: column.querySelector('h2').textContent, cards: [...column.querySelectorAll('[data-key]')].map((card) => card.dataset.key) }))",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(canonical[2]["title"], "Done");
+        assert_eq!(canonical[2]["cards"], serde_json::json!(["2", "3"]));
+        Ok(())
+    }
+    .await;
+    let quit = driver.quit().await;
+    result.and(quit)
+}
+
 #[tokio::test]
 async fn canonical_acknowledgement_survives_server_restart() -> WebDriverResult<()> {
     // test req: sync/001 req: sync/005 req: sync/007 req: sync/008 req: sync/013
@@ -497,6 +622,26 @@ async fn canonical_acknowledgement_survives_server_restart() -> WebDriverResult<
     result.and(quit)
 }
 
+async fn command_count(driver: &WebDriver) -> WebDriverResult<u64> {
+    let count = driver
+        .execute_async(
+            r#"
+            const done = arguments[arguments.length - 1];
+            const open = indexedDB.open('hemx-kanban-v1', 1);
+            open.onsuccess = () => {
+              const request = open.result.transaction('commands', 'readonly').objectStore('commands').count();
+              request.onsuccess = () => done(request.result);
+              request.onerror = () => done({ error: request.error && request.error.name });
+            };
+            "#,
+            Vec::new(),
+        )
+        .await?
+        .json()
+        .clone();
+    Ok(count.as_u64().expect("durable command count"))
+}
+
 async fn wait_until(driver: &WebDriver, script: &str) -> WebDriverResult<()> {
     for _ in 0..200 {
         if driver.execute(script, Vec::new()).await?.json().as_bool() == Some(true) {