feat(kanban): stop mixed queue on rejection

req: sync/009 req: sync/010
2026-07-13 17:53:53 +02:00
parent 80f1ae155e
commit 3ace17f9f1
3 changed files with 171 additions and 10 deletions
@@ -42,11 +42,11 @@ encryption, retention, backup, and deployment policy remain host concerns.
 ## Slice 4 — authoritative reconnect and convergence

 - [ ] **User value:** offline and concurrent work reconnects without duplicate mutation, silent loss, stale authorization, or ambiguous conflict.
- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Each activation serializes uploads with one in flight, processes at most two acknowledged commands, exposes the retained durable count when backpressured, and resumes the next bounded run only through the visible retry action. Broader conflict decisions, upgrade mid-queue, and auth isolation remain.
+- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Each activation serializes uploads with one in flight, processes at most two acknowledged commands, exposes the retained durable count when backpressured, and resumes the next bounded run only through the visible retry action. A mixed queue commits and removes its accepted prefix exactly once, then stops on the first permanent rejection with the typed server cause visible, the rejected command plus untouched suffix durable, and blind retry disabled. Broader conflict decisions, upgrade mid-queue, and auth isolation remain.
 - **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics.
 - **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy.
 - **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`.
- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_converges_without_losing_local_intent -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit with acknowledged removal, then a divergent canonical update producing explicit conflict while the local command and prior committed snapshot/cursor remain intact. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. The completed slice proof must additionally cover broader partial reject/conflict decisions, upgrade mid-queue, and multi-user isolation.
+- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_converges_without_losing_local_intent -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit with acknowledged removal, then a divergent canonical update producing explicit conflict while the local command and prior committed snapshot/cursor remain intact. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. `cargo test -p hemx-kanban-example --test browser_e2e mixed_queue_removes_accepted_prefix_and_retains_rejected_tail -- --exact` proves an accepted prefix is canonically applied and removed once before a permanent rejection stops processing, exposes its typed HTTP/server cause, disables blind retry, and leaves both the rejected command and untouched suffix durable. The completed slice proof must additionally cover broader conflict decisions, upgrade mid-queue, and multi-user isolation.

 ## Slice 5 — local-first multiplayer Kanban milestone

@@ -19,10 +19,12 @@ let maxObservedInFlight = 0;
 let stopped = false;

 class UploadError extends Error {
-  constructor(status, retryable) {
+  constructor(status, retryable, reason) {
    super(`sync upload failed with ${status}`);
    this.name = "UploadError";
+    this.status = status;
    this.retryable = retryable;
+    this.reason = reason;
  }
 }

@@ -132,10 +134,17 @@ function setOnline(online) {
  root.setAttribute("data-sync-connection", online ? "online" : "offline");
 }

+function setManualRetryAvailable(available) {
+  const retry = root.querySelector("[data-sync-retry]");
+  retry.disabled = !available;
+  if (available) root.setAttribute("data-sync-manual-retry", "available");
+  else root.removeAttribute("data-sync-manual-retry");
+}
+
 function scheduleManualRetry(command, error) {
  clearTimeout(retryTimer);
  root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error));
-  root.setAttribute("data-sync-manual-retry", "available");
+  setManualRetryAvailable(true);
  setPhase("offline", "Sync is offline after bounded retries; the durable command remains queued. Retry now when ready.");
  root.dispatchEvent(new CustomEvent("kanban:sync-exhausted", { detail: { commandId: command.id, attempts: MAX_ATTEMPTS } }));
 }
@@ -157,7 +166,11 @@ async function upload(command) {
        await new Promise((resolve) => setTimeout(resolve, delay));
        continue;
      }
-      if (!response.ok) throw new UploadError(response.status, response.status >= 500);
+      if (!response.ok) {
+        const problem = await response.json().catch(() => ({}));
+        const reason = typeof problem.error === "string" ? problem.error : "unclassified rejection";
+        throw new UploadError(response.status, response.status >= 500, reason);
+      }
      return response.json();
    } catch (error) {
      if (error instanceof UploadError && !error.retryable) throw error;
@@ -190,7 +203,7 @@ async function continuePendingWork() {
  root.setAttribute("data-sync-pending-count", String(commands.length));
  if (commands.length === 0) return;
  if (uploadsThisRun >= uploadLimit) {
-    root.setAttribute("data-sync-manual-retry", "available");
+    setManualRetryAvailable(true);
    setPhase("backpressured", `Upload limit ${uploadLimit} reached; ${commands.length} durable command${commands.length === 1 ? " remains" : "s remain"} queued. Retry now to continue.`);
    return;
  }
@@ -285,8 +298,23 @@ async function synchronize(command) {
    });
  } catch (error) {
    synchronizing = false;
+    if (error instanceof UploadError && !error.retryable) {
+      setOnline(true);
+      clearTimeout(leaseTimer);
+      root.setAttribute("data-sync-error", error.message);
+      root.setAttribute("data-sync-error-kind", "permanent-rejection");
+      root.setAttribute("data-sync-error-status", String(error.status));
+      root.setAttribute("data-sync-error-reason", error.reason);
+      root.setAttribute("data-sync-rejected-command-id", command.id);
+      const remaining = await pendingCommands(database);
+      root.setAttribute("data-sync-pending-count", String(remaining.length));
+      setManualRetryAvailable(false);
+      setPhase("rejected", `Command ${command.id} was permanently rejected (${error.status}: ${error.reason}); ${remaining.length} durable command${remaining.length === 1 ? " remains" : "s remain"} queued for review.`);
+      await releaseUploaderLease(database);
+      root.setAttribute("data-sync-leader", "false");
+      return;
+    }
    setOnline(false);
-    if (error instanceof UploadError && !error.retryable) throw error;
    scheduleManualRetry(command, error);
  }
 }
@@ -320,6 +348,7 @@ async function start() {
  root.setAttribute("data-sync-in-flight", "0");
  root.setAttribute("data-sync-max-observed-in-flight", "0");
  root.setAttribute("data-sync-pending-count", String(commands.length));
+  setManualRetryAvailable(false);
  if (commands.length === 0) {
    setPhase("idle", "No pending commands.");
    return;
@@ -329,12 +358,13 @@ async function start() {
    if (!event.target.closest("[data-sync-retry]")) return;
    uploadsThisRun = 0;
    root.setAttribute("data-sync-uploaded-this-run", "0");
-    root.removeAttribute("data-sync-manual-retry");
+    setManualRetryAvailable(false);
    const [next] = await pendingCommands(database);
    if (next) synchronize(validatePending(next)).catch(failPermanently);
  });
  window.addEventListener("online", async () => {
    if (root.getAttribute("data-sync-phase") !== "offline") return;
+    setManualRetryAvailable(false);
    const [next] = await pendingCommands(database);
    if (next) synchronize(validatePending(next)).catch(failPermanently);
  });
@@ -335,7 +335,7 @@ async fn pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack(
        driver.goto(&format!("http://{app_addr}/sync-demo")).await?;
        wait_until(
            &driver,
-            "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'failed'",
+            "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'rejected'",
        )
        .await?;
        let rejected = driver
@@ -349,7 +349,10 @@ async fn pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack(
        assert_eq!(rejected["pending"], "1");
        assert_eq!(rejected["attempts"], "1");
        assert_eq!(rejected["error"], "sync upload failed with 409");
-        assert_eq!(rejected["status"], "Sync failed; the durable command remains queued.");
+        assert_eq!(
+            rejected["status"],
+            "Command sync-actor:1 was permanently rejected (409: command_id was already used for a different payload); 1 durable command remains queued for review."
+        );

        driver.goto(&format!("http://{app_addr}/")).await?;
        let canonical = driver
@@ -369,6 +372,134 @@ async fn pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack(
    result.and(quit)
 }

+#[tokio::test]
+async fn mixed_queue_removes_accepted_prefix_and_retains_rejected_tail() -> WebDriverResult<()> {
+    // test req: sync/009 req: sync/010
+    let app_port = available_port();
+    let app_addr = format!("127.0.0.1:{app_port}");
+    let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example"));
+    app_command.env("HEMX_KANBAN_ADDR", &app_addr);
+    let _app = TestProcess::start(app_command, "hemx-kanban", &app_addr, STARTUP_TIMEOUT)
+        .expect("start ready hemx-kanban");
+
+    let webdriver_port = available_port();
+    let webdriver_addr = format!("127.0.0.1:{webdriver_port}");
+    let mut webdriver = Command::new("geckodriver");
+    webdriver.arg("--port").arg(webdriver_port.to_string());
+    let _webdriver = TestProcess::start(webdriver, "geckodriver", &webdriver_addr, STARTUP_TIMEOUT)
+        .expect("start ready geckodriver");
+    let mut caps = DesiredCapabilities::firefox();
+    caps.set_headless()?;
+    let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?;
+
+    let result = async {
+        driver.goto(&format!("http://{app_addr}/")).await?;
+        let seeded = driver
+            .execute_async(
+                r#"
+                const done = arguments[arguments.length - 1];
+                const open = indexedDB.open('hemx-kanban-v1', 1);
+                open.onupgradeneeded = () => {
+                  const database = open.result;
+                  if (!database.objectStoreNames.contains('commands')) database.createObjectStore('commands', { keyPath: 'id' });
+                  if (!database.objectStoreNames.contains('meta')) database.createObjectStore('meta');
+                };
+                open.onsuccess = () => {
+                  const tx = open.result.transaction('commands', 'readwrite');
+                  const commands = tx.objectStore('commands');
+                  for (const [causal, cardId] of [[1, '1'], [2, '999'], [3, '2']]) {
+                    commands.add({
+                      id: `mixed:${causal}`, schemaVersion: 1, actor: 'mixed', session: 'mixed-session',
+                      causal, kind: 'reorder_card', cardId, eventKind: 'click', key: null,
+                    });
+                  }
+                  tx.oncomplete = () => done({ seeded: true });
+                  tx.onabort = () => done({ error: tx.error && tx.error.name });
+                };
+                "#,
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(seeded["seeded"], true, "failed to seed mixed queue: {seeded}");
+
+        driver.goto(&format!("http://{app_addr}/sync-demo")).await?;
+        wait_until(
+            &driver,
+            "return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'rejected'",
+        )
+        .await?;
+        tokio::time::sleep(Duration::from_millis(150)).await;
+        let rejected = driver
+            .execute(
+                "const root = document.querySelector('[data-kanban-sync]'); const retry = root.querySelector('[data-sync-retry]'); return { phase: root.getAttribute('data-sync-phase'), pending: root.getAttribute('data-sync-pending-count'), uploaded: root.getAttribute('data-sync-uploaded-total'), ackSequence: root.getAttribute('data-sync-ack-sequence'), attempts: root.getAttribute('data-sync-attempts'), inFlight: root.getAttribute('data-sync-in-flight'), maxInFlight: root.getAttribute('data-sync-max-observed-in-flight'), kind: root.getAttribute('data-sync-error-kind'), errorStatus: root.getAttribute('data-sync-error-status'), reason: root.getAttribute('data-sync-error-reason'), rejectedId: root.getAttribute('data-sync-rejected-command-id'), retryDisabled: retry.disabled, status: root.querySelector('[role=status]').textContent }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(rejected["phase"], "rejected");
+        assert_eq!(rejected["pending"], "2");
+        assert_eq!(rejected["uploaded"], "1");
+        assert_eq!(rejected["ackSequence"], "1");
+        assert_eq!(rejected["attempts"], "1");
+        assert_eq!(rejected["inFlight"], "0");
+        assert_eq!(rejected["maxInFlight"], "1");
+        assert_eq!(rejected["kind"], "permanent-rejection");
+        assert_eq!(rejected["errorStatus"], "400");
+        assert_eq!(rejected["reason"], "unknown card_id");
+        assert_eq!(rejected["rejectedId"], "mixed:2");
+        assert_eq!(rejected["retryDisabled"], true);
+        assert_eq!(
+            rejected["status"],
+            "Command mixed:2 was permanently rejected (400: unknown card_id); 2 durable commands remain queued for review."
+        );
+
+        let queued = driver
+            .execute_async(
+                r#"
+                const done = arguments[arguments.length - 1];
+                const open = indexedDB.open('hemx-kanban-v1', 1);
+                open.onsuccess = () => {
+                  const request = open.result.transaction('commands', 'readonly').objectStore('commands').getAll();
+                  request.onsuccess = () => done(request.result.sort((a, b) => a.causal - b.causal).map(({ id, cardId }) => ({ id, cardId })));
+                  request.onerror = () => done({ error: request.error && request.error.name });
+                };
+                "#,
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(
+            queued,
+            serde_json::json!([
+                { "id": "mixed:2", "cardId": "999" },
+                { "id": "mixed:3", "cardId": "2" }
+            ])
+        );
+
+        driver.goto(&format!("http://{app_addr}/")).await?;
+        let canonical = driver
+            .execute(
+                "return [...document.querySelectorAll('section.column')].map((column) => ({ title: column.querySelector('h2').textContent, cards: [...column.querySelectorAll('[data-key]')].map((card) => card.dataset.key) }))",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(canonical[1]["title"], "Doing");
+        assert_eq!(canonical[1]["cards"], serde_json::json!(["2"]));
+        assert_eq!(canonical[2]["title"], "Done");
+        assert_eq!(canonical[2]["cards"], serde_json::json!(["1", "3"]));
+        Ok(())
+    }
+    .await;
+    let quit = driver.quit().await;
+    result.and(quit)
+}
+
 #[tokio::test]
 async fn upload_backpressure_keeps_pending_work_visible_and_recoverable() -> WebDriverResult<()> {
    // test req: sync/017