fix(kanban): recover from quota exhaustion

req: sync/015

PLAN.md

@@ -33,11 +33,11 @@ encryption, retention, backup, and deployment policy remain host concerns.
 ## Slice 3 — durable offline command log
 
 - [ ] **User value:** an opted-in Kanban mutation remains available after network loss and browser reload without storing DOM patches as truth.
-- **State:** In progress — the first app-owned `reorder_card` command is transactionally persisted with schema, actor, session, causal id, and app payload before projection; an app-owned service worker caches only the generated shell/resources, and reload restores the projection after the fixture server is stopped and proven unreachable. Native recovery controls export a versioned credential-free command envelope, delete queued commands while preserving actor/causal identity, and reset command data, identity, cache, and registration behind explicit confirmation. Unknown schemas stop with an explicit diagnostic. An injected transactional conflict proves failed persistence neither projects nor emits a durability claim, exposes stage/code without command payload, and remains recoverable through deletion. Malformed current-schema records are rejected field-specifically without partial projection while export/delete recovery remains available. Replay is preflighted before effects, capped by the app at 64 commands, measured against a 100 ms browser budget, and over-limit queues fail closed with export/delete recovery. Quota-specific recovery remains.
+- **State:** In progress — the first app-owned `reorder_card` command is transactionally persisted with schema, actor, session, causal id, and app payload before projection; an app-owned service worker caches only the generated shell/resources, and reload restores the projection after the fixture server is stopped and proven unreachable. Native recovery controls export a versioned credential-free command envelope, delete queued commands while preserving actor/causal identity, and reset command data, identity, cache, and registration behind explicit confirmation. Unknown schemas stop with an explicit diagnostic. Transaction conflicts and injected quota exhaustion prove failed persistence neither projects nor emits a durability claim, expose stage/code without command payload, and remain recoverable through export/delete/reset. Malformed current-schema records are rejected field-specifically without partial projection while export/delete recovery remains available. Replay is preflighted before effects, capped by the app at 64 commands, measured against a 100 ms browser budget, and over-limit queues fail closed with export/delete recovery. Visible queued-status latency proof for `performance/006` remains.
 - **Build:** add an optional durable command-log adapter around platform transactional storage; persist versioned command ids and app payload before projection; restore projection after reload; expose queue state, export/delete/reset, quota/corruption failure, and migration refusal.
 - **Refusals:** no server reconciliation, CRDT, mandatory IndexedDB, credential storage, or policy hidden in core.
 - **Requirements:** `local/001-004`, `sync/009`, `sync/014-015`, `sync/020`, `security/007`, `performance/005-006`.
-- **Proof:** `cargo test -p hemx-wasm --test browser kanban_command_persists_before_projection_and_restores_after_reload -- --exact` proves transactional persist-before-project ordering, app-owned shell caching, current-version replay after a real Firefox reload with the fixture server unreachable, stable identity metadata, and explicit unknown-schema refusal through real WASM. `cargo test -p hemx-wasm --test browser kanban_command_export_delete_and_reset_are_recoverable -- --exact` proves accessible export/delete/reset entry points, versioned credential-free export, confirmation before destructive actions, preserved identity after queue deletion, and fresh identity/baseline projection after reset. `cargo test -p hemx-wasm --test browser kanban_persistence_failure_does_not_project_and_recovers -- --exact` proves transactional failure does not project or emit `kanban:command-persisted`, reports non-payload stage/code diagnostics, and recovers through the ordinary deletion path. `cargo test -p hemx-wasm --test browser kanban_corrupt_command_refuses_projection_and_recovers -- --exact` proves strict current-schema validation, no partial projection, visible non-payload diagnostics, raw versioned export for recovery, and ordinary deletion recovery. `cargo test -p hemx-wasm --test browser kanban_replay_is_bounded_and_within_budget -- --exact` proves 64-command preflight/replay within the 100 ms browser budget, zero partial projection at 65 commands, and export/delete recovery. The completed slice proof must additionally cover quota specifically.
+- **Proof:** `cargo test -p hemx-wasm --test browser kanban_command_persists_before_projection_and_restores_after_reload -- --exact` proves transactional persist-before-project ordering, app-owned shell caching, current-version replay after a real Firefox reload with the fixture server unreachable, stable identity metadata, and explicit unknown-schema refusal through real WASM. `cargo test -p hemx-wasm --test browser kanban_command_export_delete_and_reset_are_recoverable -- --exact` proves accessible export/delete/reset entry points, versioned credential-free export, confirmation before destructive actions, preserved identity after queue deletion, and fresh identity/baseline projection after reset. `cargo test -p hemx-wasm --test browser kanban_persistence_failure_does_not_project_and_recovers -- --exact` proves transactional failure does not project or emit `kanban:command-persisted`, reports non-payload stage/code diagnostics, and recovers through the ordinary deletion path. `cargo test -p hemx-wasm --test browser kanban_corrupt_command_refuses_projection_and_recovers -- --exact` proves strict current-schema validation, no partial projection, visible non-payload diagnostics, raw versioned export for recovery, and ordinary deletion recovery. `cargo test -p hemx-wasm --test browser kanban_replay_is_bounded_and_within_budget -- --exact` proves 64-command preflight/replay within the 100 ms browser budget, zero partial projection at 65 commands, and export/delete recovery. `cargo test -p hemx-wasm --test browser kanban_quota_failure_is_fail_closed_and_recoverable -- --exact` proves quota-specific fail-closed behavior, no false durability event, transactional metadata rollback, and export/delete/reset recovery. The completed slice proof must additionally show visible queued status within the `performance/006` latency budget.
 
 ## Slice 4 — authoritative reconnect and convergence
 

examples/kanban/static/command-log.js

@@ -67,6 +67,10 @@ function stableSession() {
 async function appendReorder(database, wire) {
   const transaction = database.transaction([COMMANDS, META], "readwrite");
   const done = completed(transaction);
+  const completion = done.then(
+    () => null,
+    (error) => error,
+  );
   const meta = transaction.objectStore(META);
   const commands = transaction.objectStore(COMMANDS);
   const actorRequest = result(meta.get("actor"));
@@ -85,14 +89,18 @@ async function appendReorder(database, wire) {
     eventKind: String(wire[1] || "click"),
     key: wire[4] ? String(wire[4]) : null,
   };
+  let append;
+  let counted;
+  try {
     meta.put(actor, "actor");
     meta.put(causal, "causal");
-  const append = result(commands.add(command));
+    append = result(commands.add(command));
-  const counted = result(commands.count());
+    counted = result(commands.count());
-  const completion = done.then(
+  } catch (error) {
-    () => null,
+    transaction.abort();
-    (error) => error,
+    await completion;
-  );
+    throw error;
+  }
   try {
     const [, count] = await Promise.all([append, counted]);
     const transactionError = await completion;

hemx-wasm/tests/browser.rs

@@ -528,6 +528,149 @@ async fn kanban_command_export_delete_and_reset_are_recoverable() -> WebDriverRe
     result.and(quit)
 }
 
+#[tokio::test]
+async fn kanban_quota_failure_is_fail_closed_and_recoverable() -> WebDriverResult<()> {
+    // test req: sync/015
+    let workspace = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
+        .parent()
+        .expect("workspace root")
+        .to_owned();
+    let (package, bootstrap, rendered) = build_kanban_artifact(&workspace);
+    let runtime = workspace.join("hemx-js/runtime/hemx.js");
+    let server = StaticServer::start(
+        package,
+        runtime,
+        bootstrap,
+        rendered,
+        "kanban_client",
+        Some(kanban_app_assets(&workspace)),
+    );
+
+    let webdriver_port = available_port();
+    let webdriver_addr = format!("127.0.0.1:{webdriver_port}");
+    let mut webdriver = Command::new("geckodriver");
+    webdriver.arg("--port").arg(webdriver_port.to_string());
+    let _webdriver = ProcessGuard::start(webdriver, &webdriver_addr);
+    let mut caps = DesiredCapabilities::firefox();
+    caps.set_headless()?;
+    let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?;
+    let result = async {
+        driver.goto(&server.url()).await?;
+        wait_until(
+            &driver,
+            "const root = document.querySelector('[data-hemx-root]'); return root.hasAttribute('data-kanban-command-ready') && root.hasAttribute('data-kanban-offline-ready')",
+        )
+        .await?;
+        inject_quota_failure(&driver).await?;
+        driver
+            .execute(
+                "const root = document.querySelector('[data-hemx-root]'); window.__persistedAfterQuota = false; window.__quotaFailure = null; root.addEventListener('kanban:command-persisted', () => { window.__persistedAfterQuota = true; }, { once: true }); root.addEventListener('kanban:command-error', (event) => { window.__quotaFailure = event.detail; }, { once: true }); return true;",
+                Vec::new(),
+            )
+            .await?;
+        driver.find(By::Css("[data-card-id='1']")).await?.click().await?;
+        wait_until(&driver, "return window.__quotaFailure !== null").await?;
+        let failed = driver
+            .execute(
+                "const root = document.querySelector('[data-hemx-root]'); return { order: [...root.querySelectorAll('[data-key]')].map((node) => node.dataset.key).join('|'), count: root.getAttribute('data-kanban-command-count'), stage: root.getAttribute('data-kanban-command-error-stage'), code: root.getAttribute('data-kanban-command-error-code'), notice: root.querySelector('[role=status]').textContent, persisted: window.__persistedAfterQuota, controls: [...root.querySelectorAll('[data-kanban-command-action]')].map((button) => ({ action: button.dataset.kanbanCommandAction, disabled: button.disabled })) }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(failed["order"], "1|2");
+        assert_eq!(failed["count"], "0");
+        assert_eq!(failed["stage"], "persist");
+        assert_eq!(failed["code"], "QuotaExceededError");
+        assert_eq!(failed["persisted"], false);
+        assert_eq!(
+            failed["notice"],
+            "Local command persist failed (QuotaExceededError). Recovery controls remain available."
+        );
+        assert!(
+            failed["controls"]
+                .as_array()
+                .is_some_and(|controls| controls.len() == 3
+                    && controls.iter().all(|control| control["disabled"] == false)),
+            "recovery controls unavailable: {failed}"
+        );
+        let empty_export = export_commands(&driver).await?.json().clone();
+        assert_eq!(empty_export["commands"].as_array().map(Vec::len), Some(0));
+
+        driver
+            .execute("window.__reloadPending = true", Vec::new())
+            .await?;
+        let delete = driver
+            .find(By::Css("[data-kanban-command-action='delete']"))
+            .await?;
+        delete.click().await?;
+        delete.click().await?;
+        wait_until(
+            &driver,
+            "const root = document.querySelector('[data-hemx-root]'); return !window.__reloadPending && root.hasAttribute('data-kanban-command-ready') && root.getAttribute('data-kanban-command-count') === '0'",
+        )
+        .await?;
+        driver.find(By::Css("[data-card-id='1']")).await?.click().await?;
+        wait_until(
+            &driver,
+            "return document.querySelector('[data-hemx-root]').getAttribute('data-kanban-command-count') === '1'",
+        )
+        .await?;
+        let after_delete = export_commands(&driver).await?.json().clone();
+        assert_eq!(after_delete["commands"].as_array().map(Vec::len), Some(1));
+        assert_eq!(after_delete["commands"][0]["causal"], 1);
+
+        inject_quota_failure(&driver).await?;
+        driver
+            .execute(
+                "window.__quotaFailure = null; document.querySelector('[data-hemx-root]').addEventListener('kanban:command-error', (event) => { window.__quotaFailure = event.detail; }, { once: true }); return true;",
+                Vec::new(),
+            )
+            .await?;
+        driver.find(By::Css("[data-card-id='2']")).await?.click().await?;
+        wait_until(&driver, "return window.__quotaFailure !== null").await?;
+        let second_failure = driver
+            .execute(
+                "const root = document.querySelector('[data-hemx-root]'); return { order: [...root.querySelectorAll('[data-key]')].map((node) => node.dataset.key).join('|'), count: root.getAttribute('data-kanban-command-count'), code: root.getAttribute('data-kanban-command-error-code') }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(second_failure["order"], "2|1");
+        assert_eq!(second_failure["count"], "1");
+        assert_eq!(second_failure["code"], "QuotaExceededError");
+
+        driver
+            .execute("window.__reloadPending = true", Vec::new())
+            .await?;
+        let reset = driver
+            .find(By::Css("[data-kanban-command-action='reset']"))
+            .await?;
+        reset.click().await?;
+        reset.click().await?;
+        wait_until(
+            &driver,
+            "const root = document.querySelector('[data-hemx-root]'); return !window.__reloadPending && root.hasAttribute('data-kanban-command-ready') && root.hasAttribute('data-kanban-offline-ready') && root.getAttribute('data-kanban-command-count') === '0'",
+        )
+        .await?;
+        let after_reset = driver
+            .execute(
+                "const root = document.querySelector('[data-hemx-root]'); return { order: [...root.querySelectorAll('[data-key]')].map((node) => node.dataset.key).join('|'), error: root.getAttribute('data-kanban-command-error') }",
+                Vec::new(),
+            )
+            .await?
+            .json()
+            .clone();
+        assert_eq!(after_reset["order"], "1|2");
+        assert!(after_reset["error"].is_null());
+        Ok(())
+    }
+    .await;
+    let quit = driver.quit().await;
+    result.and(quit)
+}
+
 #[tokio::test]
 async fn kanban_persistence_failure_does_not_project_and_recovers() -> WebDriverResult<()> {
     // test req: sync/015
@@ -1260,6 +1403,29 @@ fn fixture_html(rendered: &str, has_app_module: bool) -> String {
     )
 }
 
+async fn inject_quota_failure(driver: &WebDriver) -> WebDriverResult<()> {
+    let injected = driver
+        .execute(
+            r#"
+            if (!window.__kanbanOriginalAdd) window.__kanbanOriginalAdd = IDBObjectStore.prototype.add;
+            IDBObjectStore.prototype.add = function(value) {
+              if (this.name === 'commands' && value && value.kind === 'reorder_card') {
+                IDBObjectStore.prototype.add = window.__kanbanOriginalAdd;
+                throw new DOMException('Injected storage quota exhaustion', 'QuotaExceededError');
+              }
+              return window.__kanbanOriginalAdd.call(this, value);
+            };
+            return IDBObjectStore.prototype.add !== window.__kanbanOriginalAdd;
+            "#,
+            Vec::new(),
+        )
+        .await?
+        .json()
+        .clone();
+    assert_eq!(injected, true, "failed to inject quota error");
+    Ok(())
+}
+
 async fn store_replay_commands(driver: &WebDriver, first: u64, last: u64) -> WebDriverResult<()> {
     let stored = driver
         .execute_async(