feat(kanban): retry explicit keep-local choice

req: sync/009

req: sync/010

req: sync/011

req: sync/016
This commit is contained in:
slhx agent
2026-07-13 19:47:39 +02:00
parent 54301c6aa5
commit fbc0c9db30
4 changed files with 304 additions and 8 deletions
+2 -2
View File
@@ -42,11 +42,11 @@ encryption, retention, backup, and deployment policy remain host concerns.
## Slice 4 — authoritative reconnect and convergence ## Slice 4 — authoritative reconnect and convergence
- [ ] **User value:** offline and concurrent work reconnects without duplicate mutation, silent loss, stale authorization, or ambiguous conflict. - [ ] **User value:** offline and concurrent work reconnects without duplicate mutation, silent loss, stale authorization, or ambiguous conflict.
- **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Each activation serializes uploads with one in flight, processes at most two acknowledged commands, exposes the retained durable count when backpressured, and resumes the next bounded run only through the visible retry action. A mixed queue commits and removes its accepted prefix exactly once, then stops on the first permanent rejection with the typed server cause visible, the rejected command plus untouched suffix durable, and blind retry disabled. The IndexedDB v1-to-v2 command migration transactionally adds the explicit target column, records a typed migration receipt, preserves causal order and interaction intent through an interrupted upload, and later drains in the original order. Every command POST now derives its principal, permission, and tenant from the current same-origin session before idempotency lookup or mutation; authenticated mode accepts only app-configured opaque session tokens and fails closed when signed out. Direct cross-tenant and stale-permission command attempts receive typed authorization denial before mutation. IndexedDB v3 additionally indexes every command and lease by the current server-derived tenant/principal partition: switched users enumerate zero foreign commands, issue no foreign replay, cannot export foreign payloads, and the owning account can export then resume its intact queue. Signed-out context lookup fails closed before opening the queue. Canonical snapshots and replayable acknowledgement history now derive the same current session, filter card state and events by tenant before serialization, use tenant-local sequence cursors, and reject signed-out reads. A divergent snapshot enables one explicit “Use canonical state and continue” action: it removes only the conflicted head, records that choice, retains the ordered suffix, releases uploader ownership, and resumes the suffix through the ordinary bounded sync path. Broader domain-specific merge choices remain. - **State:** In progress — one app-owned `move_card` server command validates a durable client command id, applies the authoritative canonical column once, returns the same acknowledgement for an identical retry, rejects id reuse with a different payload, assigns one server sequence, and redelivers that canonical acknowledgement after a real EventSource disconnect/reconnect. Canonical acknowledgements and the next sequence are durably stored in a strict versioned JSON envelope using fsync plus atomic replacement; startup refuses malformed/unknown state, rebuilds the canonical board, and preserves idempotency and event replay across a real process restart. A dedicated opt-in sync route reads one pending IndexedDB command, retries transient failures with capped exponential backoff and randomized jitter, exposes online/offline state plus an accessible manual retry after exhaustion, waits for canonical acknowledgement on the reconnecting transport, and only then removes the durable command; later retry converges without a new local mutation. Canonical payload conflicts are not retried and remain durable with a visible reason. If requested history predates retained events, the stream emits a typed snapshot-required event and the client loads a versioned canonical snapshot. One deterministic rebase rule treats `reorder_card` as converged only when the canonical snapshot already places that card in `done`; it then atomically stores the snapshot/cursor and removes the satisfied command. If a later canonical command instead places the same card in `doing`, the rebase is explicitly `conflicted`, retains the local command and last committed snapshot/cursor unchanged, and exposes the divergent canonical column/reason. Two same-origin tabs coordinate an app-owned expiring IndexedDB lease so only one uploads; the standby exposes its role without issuing a request, and after the leader closes it takes over, receives one canonical acknowledgement/sequence, and removes the queue once. Each activation serializes uploads with one in flight, processes at most two acknowledged commands, exposes the retained durable count when backpressured, and resumes the next bounded run only through the visible retry action. A mixed queue commits and removes its accepted prefix exactly once, then stops on the first permanent rejection with the typed server cause visible, the rejected command plus untouched suffix durable, and blind retry disabled. The IndexedDB v1-to-v2 command migration transactionally adds the explicit target column, records a typed migration receipt, preserves causal order and interaction intent through an interrupted upload, and later drains in the original order. Every command POST now derives its principal, permission, and tenant from the current same-origin session before idempotency lookup or mutation; authenticated mode accepts only app-configured opaque session tokens and fails closed when signed out. Direct cross-tenant and stale-permission command attempts receive typed authorization denial before mutation. IndexedDB v3 additionally indexes every command and lease by the current server-derived tenant/principal partition: switched users enumerate zero foreign commands, issue no foreign replay, cannot export foreign payloads, and the owning account can export then resume its intact queue. Signed-out context lookup fails closed before opening the queue. Canonical snapshots and replayable acknowledgement history now derive the same current session, filter card state and events by tenant before serialization, use tenant-local sequence cursors, and reject signed-out reads. A divergent snapshot enables explicit “Use canonical state and “Keep local change” actions. Canonical resolution removes only the conflicted head before resuming the ordered suffix. Keep-local resolution derives one deterministic retry identity from the original command and snapshot, keeps the original head and suffix durable through permanent rejection and bounded transport failure, re-enables the explicit decision after rejection, and removes the head only after canonical acceptance before resuming the suffix. Broader domain-specific merge choices remain.
- **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics. - **Build:** materialize `hemx-sync` over an integration transport with idempotent server command processing, snapshot/change cursor, durable acknowledgements, bounded ordered replay, current auth checks, rejection/conflict results, canonical replacement, reconnect jitter/backoff, multi-tab coordination, and redacted diagnostics.
- **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy. - **Refusals:** no default CRDT, transport in core, cached enqueue-time permission, unbounded queue, or silent last-write-wins policy.
- **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`. - **Requirements:** `sync/001-023`, `operations/001-005`, `security/002-005`, `performance/004-005`.
- **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_and_user_conflict_resolution_preserve_suffix -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit, then a divergent canonical update producing explicit conflict with a two-command queue; the users canonical-state action removes only the conflicted head, retains and resumes the suffix, and reaches canonical convergence with an empty queue. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. `cargo test -p hemx-kanban-example --test browser_e2e mixed_queue_removes_accepted_prefix_and_retains_rejected_tail -- --exact` proves an accepted prefix is canonically applied and removed once before a permanent rejection stops processing, exposes its typed HTTP/server cause, disables blind retry, and leaves both the rejected command and untouched suffix durable. `cargo test -p hemx-kanban-example --test browser_e2e schema_upgrade_preserves_queued_order_and_local_intent -- --exact` proves a three-command v1 queue migrates atomically to the explicit-target v2 schema, remains byte-for-intent ordered after interrupted upload, then receives canonical sequences 13 in original order and drains without loss. `cargo test -p hemx-kanban-example --test browser_e2e account_partition_hides_replay_and_export_until_owner_returns -- --exact` proves a beta editor and alpha viewer enumerate zero commands and issue no replay for an alpha owner queue, signed-out startup cannot open a partition, no foreign id/export surface leaks, and only the returning alpha owner can export the intact command then resume it exactly once. `cargo test -p hemx-kanban-example --test browser_e2e canonical_snapshot_and_history_are_tenant_scoped -- --exact` proves alpha and beta sessions receive only their own canonical cards and acknowledgement events with independent sequence 1 cursors, while signed-out snapshot/history reads receive 401 and expose no stream. The completed slice proof must additionally cover broader domain-specific merge choices. - **Proof:** `cargo test -p hemx-kanban-example --test browser_e2e idempotent_server_command_is_acknowledged_after_reconnect -- --exact` proves duplicate POST delivery yields one identical canonical acknowledgement/sequence, conflicting id reuse is rejected, EventSource reconnects after a server-closed first stream, the acknowledgement is delivered once with its sequence as event id, and a page reload shows the authoritative card in the canonical column. `cargo test -p hemx-kanban-example --test browser_e2e pending_local_command_uploads_with_bounded_retry_and_is_removed_on_ack -- --exact` proves automatic platform-store upload, one explicit transient failure, bounded exponential backoff plus jitter, reconnect acknowledgement, pending-before-ack ordering, acknowledged removal, canonical board convergence, and non-retried 409 rejection remaining durable with a visible reason. `cargo test -p hemx-kanban-example --test browser_e2e canonical_acknowledgement_survives_server_restart -- --exact` proves the versioned store is materialized before success, a real process restart reloads the same idempotent acknowledgement/sequence, EventSource replays it by id, and canonical board state is rebuilt. `cargo test -p hemx-kanban-example --test browser_e2e exhausted_offline_retries_keep_command_until_later_reconnect -- --exact` proves three bounded retries exhaust into visible offline/manual-recovery state while the command remains durable, then a later retry acknowledges/removes it and converges canonically. `cargo test -p hemx-kanban-example --test browser_e2e missing_history_rebase_and_user_conflict_resolution_preserve_suffix -- --exact` proves retained-history gap detection, typed/versioned snapshot fallback, deterministic already-canonical convergence, atomic snapshot/cursor commit, then a divergent canonical update producing explicit conflict with a two-command queue; the users canonical-state action removes only the conflicted head, retains and resumes the suffix, and reaches canonical convergence with an empty queue. `cargo test -p hemx-kanban-example --test browser_e2e keep_local_retry_preserves_conflicted_command_and_suffix_order -- --exact` proves the keep-local action derives one resolution identity; a permanent rejection preserves order and re-enables the choice; three later transport failures still preserve the original conflicted head plus suffix and expose manual recovery; final acceptance removes only the head and converges the untouched suffix without duplication or loss. `cargo test -p hemx-kanban-example --test browser_e2e two_tabs_coordinate_single_uploader_and_takeover_without_duplicate_application -- --exact` proves one retry-exhausted leader/one explicit standby, zero follower upload before takeover, lease takeover after the leader closes, one canonical sequence/event, one queue removal, and one board application. `cargo test -p hemx-kanban-example --test browser_e2e upload_backpressure_keeps_pending_work_visible_and_recoverable -- --exact` proves one in-flight upload, a two-acknowledgement activation limit, one retained durable command with visible recovery state, and explicit retry draining the final command without loss. `cargo test -p hemx-kanban-example --test browser_e2e mixed_queue_removes_accepted_prefix_and_retains_rejected_tail -- --exact` proves an accepted prefix is canonically applied and removed once before a permanent rejection stops processing, exposes its typed HTTP/server cause, disables blind retry, and leaves both the rejected command and untouched suffix durable. `cargo test -p hemx-kanban-example --test browser_e2e schema_upgrade_preserves_queued_order_and_local_intent -- --exact` proves a three-command v1 queue migrates atomically to the explicit-target v2 schema, remains byte-for-intent ordered after interrupted upload, then receives canonical sequences 13 in original order and drains without loss. `cargo test -p hemx-kanban-example --test browser_e2e account_partition_hides_replay_and_export_until_owner_returns -- --exact` proves a beta editor and alpha viewer enumerate zero commands and issue no replay for an alpha owner queue, signed-out startup cannot open a partition, no foreign id/export surface leaks, and only the returning alpha owner can export the intact command then resume it exactly once. `cargo test -p hemx-kanban-example --test browser_e2e canonical_snapshot_and_history_are_tenant_scoped -- --exact` proves alpha and beta sessions receive only their own canonical cards and acknowledgement events with independent sequence 1 cursors, while signed-out snapshot/history reads receive 401 and expose no stream. The completed slice proof must additionally cover broader domain-specific merge choices.
## Slice 5 — local-first multiplayer Kanban milestone ## Slice 5 — local-first multiplayer Kanban milestone
+59 -6
View File
@@ -22,6 +22,7 @@ let uploadedTotal = 0;
let inFlightUploads = 0; let inFlightUploads = 0;
let maxObservedInFlight = 0; let maxObservedInFlight = 0;
let activeConflict; let activeConflict;
let manualRetryCommand;
let stopped = false; let stopped = false;
class UploadError extends Error { class UploadError extends Error {
@@ -148,7 +149,7 @@ async function commitConvergedRebase(database, snapshot, command) {
const done = transactionDone(transaction); const done = transactionDone(transaction);
transaction.objectStore("meta").put(snapshot, "canonicalSnapshot"); transaction.objectStore("meta").put(snapshot, "canonicalSnapshot");
transaction.objectStore("meta").put(snapshot.serverSequence, "acknowledgementCursor"); transaction.objectStore("meta").put(snapshot.serverSequence, "acknowledgementCursor");
transaction.objectStore(COMMANDS).delete(command.id); transaction.objectStore(COMMANDS).delete(command.queueCommandId || command.id);
await done; await done;
} }
@@ -181,6 +182,28 @@ function setExportAvailable(available) {
function setConflictResolutionAvailable(available) { function setConflictResolutionAvailable(available) {
root.querySelector("[data-sync-use-canonical]").disabled = !available; root.querySelector("[data-sync-use-canonical]").disabled = !available;
root.querySelector("[data-sync-keep-local]").disabled = !available;
}
async function keepLocalChange() {
if (!activeConflict) return;
const { command, snapshot } = activeConflict;
const retryCommand = {
...command,
id: `${command.id}:keep:${snapshot.serverSequence}`,
queueCommandId: command.id,
conflictResolution: "keep-local-change",
basedOnServerSequence: snapshot.serverSequence,
};
setConflictResolutionAvailable(false);
root.setAttribute("data-sync-conflict-resolution", "keep-local-pending");
root.setAttribute("data-sync-resolution-command-id", retryCommand.id);
root.setAttribute("data-sync-resolved-command-id", command.id);
synchronizing = false;
clearTimeout(leaseTimer);
await releaseUploaderLease(database);
root.setAttribute("data-sync-leader", "false");
await synchronize(retryCommand);
} }
async function useCanonicalState() { async function useCanonicalState() {
@@ -218,6 +241,7 @@ async function exportPendingWork() {
function scheduleManualRetry(command, error) { function scheduleManualRetry(command, error) {
clearTimeout(retryTimer); clearTimeout(retryTimer);
manualRetryCommand = command;
root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error)); root.setAttribute("data-sync-error", error instanceof Error ? error.message : String(error));
setManualRetryAvailable(true); setManualRetryAvailable(true);
setPhase("offline", "Sync is offline after bounded retries; the durable command remains queued. Retry now when ready."); setPhase("offline", "Sync is offline after bounded retries; the durable command remains queued. Retry now when ready.");
@@ -330,7 +354,15 @@ async function synchronize(command) {
if (canonical.commandId !== command.id) return; if (canonical.commandId !== command.id) return;
source.close(); source.close();
root.setAttribute("data-sync-pending-before-ack", String((await pendingCommands(database)).length)); root.setAttribute("data-sync-pending-before-ack", String((await pendingCommands(database)).length));
await removePendingCommand(database, command.id); const queueCommandId = command.queueCommandId || command.id;
await removePendingCommand(database, queueCommandId);
manualRetryCommand = undefined;
if (command.conflictResolution === "keep-local-change") {
activeConflict = undefined;
setConflictResolutionAvailable(false);
root.setAttribute("data-sync-conflict-resolution", "kept-local-change");
root.setAttribute("data-sync-resolved-command-id", queueCommandId);
}
uploadsThisRun += 1; uploadsThisRun += 1;
uploadedTotal += 1; uploadedTotal += 1;
root.setAttribute("data-sync-uploaded-this-run", String(uploadsThisRun)); root.setAttribute("data-sync-uploaded-this-run", String(uploadsThisRun));
@@ -356,6 +388,7 @@ async function synchronize(command) {
const snapshot = await response.json(); const snapshot = await response.json();
const queued = await pendingCommands(database); const queued = await pendingCommands(database);
const decision = decideRebase(snapshot, command); const decision = decideRebase(snapshot, command);
const converged = decision.kind === "converged";
root.setAttribute("data-sync-snapshot-sequence", String(snapshot.serverSequence)); root.setAttribute("data-sync-snapshot-sequence", String(snapshot.serverSequence));
root.setAttribute("data-sync-snapshot-schema", String(snapshot.schemaVersion)); root.setAttribute("data-sync-snapshot-schema", String(snapshot.schemaVersion));
root.setAttribute("data-sync-snapshot-card-count", String(snapshot.cards.length)); root.setAttribute("data-sync-snapshot-card-count", String(snapshot.cards.length));
@@ -363,10 +396,15 @@ async function synchronize(command) {
root.setAttribute("data-sync-rebase-decision", decision.kind); root.setAttribute("data-sync-rebase-decision", decision.kind);
root.setAttribute("data-sync-rebase-reason", decision.reason); root.setAttribute("data-sync-rebase-reason", decision.reason);
root.setAttribute("data-sync-canonical-column", decision.canonicalColumn); root.setAttribute("data-sync-canonical-column", decision.canonicalColumn);
if (decision.kind === "converged") { if (converged) {
activeConflict = undefined; activeConflict = undefined;
manualRetryCommand = undefined;
setConflictResolutionAvailable(false); setConflictResolutionAvailable(false);
await commitConvergedRebase(database, snapshot, command); await commitConvergedRebase(database, snapshot, command);
if (command.conflictResolution === "keep-local-change") {
root.setAttribute("data-sync-conflict-resolution", "kept-local-change");
root.setAttribute("data-sync-resolved-command-id", command.queueCommandId);
}
root.setAttribute("data-sync-pending-count", String((await pendingCommands(database)).length)); root.setAttribute("data-sync-pending-count", String((await pendingCommands(database)).length));
root.setAttribute("data-sync-ack-sequence", String(snapshot.serverSequence)); root.setAttribute("data-sync-ack-sequence", String(snapshot.serverSequence));
setPhase("rebased", `Canonical snapshot ${snapshot.serverSequence} already satisfies ${command.id}; committed and removed the pending command.`); setPhase("rebased", `Canonical snapshot ${snapshot.serverSequence} already satisfies ${command.id}; committed and removed the pending command.`);
@@ -382,6 +420,7 @@ async function synchronize(command) {
clearTimeout(leaseTimer); clearTimeout(leaseTimer);
await releaseUploaderLease(database); await releaseUploaderLease(database);
root.setAttribute("data-sync-leader", "false"); root.setAttribute("data-sync-leader", "false");
if (converged) await continuePendingWork();
}); });
} catch (error) { } catch (error) {
synchronizing = false; synchronizing = false;
@@ -392,7 +431,15 @@ async function synchronize(command) {
root.setAttribute("data-sync-error-status", String(error.status)); root.setAttribute("data-sync-error-status", String(error.status));
const remaining = await pendingCommands(database); const remaining = await pendingCommands(database);
setManualRetryAvailable(false); setManualRetryAvailable(false);
if (error.kind === "authorization-denial") { if (command.conflictResolution === "keep-local-change") {
manualRetryCommand = undefined;
setConflictResolutionAvailable(true);
root.setAttribute("data-sync-error-kind", error.kind);
root.setAttribute("data-sync-error-reason", error.reason);
root.setAttribute("data-sync-pending-count", String(remaining.length));
root.setAttribute("data-sync-conflict-resolution", "keep-local-rejected");
setPhase("resolution-rejected", `Keep-local command ${command.id} was rejected (${error.status}: ${error.reason}); the conflicted command and ${remaining.length - 1} queued suffix command${remaining.length === 2 ? "" : "s"} remain in order.`);
} else if (error.kind === "authorization-denial") {
root.setAttribute("data-sync-error-kind", "authorization-denial"); root.setAttribute("data-sync-error-kind", "authorization-denial");
root.setAttribute("data-sync-pending-count", "redacted"); root.setAttribute("data-sync-pending-count", "redacted");
root.setAttribute("data-sync-redacted-pending", "true"); root.setAttribute("data-sync-redacted-pending", "true");
@@ -469,6 +516,10 @@ async function start() {
} }
const command = validatePending(commands[0]); const command = validatePending(commands[0]);
root.addEventListener("click", async (event) => { root.addEventListener("click", async (event) => {
if (event.target.closest("[data-sync-keep-local]")) {
await keepLocalChange();
return;
}
if (event.target.closest("[data-sync-use-canonical]")) { if (event.target.closest("[data-sync-use-canonical]")) {
await useCanonicalState(); await useCanonicalState();
return; return;
@@ -482,13 +533,15 @@ async function start() {
root.setAttribute("data-sync-uploaded-this-run", "0"); root.setAttribute("data-sync-uploaded-this-run", "0");
setManualRetryAvailable(false); setManualRetryAvailable(false);
const [next] = await pendingCommands(database); const [next] = await pendingCommands(database);
if (next) synchronize(validatePending(next)).catch(failPermanently); const retry = manualRetryCommand || (next && validatePending(next));
if (retry) synchronize(retry).catch(failPermanently);
}); });
window.addEventListener("online", async () => { window.addEventListener("online", async () => {
if (root.getAttribute("data-sync-phase") !== "offline") return; if (root.getAttribute("data-sync-phase") !== "offline") return;
setManualRetryAvailable(false); setManualRetryAvailable(false);
const [next] = await pendingCommands(database); const [next] = await pendingCommands(database);
if (next) synchronize(validatePending(next)).catch(failPermanently); const retry = manualRetryCommand || (next && validatePending(next));
if (retry) synchronize(retry).catch(failPermanently);
}); });
await runLeaseLoop(command); await runLeaseLoop(command);
} }
@@ -12,6 +12,7 @@
<button type="button" data-sync-retry>Retry sync now</button> <button type="button" data-sync-retry>Retry sync now</button>
<button type="button" data-sync-export disabled>Export this account's queue</button> <button type="button" data-sync-export disabled>Export this account's queue</button>
<button type="button" data-sync-use-canonical disabled>Use canonical state and continue</button> <button type="button" data-sync-use-canonical disabled>Use canonical state and continue</button>
<button type="button" data-sync-keep-local disabled>Keep local change and continue</button>
</section> </section>
<script +src="self.runtime_src" defer></script> <script +src="self.runtime_src" defer></script>
<script src="/sync.js" defer></script> <script src="/sync.js" defer></script>
+242
View File
@@ -1704,6 +1704,248 @@ async fn missing_history_rebase_and_user_conflict_resolution_preserve_suffix() -
result.and(quit) result.and(quit)
} }
#[tokio::test]
async fn keep_local_retry_preserves_conflicted_command_and_suffix_order() -> WebDriverResult<()> {
// test req: sync/009 req: sync/010 req: sync/011 req: sync/016
let app_port = available_port();
let app_addr = format!("127.0.0.1:{app_port}");
let mut app_command = Command::new(env!("CARGO_BIN_EXE_hemx-kanban-example"));
app_command
.env("HEMX_KANBAN_ADDR", &app_addr)
.env("HEMX_KANBAN_RETAINED_AFTER", "1");
let _app = TestProcess::start(app_command, "hemx-kanban", &app_addr, STARTUP_TIMEOUT)
.expect("start ready hemx-kanban");
let webdriver_port = available_port();
let webdriver_addr = format!("127.0.0.1:{webdriver_port}");
let mut webdriver = Command::new("geckodriver");
webdriver.arg("--port").arg(webdriver_port.to_string());
let _webdriver = TestProcess::start(webdriver, "geckodriver", &webdriver_addr, STARTUP_TIMEOUT)
.expect("start ready geckodriver");
let mut caps = DesiredCapabilities::firefox();
caps.set_headless()?;
let driver = WebDriver::new(&format!("http://{webdriver_addr}"), caps).await?;
let result = async {
driver.goto(&format!("http://{app_addr}/")).await?;
let seeded_server = driver
.execute_async(
r#"
const done = arguments[arguments.length - 1];
const command = (id, card, column = 'done') => fetch(`/sync/commands?command_id=${encodeURIComponent(id)}&card_id=${card}&column=${column}`, { method: 'POST' });
(async () => {
const statuses = [];
statuses.push((await command('keep:1', 1)).status);
statuses.push((await command('keep:2', 2)).status);
statuses.push((await command('keep:3', 2)).status);
statuses.push((await command('keep:external', 2, 'doing')).status);
done({ statuses });
})().catch((error) => done({ error: String(error) }));
"#,
Vec::new(),
)
.await?
.json()
.clone();
assert_eq!(seeded_server["statuses"], serde_json::json!([200, 200, 200, 200]));
let seeded_local = driver
.execute_async(
r#"
const done = arguments[arguments.length - 1];
const open = indexedDB.open('hemx-kanban-v1', 3);
open.onupgradeneeded = () => {
const database = open.result;
const commands = database.createObjectStore('commands', { keyPath: 'id' });
commands.createIndex('byAccountPartition', 'accountPartition');
database.createObjectStore('meta');
};
open.onsuccess = () => {
const tx = open.result.transaction('commands', 'readwrite');
const commands = tx.objectStore('commands');
commands.add({
id: 'keep:3', schemaVersion: 2, accountPartition: 'demo:demo', actor: 'keep', session: 'keep-session',
causal: 3, kind: 'reorder_card', cardId: '2', targetColumn: 'done', eventKind: 'click', key: null,
});
commands.add({
id: 'keep:4', schemaVersion: 2, accountPartition: 'demo:demo', actor: 'keep', session: 'keep-session',
causal: 4, kind: 'reorder_card', cardId: '1', targetColumn: 'done', eventKind: 'click', key: null,
});
tx.oncomplete = () => done({ seeded: true });
tx.onabort = () => done({ error: tx.error && tx.error.name });
};
"#,
Vec::new(),
)
.await?
.json()
.clone();
assert_eq!(seeded_local["seeded"], true);
driver.goto(&format!("http://{app_addr}/sync-demo")).await?;
wait_until(
&driver,
"return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'conflicted'",
)
.await?;
driver
.execute(
r#"
window.__keepRejections = 0;
window.__keepFailures = 0;
const originalFetch = window.fetch.bind(window);
window.fetch = (input, init = {}) => {
const url = new URL(typeof input === 'string' ? input : input.url, location.href);
if (url.pathname === '/sync/commands' && url.searchParams.get('command_id') === 'keep:3:keep:4') {
if (window.__keepRejections < 1) {
window.__keepRejections += 1;
return Promise.resolve(new Response(JSON.stringify({ kind: 'command-conflict', error: 'injected stale keep-local decision' }), {
status: 409,
headers: { 'content-type': 'application/json' },
}));
}
if (window.__keepFailures < 3) {
window.__keepFailures += 1;
return Promise.resolve(new Response(JSON.stringify({ kind: 'transport-failure', error: 'injected keep-local retry' }), {
status: 503,
headers: { 'content-type': 'application/json' },
}));
}
}
return originalFetch(input, init);
};
return true;
"#,
Vec::new(),
)
.await?;
driver
.find(By::Css("[data-sync-keep-local]"))
.await?
.click()
.await?;
wait_until(
&driver,
"return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'resolution-rejected'",
)
.await?;
let rejected = driver
.execute_async(
r#"
const done = arguments[arguments.length - 1];
const root = document.querySelector('[data-kanban-sync]');
const open = indexedDB.open('hemx-kanban-v1');
open.onsuccess = () => {
const request = open.result.transaction('commands', 'readonly').objectStore('commands').getAll();
request.onsuccess = () => done({
resolution: root.getAttribute('data-sync-conflict-resolution'),
resolutionDisabled: root.querySelector('[data-sync-keep-local]').disabled,
pending: root.getAttribute('data-sync-pending-count'),
rejections: window.__keepRejections,
commands: request.result.sort((left, right) => left.causal - right.causal).map(({ id, causal, cardId }) => ({ id, causal, cardId })),
});
};
"#,
Vec::new(),
)
.await?
.json()
.clone();
assert_eq!(rejected["resolution"], "keep-local-rejected");
assert_eq!(rejected["resolutionDisabled"], false);
assert_eq!(rejected["pending"], "2");
assert_eq!(rejected["rejections"], 1);
assert_eq!(
rejected["commands"],
serde_json::json!([
{ "id": "keep:3", "causal": 3, "cardId": "2" },
{ "id": "keep:4", "causal": 4, "cardId": "1" }
])
);
driver
.find(By::Css("[data-sync-keep-local]"))
.await?
.click()
.await?;
wait_until(
&driver,
"return document.querySelector('[data-kanban-sync]')?.getAttribute('data-sync-phase') === 'offline'",
)
.await?;
let retrying = driver
.execute_async(
r#"
const done = arguments[arguments.length - 1];
const root = document.querySelector('[data-kanban-sync]');
const open = indexedDB.open('hemx-kanban-v1');
open.onsuccess = () => {
const request = open.result.transaction('commands', 'readonly').objectStore('commands').getAll();
request.onsuccess = () => done({
phase: root.getAttribute('data-sync-phase'),
resolution: root.getAttribute('data-sync-conflict-resolution'),
resolutionCommand: root.getAttribute('data-sync-resolution-command-id'),
resolvedCommand: root.getAttribute('data-sync-resolved-command-id'),
attempts: root.getAttribute('data-sync-attempts'),
pending: root.getAttribute('data-sync-pending-count'),
manualRetry: root.getAttribute('data-sync-manual-retry'),
failures: window.__keepFailures,
commands: request.result.sort((left, right) => left.causal - right.causal).map(({ id, causal, cardId }) => ({ id, causal, cardId })),
});
};
"#,
Vec::new(),
)
.await?
.json()
.clone();
assert_eq!(retrying["phase"], "offline");
assert_eq!(retrying["resolution"], "keep-local-pending");
assert_eq!(retrying["resolutionCommand"], "keep:3:keep:4");
assert_eq!(retrying["resolvedCommand"], "keep:3");
assert_eq!(retrying["attempts"], "3");
assert_eq!(retrying["pending"], "2");
assert_eq!(retrying["manualRetry"], "available");
assert_eq!(retrying["failures"], 3);
assert_eq!(
retrying["commands"],
serde_json::json!([
{ "id": "keep:3", "causal": 3, "cardId": "2" },
{ "id": "keep:4", "causal": 4, "cardId": "1" }
])
);
driver.find(By::Css("[data-sync-retry]")).await?.click().await?;
wait_until(
&driver,
"const root = document.querySelector('[data-kanban-sync]'); return root?.getAttribute('data-sync-phase') === 'rebased' && root?.getAttribute('data-sync-pending-count') === '0'",
)
.await?;
let resolved = driver
.execute(
"const root = document.querySelector('[data-kanban-sync]'); return { resolution: root.getAttribute('data-sync-conflict-resolution'), resolvedCommand: root.getAttribute('data-sync-resolved-command-id'), pending: root.getAttribute('data-sync-pending-count'), ackSequence: root.getAttribute('data-sync-ack-sequence'), canonicalColumn: root.getAttribute('data-sync-canonical-column'), status: root.querySelector('[role=status]').textContent }",
Vec::new(),
)
.await?
.json()
.clone();
assert_eq!(resolved["resolution"], "kept-local-change");
assert_eq!(resolved["resolvedCommand"], "keep:3");
assert_eq!(resolved["pending"], "0");
assert_eq!(resolved["ackSequence"], "6");
assert_eq!(resolved["canonicalColumn"], "done");
assert_eq!(
resolved["status"],
"Canonical snapshot 6 already satisfies keep:4; committed and removed the pending command."
);
assert_eq!(command_count(&driver).await?, 0);
Ok(())
}
.await;
let quit = driver.quit().await;
result.and(quit)
}
#[tokio::test] #[tokio::test]
async fn canonical_acknowledgement_survives_server_restart() -> WebDriverResult<()> { async fn canonical_acknowledgement_survives_server_restart() -> WebDriverResult<()> {
// test req: sync/001 req: sync/005 req: sync/007 req: sync/008 req: sync/013 // test req: sync/001 req: sync/005 req: sync/007 req: sync/008 req: sync/013