docs(recipes): add observability flags boundary

Document provider-explicit observability, feature flag, A/B, and killswitch boundaries for hemx apps. The recipe keeps metrics providers and rollout controls in app/platform code while handlers return generated effects and the tiny runtime stays provider-free.

req: laws/002

req: laws/004

req: runtime/003

req: runtime/004

req: failure/004

req: examples/001

req: test/001

README.md

@@ -87,7 +87,7 @@ and integrate at explicit boundaries. req: laws/002 req: auth/001
   fields and credentials semantics. See `docs/recipes/auth-session-csrf.md`. req: auth/004 req: auth/005
 - **Observability, feature flags, killswitches, deploy:** use explicit platform
   integrations around handlers, routes, and runtime assets. Core hemx must not
-  vendor providers or add framework-specific magic. See `docs/recipes/deploy-versioning.md`.
+  vendor providers or add framework-specific magic. See `docs/recipes/observability-flags.md` and `docs/recipes/deploy-versioning.md`.
 - **PWA/offline/sync:** optional adapters may reuse generated targets/effects,
   but core hemx must not gain a mandatory client state graph or local app
   runtime. req: canonical_authoring/008

docs/recipes/observability-flags.md

@@ -0,0 +1,206 @@
+# Recipe: observability, feature flags, and killswitches
+
+This recipe shows where production telemetry and rollout controls belong in a
+hemx app. Metrics, traces, feature flags, A/B assignment, and killswitches are
+application/platform integrations, not hemx core features. hemx should expose a
+small effect boundary, preserve normal HTTP behavior, and leave provider choice
+to the app. req: laws/002 req: laws/004
+
+Use this with `examples/saas` after the auth/session, CSRF, persistence, and
+deploy/versioning boundaries are in place.
+
+## Boundary rule
+
+Keep these concerns outside hemx crates:
+
+- metrics/tracing providers such as OpenTelemetry, Datadog, Prometheus, Honeycomb,
+  or platform logs
+- feature flag providers and assignment stores
+- A/B test bucketing and analytics destinations
+- rollout and killswitch policy
+- alerting, dashboards, and incident response
+
+Keep these concerns in app/integration code:
+
+- route and handler spans
+- effect-response counters
+- provider-specific labels and sampling policy
+- generated UI effects that show degraded or disabled states
+- app-owned flags passed through typed state or extractors
+
+The normal handler shape remains typed Rust returning generated effects.
+
+## Instrument routes and dispatch, not the runtime
+
+Instrument the server boundary around ordinary Axum routes and hemx interaction
+dispatch. The browser runtime should not become an analytics SDK.
+
+```rust
+async fn interact(
+    State(app): State<AppState>,
+    session: CurrentSession,
+    request: InteractionRequest,
+) -> Result<EffectResponse, impl IntoResponse> {
+    let handle_id = request.handle_id();
+    let span = tracing::info_span!(
+        "hemx.interaction",
+        handle_id,
+        user_id = %session.user_id,
+        release = %app.release_id,
+    );
+
+    async move {
+        let ctx = AppContext::new(session, app.store.clone(), app.flags.clone());
+        let result = request.dispatch_async(registry(ctx)).await;
+
+        match &result {
+            Ok(_) => metrics::counter!("hemx.interaction.ok").increment(1),
+            Err(_) => metrics::counter!("hemx.interaction.error").increment(1),
+        }
+
+        result
+    }
+    .instrument(span)
+    .await
+}
+```
+
+The exact crates are app choices. The important part is that observability wraps
+routes, handlers, and provider adapters instead of adding client-side state or
+selector-based probes. req: runtime/003 req: runtime/004
+
+## Feature flags as typed app state
+
+Flags should be ordinary typed state. Handlers read the flag and return generated
+UI effects or normal HTTP responses.
+
+```rust
+#[derive(Clone)]
+pub struct FeatureFlags {
+    project_creation: bool,
+    beta_metrics_island: bool,
+}
+
+#[derive(Clone)]
+pub struct AppContext {
+    session: CurrentSession,
+    store: ProjectStore,
+    flags: FeatureFlags,
+}
+
+#[hemx::handler]
+async fn create_project(
+    State(ctx): State<AppContext>,
+    Form(form): Form<NewProject>,
+) -> Result<impl IntoEffect, AppError> {
+    if !ctx.flags.project_creation {
+        return Ok((
+            dashboard::flash.set("Project creation is temporarily disabled"),
+            dashboard::new_project.disable_while_pending(),
+        ));
+    }
+
+    ctx.verify_csrf(&form.csrf)?;
+    let project = ctx.store.insert(form.name, &ctx.session).await?;
+
+    Ok((
+        dashboard::project_row.append(ProjectRow::from(project)),
+        dashboard::new_project.clear(),
+        dashboard::flash.set("Project created"),
+    ))
+}
+```
+
+A flag provider may refresh `FeatureFlags` from a database, config service, or
+static file. hemx does not need a flag API; the generated helpers are enough to
+show enabled, disabled, or degraded UI.
+
+## Killswitches
+
+A killswitch is a product decision at the application boundary. Prefer explicit
+failure or degraded UI over silently dropping effects.
+
+Good killswitch targets:
+
+- disable one mutation handler while leaving page rendering intact
+- switch from enhanced interaction to full-page form response
+- disable an island or live status stream while keeping the server-rendered page
+  usable
+- pause SSE/polling and show a generated status message
+
+Example for an SSE/live-status killswitch:
+
+```rust
+pub fn live_status(ctx: &AppContext) -> impl IntoEffect {
+    if !ctx.flags.live_status {
+        return dashboard::live_status.set("Live status is paused");
+    }
+
+    dashboard::live_status.set(format!("heartbeat: {} projects", ctx.projects().len()))
+}
+```
+
+Do not add a generic client-side killswitch to the runtime. The runtime applies
+checked effects; the app decides which effects to produce. req: failure/004
+
+## A/B tests and analytics
+
+A/B assignment belongs in auth/session or request context:
+
+```rust
+pub struct ExperimentContext {
+    variant: &'static str,
+}
+
+#[hemx::handler]
+async fn open_settings(
+    State(ctx): State<AppContext>,
+) -> impl IntoEffect {
+    let panel = if ctx.experiments.variant == "compact" {
+        SettingsPage::compact()
+    } else {
+        SettingsPage::full()
+    };
+
+    (
+        dashboard::page_panel.put(&panel),
+        dashboard::nav.set("Settings"),
+        hemx::push("/settings"),
+    )
+}
+```
+
+Analytics can be emitted server-side when the handler runs or through explicit
+native events returned by the handler. Avoid hidden DOM scraping or selector
+listeners as the normal path.
+
+## Metrics to track
+
+Suggested app/platform metrics:
+
+- `hemx.interaction.ok`
+- `hemx.interaction.error`
+- `hemx.form.parse_error`
+- `hemx.handler.failure`
+- `hemx.fingerprint_mismatch`
+- `hemx.missing_target`
+- `hemx.sse.reconnect`
+- `hemx.killswitch.active`
+
+Provider names, label sets, sampling, and retention are platform decisions. Do
+not bake them into hemx core.
+
+## Tests
+
+Keep tests at the app boundary:
+
+- flag disabled: handler does not call the store and returns a generated disabled
+  or flash effect
+- flag enabled: handler follows the normal generated-helper path
+- killswitch active: live status or island is paused with generated UI feedback
+- provider failure: app maps the failure through `AppError` without panicking
+- metrics wrapper records ok/error paths without changing effect contents
+
+`examples/saas` can exercise those checks with an in-memory fake flag provider;
+a real deployment can use the same tests around a provider-backed `FeatureFlags`
+loader. req: examples/001 req: test/001

examples/saas/README.md

@@ -12,7 +12,7 @@ What it proves today:
 - page shell with plain CSS and one explicit metrics island script
 - compile-time surface generation plus interaction tests
 
-For provider-explicit boundaries, see `../../docs/recipes/sqlx-persistence.md`, `../../docs/recipes/auth-session-csrf.md`, and `../../docs/recipes/deploy-versioning.md`.
+For provider-explicit boundaries, see `../../docs/recipes/sqlx-persistence.md`, `../../docs/recipes/auth-session-csrf.md`, `../../docs/recipes/observability-flags.md`, and `../../docs/recipes/deploy-versioning.md`.
 
 What it deliberately does not claim yet: