# Recipe: observability, feature flags, and killswitches

This recipe shows where production telemetry and rollout controls belong in a
hemx app. Metrics, traces, feature flags, A/B assignment, and killswitches are
application/platform integrations, not hemx core features. hemx should expose a
small effect boundary, preserve normal HTTP behavior, and leave provider choice
to the app. req: laws/002 req: laws/004

Use this with `examples/saas` after the auth/session, CSRF, persistence, and
deploy/versioning boundaries are in place.

## Boundary rule

Keep these concerns outside hemx crates:

- metrics/tracing providers such as OpenTelemetry, Datadog, Prometheus, Honeycomb,
  or platform logs
- feature flag providers and assignment stores
- A/B test bucketing and analytics destinations
- rollout and killswitch policy
- alerting, dashboards, and incident response

Keep these concerns in app/integration code:

- route and handler spans
- effect-response counters
- provider-specific labels and sampling policy
- generated UI effects that show degraded or disabled states
- app-owned flags passed through typed state or extractors

The normal handler shape remains typed Rust returning generated effects.

## Instrument routes and dispatch, not the runtime

Instrument the server boundary around ordinary Axum routes and hemx interaction
dispatch. The browser runtime should not become an analytics SDK.

```rust
async fn interact(
    State(app): State<AppState>,
    session: CurrentSession,
    request: InteractionRequest,
) -> Result<EffectResponse, impl IntoResponse> {
    let handle_id = request.handle_id();
    let span = tracing::info_span!(
        "hemx.interaction",
        handle_id,
        user_id = %session.user_id,
        release = %app.release_id,
    );

    async move {
        let ctx = AppContext::new(session, app.store.clone(), app.flags.clone());
        let result = request.dispatch_async(registry(ctx)).await;

        match &result {
            Ok(_) => metrics::counter!("hemx.interaction.ok").increment(1),
            Err(_) => metrics::counter!("hemx.interaction.error").increment(1),
        }

        result
    }
    .instrument(span)
    .await
}
```

The exact crates are app choices. The important part is that observability wraps
routes, handlers, and provider adapters instead of adding client-side state or
selector-based probes. req: runtime/003 req: runtime/004

## Feature flags as typed app state

Flags should be ordinary typed state. Handlers read the flag and return generated
UI effects or normal HTTP responses.

```rust
#[derive(Clone)]
pub struct FeatureFlags {
    project_creation: bool,
    beta_metrics_island: bool,
}

#[derive(Clone)]
pub struct AppContext {
    session: CurrentSession,
    store: ProjectStore,
    flags: FeatureFlags,
}

#[hemx::handler]
async fn create_project(
    State(ctx): State<AppContext>,
    Form(form): Form<NewProject>,
) -> Result<impl IntoEffect, AppError> {
    if !ctx.flags.project_creation {
        return Ok((
            dashboard::flash.set("Project creation is temporarily disabled"),
            dashboard::new_project.disable_while_pending(),
        ));
    }

    ctx.verify_csrf(&form.csrf)?;
    let project = ctx.store.insert(form.name, &ctx.session).await?;

    Ok((
        dashboard::project_row.append(ProjectRow::from(project)),
        dashboard::new_project.clear(),
        dashboard::flash.set("Project created"),
    ))
}
```

A flag provider may refresh `FeatureFlags` from a database, config service, or
static file. hemx does not need a flag API; the generated helpers are enough to
show enabled, disabled, or degraded UI.

## Killswitches

A killswitch is a product decision at the application boundary. Prefer explicit
failure or degraded UI over silently dropping effects.

Good killswitch targets:

- disable one mutation handler while leaving page rendering intact
- switch from enhanced interaction to full-page form response
- disable an island or live status stream while keeping the server-rendered page
  usable
- pause SSE/polling and show a generated status message

Example for an SSE/live-status killswitch:

```rust
pub fn live_status(ctx: &AppContext) -> impl IntoEffect {
    if !ctx.flags.live_status {
        return dashboard::live_status.set("Live status is paused");
    }

    dashboard::live_status.set(format!("heartbeat: {} projects", ctx.projects().len()))
}
```

Do not add a generic client-side killswitch to the runtime. The runtime applies
checked effects; the app decides which effects to produce. req: failure/004

## A/B tests and analytics

A/B assignment belongs in auth/session or request context:

```rust
pub struct ExperimentContext {
    variant: &'static str,
}

#[hemx::handler]
async fn open_settings(
    State(ctx): State<AppContext>,
) -> impl IntoEffect {
    let panel = if ctx.experiments.variant == "compact" {
        SettingsPage::compact()
    } else {
        SettingsPage::full()
    };

    (
        dashboard::page_panel.put(&panel),
        dashboard::nav.set("Settings"),
        hemx::push("/settings"),
    )
}
```

Analytics can be emitted server-side when the handler runs or through explicit
native events returned by the handler. Avoid hidden DOM scraping or selector
listeners as the normal path.

## Metrics to track

Suggested app/platform metrics:

- `hemx.interaction.ok`
- `hemx.interaction.error`
- `hemx.form.parse_error`
- `hemx.handler.failure`
- `hemx.fingerprint_mismatch`
- `hemx.missing_target`
- `hemx.sse.reconnect`
- `hemx.killswitch.active`

Provider names, label sets, sampling, and retention are platform decisions. Do
not bake them into hemx core.

## Tests

Keep tests at the app boundary:

- flag disabled: handler does not call the store and returns a generated disabled
  or flash effect
- flag enabled: handler follows the normal generated-helper path
- killswitch active: live status or island is paused with generated UI feedback
- provider failure: app maps the failure through `AppError` without panicking
- metrics wrapper records ok/error paths without changing effect contents

`examples/saas` can exercise those checks with an in-memory fake flag provider;
a real deployment can use the same tests around a provider-backed `FeatureFlags`
loader. req: examples/001 req: test/001