hemx/docs/recipes/observability-flags.md

Recipe: observability, feature flags, and killswitches

This recipe shows where production telemetry and rollout controls belong in a hemx app. Metrics, traces, feature flags, A/B assignment, and killswitches are application/platform integrations, not hemx core features. hemx should expose a small effect boundary, preserve normal HTTP behavior, and leave provider choice to the app. req: laws/002 req: laws/004

Use this with examples/saas after the auth/session, CSRF, persistence, and deploy/versioning boundaries are in place.

Boundary rule

Keep these concerns outside hemx crates:

• metrics/tracing providers such as OpenTelemetry, Datadog, Prometheus, Honeycomb, or platform logs
• feature flag providers and assignment stores
• A/B test bucketing and analytics destinations
• rollout and killswitch policy
• alerting, dashboards, and incident response

Keep these concerns in app/integration code:

• route and handler spans
• effect-response counters
• provider-specific labels and sampling policy
• generated UI effects that show degraded or disabled states
• app-owned flags passed through typed state or extractors

The normal handler shape remains typed Rust returning generated effects.

Instrument routes and dispatch, not the runtime

Instrument the server boundary around ordinary Axum routes and hemx interaction dispatch. The browser runtime should not become an analytics SDK.

async fn interact(
    State(app): State<AppState>,
    session: CurrentSession,
    request: InteractionRequest,
) -> Result<EffectResponse, impl IntoResponse> {
    let handle_id = request.handle_id();
    let span = tracing::info_span!(
        "hemx.interaction",
        handle_id,
        user_id = %session.user_id,
        release = %app.release_id,
    );

    async move {
        let ctx = AppContext::new(session, app.store.clone(), app.flags.clone());
        let result = request.dispatch_async(registry(ctx)).await;

        match &result {
            Ok(_) => metrics::counter!("hemx.interaction.ok").increment(1),
            Err(_) => metrics::counter!("hemx.interaction.error").increment(1),
        }

        result
    }
    .instrument(span)
    .await
}

The exact crates are app choices. The important part is that observability wraps routes, handlers, and provider adapters instead of adding client-side state or selector-based probes. req: runtime/003 req: runtime/004

Feature flags as typed app state

Flags should be ordinary typed state. Handlers read the flag and return generated UI effects or normal HTTP responses.

#[derive(Clone)]
pub struct FeatureFlags {
    project_creation: bool,
    beta_metrics_island: bool,
}

#[derive(Clone)]
pub struct AppContext {
    session: CurrentSession,
    store: ProjectStore,
    flags: FeatureFlags,
}

#[hemx::handler]
async fn create_project(
    State(ctx): State<AppContext>,
    Form(form): Form<NewProject>,
) -> Result<impl IntoEffect, AppError> {
    if !ctx.flags.project_creation {
        return Ok((
            dashboard::flash.set("Project creation is temporarily disabled"),
            dashboard::new_project.disable_while_pending(),
        ));
    }

    ctx.verify_csrf(&form.csrf)?;
    let project = ctx.store.insert(form.name, &ctx.session).await?;

    Ok((
        dashboard::project_row.append(ProjectRow::from(project)),
        dashboard::new_project.clear(),
        dashboard::flash.set("Project created"),
    ))
}

A flag provider may refresh FeatureFlags from a database, config service, or static file. hemx does not need a flag API; the generated helpers are enough to show enabled, disabled, or degraded UI.

Killswitches

A killswitch is a product decision at the application boundary. Prefer explicit failure or degraded UI over silently dropping effects.

Good killswitch targets:

• disable one mutation handler while leaving page rendering intact
• switch from enhanced interaction to full-page form response
• disable an island or live status stream while keeping the server-rendered page usable
• pause SSE/polling and show a generated status message

Example for an SSE/live-status killswitch:

pub fn live_status(ctx: &AppContext) -> impl IntoEffect {
    if !ctx.flags.live_status {
        return dashboard::live_status.set("Live status is paused");
    }

    dashboard::live_status.set(format!("heartbeat: {} projects", ctx.projects().len()))
}

Do not add a generic client-side killswitch to the runtime. The runtime applies checked effects; the app decides which effects to produce. req: failure/004

A/B tests and analytics

A/B assignment belongs in auth/session or request context:

pub struct ExperimentContext {
    variant: &'static str,
}

#[hemx::handler]
async fn open_settings(
    State(ctx): State<AppContext>,
) -> impl IntoEffect {
    let panel = if ctx.experiments.variant == "compact" {
        SettingsPage::compact()
    } else {
        SettingsPage::full()
    };

    (
        dashboard::page_panel.put(&panel),
        dashboard::nav.set("Settings"),
        hemx::push("/settings"),
    )
}

Analytics can be emitted server-side when the handler runs or through explicit native events returned by the handler. Avoid hidden DOM scraping or selector listeners as the normal path.

Metrics to track

Suggested app/platform metrics:

• hemx.interaction.ok
• hemx.interaction.error
• hemx.form.parse_error
• hemx.handler.failure
• hemx.fingerprint_mismatch
• hemx.missing_target
• hemx.sse.reconnect
• hemx.killswitch.active

Provider names, label sets, sampling, and retention are platform decisions. Do not bake them into hemx core.

Tests

Keep tests at the app boundary:

• flag disabled: handler does not call the store and returns a generated disabled or flash effect
• flag enabled: handler follows the normal generated-helper path
• killswitch active: live status or island is paused with generated UI feedback
• provider failure: app maps the failure through AppError without panicking
• metrics wrapper records ok/error paths without changing effect contents

examples/saas can exercise those checks with an in-memory fake flag provider; a real deployment can use the same tests around a provider-backed FeatureFlags loader. req: examples/001 req: test/001