apsuflow-skills/skills/apsuflow/SKILL.md

---
name: apsuflow
description: >-
  Use when operating, deploying, configuring, troubleshooting, or explaining
  Apsuflow. Build a supported local, multi-environment, or cluster workflow from
  current requirements and CLI help; explain .apsu, networking, quorum, and
  backup/restore; refuse direct SQLite, runtime, or host-network mutation. Do
  not use for implementing Apsuflow, changing requirements, or generic container
  architecture advice.
---

# Operate Apsuflow

## One job

Turn an operator goal into the supported path:

```text
select target -> validate desired state -> inspect diff -> apply -> verify -> recover through public controls
```

This skill prevents agents from bypassing the control plane by editing SQLite,
invoking the OCI runtime, or changing derived networking.

## Authority

Before giving commands or causing effects, check only what affects the answer:

1. `REQUIREMENTS.md` — current product obligations;
2. `apsuflow <command> --help` — installed command contract;
3. `docs/getting-started.md`, `docs/cli.md`, `docs/solo-vs-cluster.md`,
   `docs/operations.md`, and `docs/grammar.md` — explanations;
4. `examples/` — syntax examples, never higher authority.

Follow requirements and observed CLI behavior when they conflict with prose, and
report the stale surface. `SKILL.md` is derived guidance, not authority. Update it
in the same change when applicable requirements or CLI behavior change.

## Mental model

- One server owns desired state, reconciliation, authentication, and the API.
- Solo mode is one loopback-bound server without cluster mesh overhead.
- Cluster servers replicate control-plane state; agents execute workloads but do
  not add quorum votes.
- The unprivileged service daemon owns OCI children inside its delegated cgroup
  v2 subtree. A supervisor may prepare root-owned host networking/firewall state,
  but the long-running daemon must not gain root merely to bypass a failed
  profile. Stable host profiles target Debian glibc/systemd and Void Linux
  musl/runit; advertise one only after its complete isolation and golden path
  pass. Routine CLI use remains unprivileged and authenticated.
- `.apsu` contains desired workloads and infrastructure, never secret values or
  mutable runtime state.
- Services are continuously reconciled. Jobs are finite run-to-completion work.
  Application queues, message retries, DLQs, and workflow state belong in
  broker-backed workloads.

## Infrastructure as code is central

Treat reviewed, version-controlled `.apsu` plus the selected context as the
source of production intent. Put workloads, provider pools, placement, routes,
health checks, resources, update policy, and sealed-name references there; keep
secret values and credentials outside Git. A production change starts as a
declaration diff, passes `fmt --check`, `validate`, and preferably `apply
--dry-run`, then applies that exact reviewed file set. Record its revision,
context, dry-run, and post-apply workload/route evidence.

Do not create a second source of truth through hand-edited runtime state or an
uncommitted mystery file. A manual restart may be an emergency diagnostic action,
but the durable repair belongs in IaC and must converge after re-apply. If live
state cannot be reconstructed from the elected declaration, stop and reconcile
that drift before making another change.

## Install and update

Use only the exact stable tag and artifact named by `docs/getting-started.md`.
Verify `SHA256SUMS` and its SSH signature before installing; never use a mutable
`latest` URL. Keep the previous executable and state backup until the new one
passes `version`, `doctor`, workload diagnosis, and each operator-facing route.
For host-network workloads, confirm the old container released its host ports and
the successor is the sole running generation; repeated replacement failures are
a rollback signal, not a reason to mutate runtime state directly. The first
stable release has no predecessor compatibility case; every later stable release
must carry proved upgrade and rollback evidence against its adjacent stable
predecessor. Apsuflow is not published to a Cargo registry.

## Local operation

```sh
apsuflow fmt --check app.apsu
apsuflow validate app.apsu
apsuflow diff app.apsu
apsuflow apply app.apsu
apsuflow status
apsuflow diagnose web
```

Use `fmt` without `--check` only when reformatting is intended. Use `apply
--prune` only when declarations omitted from all supplied files should be
removed. Validation reports portability limits before mutation. Capabilities
whose failure can require manual recovery—host networking, devices, and writable
host paths—also require `apply --acknowledge-recovery-risk`; read the reported
source location and consequence before accepting it. Verify the actual route or
job result; a successful apply alone proves only admission.

The daemon normally needs root for host effects. Do not therefore run every
client command with `sudo`: protect the daemon data directory and use an operator
context or token. Never make bootstrap tokens or the data directory broadly
readable. `server --no-auth` is acceptable only for disposable loopback use.

## Multiple environments

A context selects an endpoint and credentials; it is not a namespace. Use
separate servers/clusters for environments requiring isolation.

```sh
apsuflow context set --name dev  --server https://dev.example:7654
apsuflow context set --name prod --server https://prod.example:7654
apsuflow context list
apsuflow status --context dev --token "$APSUFLOW_DEV_TOKEN"
```

Prefer protected secret injection and explicit `--context` in automation; avoid
putting tokens in shell history or committed files. If storing a token in an
Apsuflow context with `context set --token`, ensure the config file is readable
only by its owner. Keep shared and environment-specific
inputs as ordinary files:

```text
apsu/base.apsu
apsu/env/dev.apsu
apsu/env/prod.apsu
```

Use the same ordered set for validate, diff, and apply:

```sh
apsuflow validate --context dev apsu/base.apsu apsu/env/dev.apsu
apsuflow diff     --context dev apsu/base.apsu apsu/env/dev.apsu
apsuflow apply    --context dev apsu/base.apsu apsu/env/dev.apsu
```

A later duplicate declaration replaces the earlier declaration completely. Do
not add templating, inline secrets, or database edits to simulate environments.

## `.apsu` essentials

Top-level declarations are `service`, `job`, and `infra`:

```apsu
service web {
  image: "nginx:1.25"
  replicas: 2
  cpu: 0.25
  memory: 128Mi
  route { listen: 80 expose: 8080 protocol: tcp }
}

job backup {
  image: "ghcr.io/example/backup@sha256:<digest>"
  command: ["/app/backup"]
  restart: on-failure
  max_retries: 2
  schedule { trigger: cron "0 3 * * *" concurrency: forbid }
}
```

- Use markerless canonical syntax and let `fmt` normalize it.
- Prefer digest-pinned production images.
- Reference secret names and set values with `apsuflow secret set`; never embed
  secret values in `.apsu`.
- Routes are L4 TCP/UDP forwarding, not HTTP hostname routing or application TLS.
- Writable host paths are node-local and need placement plus separate data backup.
- Use a service plus an external broker for consumers; use a job for bounded
  migration, backup, maintenance, or scheduled work.
- Scheduled declarations create job runs; inspect them with `apsuflow job runs
  NAME`. Use `apsuflow run` for an intentional ad-hoc foreground container; it
  does not create durable scheduled-job history.

## Cluster and network

Start the first server on a stable address. Create a short-lived role-scoped join
token with `apsuflow token create --name NODE --role server`, then start another
server with `apsuflow server --peer FIRST:7946 --token TOKEN`. For an agent,
create an `agent` token and run `apsuflow agent --join FIRST:7654 --token TOKEN`.
Never reuse or commit a bootstrap token.

Allow between nodes only required control-plane traffic:

- TCP 7654 — API;
- UDP 7946 — membership;
- UDP 51820 — WireGuard overlay.

Expose workload route ports separately. Do not manually create WireGuard peers,
CNI networks, nftables rules, or DNS records. Verify with `node list`, `status`,
`containers`, `routes`, and `diagnose`. Before infrastructure apply, inspect
`status --json`: the configured provider must be compiled and the needed effect
must not appear in `unavailable_capabilities`; resolve any
`leaked_infrastructure_resources` before retrying destructive operations.

### Quorum

Use three server-role nodes when control-plane failure tolerance matters. Writes
need a majority: one server has no redundancy, two require both, and three can
lose one. Agents do not count toward quorum.

On quorum loss, preserve committed state and reject unsafe writes. Use explicit
promotion only for its documented recovery case. `store force-leader --confirm`
is a last resort after the old writer is proven unavailable and possible loss is
accepted; scripts must acknowledge the committed writer term and sequence. It is
not a way to clear an error.

## Backup and recovery

Create a client-owned encrypted control-plane backup and move both outputs
off-cluster, storing the key separately from the archive:

```sh
apsuflow backup --context prod \
  --key-out prod-control-plane.key \
  prod-control-plane.tar.zst
```

Regularly prove restore against an empty, disposable target:

```sh
apsuflow restore --context recovery \
  --key prod-control-plane.key \
  prod-control-plane.tar.zst
apsuflow status --context recovery
```

This backs up Apsuflow control-plane state, not bind-mounted application data,
databases, object stores, or brokers. Back those up natively and test both paths.

Never mutate `store.db`, WAL files, generations, assignments, certificates, or
sealed values with SQLite/SQL. Never invoke crun/youki or host-network tools to
force convergence. Read-only host inspection may support diagnosis, but effects
must return through Apsuflow or stop with a precise blocker.

## Decision and refusal

1. Identify the elected IaC revision, exact file order, context, and topology.
2. Validate `.apsu` inputs and secret references; run diff or dry-run before apply.
3. Ask before prune, restore, promotion, force-leader, or billable/destructive
   provider actions.
4. Apply the exact reviewed declaration through CLI/API and prove convergence,
   workload behavior, and operator-facing routes.
5. Diagnose drift through public status, logs, and recovery controls; repair the
   declaration rather than the runtime.

Do not invent built-in queues, DLQs, application TLS, HTTP hostname routing, or
distributed storage; use contexts as namespaces; put values/templates in `.apsu`;
use `sudo` for routine client calls when credentials suffice; or bypass desired
state through SQLite, runtime, CNI, WireGuard, nftables, or DNS mutation.

Report the target context/topology, `.apsu` inputs, effects, validation and
workload proof, backup impact, and any refused unsupported operation.