---
name: apsuflow
description: >-
  Use when operating, deploying, configuring, troubleshooting, or explaining
  Apsuflow. Build a supported local, multi-environment, or cluster workflow from
  current requirements and CLI help; explain .apsu, networking, quorum, and
  backup/restore; refuse direct SQLite, runtime, or host-network mutation. Do
  not use for implementing Apsuflow, changing requirements, or generic container
  architecture advice.
---

# Operate Apsuflow

## One job

Turn an operator goal into the supported path:

```text
select target -> validate desired state -> inspect diff -> apply -> verify -> recover through public controls
```

This skill prevents agents from bypassing the control plane by editing SQLite,
invoking the OCI runtime, or changing derived networking.

## Authority

Before giving commands or causing effects, check only what affects the answer:

1. `REQUIREMENTS.md` — current product obligations;
2. `apsuflow <command> --help` — installed command contract;
3. `docs/getting-started.md`, `docs/cli.md`, `docs/solo-vs-cluster.md`,
   `docs/operations.md`, and `docs/grammar.md` — explanations;
4. `examples/` — syntax examples, never higher authority.

Follow requirements and observed CLI behavior when they conflict with prose, and
report the stale surface. `SKILL.md` is derived guidance, not authority. Update it
in the same change when applicable requirements or CLI behavior change.

## Mental model

- One server owns desired state, reconciliation, authentication, and the API.
- Solo mode is one loopback-bound server without cluster mesh overhead.
- Cluster servers replicate control-plane state; agents execute workloads but do
  not add quorum votes.
- The privileged daemon owns OCI, cgroups, CNI, WireGuard, nftables, and host-path
  effects. Routine CLI use should be unprivileged and authenticated.
- `.apsu` contains desired workloads and infrastructure, never secret values or
  mutable runtime state.
- Services are continuously reconciled. Jobs are finite run-to-completion work.
  Application queues, message retries, DLQs, and workflow state belong in
  broker-backed workloads.

## Install and update

Use only the exact stable tag and artifact named by `docs/getting-started.md`.
Verify `SHA256SUMS` and its SSH signature before installing; never use a mutable
`latest` URL. Keep the previous executable until the new one passes `version`,
`doctor`, and workload verification. The first stable release has no predecessor
compatibility case; every later stable release must carry proved upgrade and
rollback evidence against its adjacent stable predecessor. Apsuflow is not
published to a Cargo registry.

## Local operation

```sh
apsuflow fmt --check app.apsu
apsuflow validate app.apsu
apsuflow diff app.apsu
apsuflow apply app.apsu
apsuflow status
apsuflow diagnose web
```

Use `fmt` without `--check` only when reformatting is intended. Use `apply
--prune` only when declarations omitted from all supplied files should be
removed. Validation reports portability limits before mutation. Capabilities
whose failure can require manual recovery—host networking, devices, and writable
host paths—also require `apply --acknowledge-recovery-risk`; read the reported
source location and consequence before accepting it. Verify the actual route or
job result; a successful apply alone proves only admission.

The daemon normally needs root for host effects. Do not therefore run every
client command with `sudo`: protect the daemon data directory and use an operator
context or token. Never make bootstrap tokens or the data directory broadly
readable. `server --no-auth` is acceptable only for disposable loopback use.

## Multiple environments

A context selects an endpoint and credentials; it is not a namespace. Use
separate servers/clusters for environments requiring isolation.

```sh
apsuflow context set --name dev  --server https://dev.example:7654
apsuflow context set --name prod --server https://prod.example:7654
apsuflow context list
apsuflow status --context dev --token "$APSUFLOW_DEV_TOKEN"
```

Prefer protected secret injection and explicit `--context` in automation; avoid
putting tokens in shell history or committed files. If storing a token in an
Apsuflow context with `context set --token`, ensure the config file is readable
only by its owner. Keep shared and environment-specific
inputs as ordinary files:

```text
apsu/base.apsu
apsu/env/dev.apsu
apsu/env/prod.apsu
```

Use the same ordered set for validate, diff, and apply:

```sh
apsuflow validate --context dev apsu/base.apsu apsu/env/dev.apsu
apsuflow diff     --context dev apsu/base.apsu apsu/env/dev.apsu
apsuflow apply    --context dev apsu/base.apsu apsu/env/dev.apsu
```

A later duplicate declaration replaces the earlier declaration completely. Do
not add templating, inline secrets, or database edits to simulate environments.

## `.apsu` essentials

Top-level declarations are `service`, `job`, and `infra`:

```apsu
service web {
  image: "nginx:1.25"
  replicas: 2
  cpu: 0.25
  memory: 128Mi
  route { listen: 80 expose: 8080 protocol: tcp }
}

job backup {
  image: "ghcr.io/example/backup@sha256:<digest>"
  command: ["/app/backup"]
  restart: on-failure
  max_retries: 2
  schedule { trigger: cron "0 3 * * *" concurrency: forbid }
}
```

- Use markerless canonical syntax and let `fmt` normalize it.
- Prefer digest-pinned production images.
- Reference secret names and set values with `apsuflow secret set`; never embed
  secret values in `.apsu`.
- Routes are L4 TCP/UDP forwarding, not HTTP hostname routing or application TLS.
- Writable host paths are node-local and need placement plus separate data backup.
- Use a service plus an external broker for consumers; use a job for bounded
  migration, backup, maintenance, or scheduled work.
- Scheduled declarations create job runs; inspect them with `apsuflow job runs
  NAME`. Use `apsuflow run` for an intentional ad-hoc foreground container; it
  does not create durable scheduled-job history.

## Cluster and network

Start the first server on a stable address. Create a short-lived role-scoped join
token with `apsuflow token create --name NODE --role server`, then start another
server with `apsuflow server --peer FIRST:7946 --token TOKEN`. For an agent,
create an `agent` token and run `apsuflow agent --join FIRST:7654 --token TOKEN`.
Never reuse or commit a bootstrap token.

Allow between nodes only required control-plane traffic:

- TCP 7654 — API;
- UDP 7946 — membership;
- UDP 51820 — WireGuard overlay.

Expose workload route ports separately. Do not manually create WireGuard peers,
CNI networks, nftables rules, or DNS records. Verify with `node list`, `status`,
`containers`, `routes`, and `diagnose`. Before infrastructure apply, inspect
`status --json`: the configured provider must be compiled and the needed effect
must not appear in `unavailable_capabilities`; resolve any
`leaked_infrastructure_resources` before retrying destructive operations.

### Quorum

Use three server-role nodes when control-plane failure tolerance matters. Writes
need a majority: one server has no redundancy, two require both, and three can
lose one. Agents do not count toward quorum.

On quorum loss, preserve committed state and reject unsafe writes. Use explicit
promotion only for its documented recovery case. `store force-leader --confirm`
is a last resort after the old writer is proven unavailable and possible loss is
accepted; scripts must acknowledge the committed writer term and sequence. It is
not a way to clear an error.

## Backup and recovery

Create a client-owned encrypted control-plane backup and move both outputs
off-cluster, storing the key separately from the archive:

```sh
apsuflow backup --context prod \
  --key-out prod-control-plane.key \
  prod-control-plane.tar.zst
```

Regularly prove restore against an empty, disposable target:

```sh
apsuflow restore --context recovery \
  --key prod-control-plane.key \
  prod-control-plane.tar.zst
apsuflow status --context recovery
```

This backs up Apsuflow control-plane state, not bind-mounted application data,
databases, object stores, or brokers. Back those up natively and test both paths.

Never mutate `store.db`, WAL files, generations, assignments, certificates, or
sealed values with SQLite/SQL. Never invoke crun/youki or host-network tools to
force convergence. Read-only host inspection may support diagnosis, but effects
must return through Apsuflow or stop with a precise blocker.

## Decision and refusal

1. Name the context and solo/cluster topology.
2. Validate `.apsu` inputs and secret references; run diff before apply.
3. Ask before prune, restore, promotion, force-leader, or billable/destructive
   provider actions.
4. Apply through CLI/API and prove the workload path.
5. Diagnose through public status, logs, and recovery controls.

Do not invent built-in queues, DLQs, application TLS, HTTP hostname routing, or
distributed storage; use contexts as namespaces; put values/templates in `.apsu`;
use `sudo` for routine client calls when credentials suffice; or bypass desired
state through SQLite, runtime, CNI, WireGuard, nftables, or DNS mutation.

Report the target context/topology, `.apsu` inputs, effects, validation and
workload proof, backup impact, and any refused unsupported operation.