OpenBao initialization and recovery

This is the current operator guide for Enclavr's OpenBao 2.7.1 integration. Use the accepted image on an existing node; never downgrade a newer supported installation to reproduce an example. The paths, roles and custody reference explains where each artifact belongs. All examples are CLI commands. Replace <namespace>, <cluster_id>, hostnames and addresses before running them.

Choose the procedure

Start with the CLI command ./enclavr namespace status, or launch the TUI with ./enclavr and select Config → Roles and cluster status. This separates configured namespace-writer, DNS-authority and DNS-publisher ownership from the live elected OpenBao leader. AppRole purpose, owner, scope and connection path belong in authoritative openbao_roles; dns_topology.publication references the reader and publisher role names. They do not belong in chat history or an assumed workstation directory. See the role and custody reference and writer handover procedure. The status action reports declared paths without opening publisher credentials; publication checks the selected files and cluster before use.

Private editing (CLI and TUI), Caddy publication and runner reconciliation use the existing namespace-writer entry in openbao_roles.roles. Its owner and target must match namespace_writer_host; its literal connection_file is opened and authenticated against the declared role and accepted cluster. Missing or stale declarations must be corrected using the existing identity. The status report shows this declared path and labels generated policies as declarations, not verified live ACLs.

Shared scalar fields

Keep scalar runtime values in secrets.<namespace>. If a field must also appear in another core item, use the existing template syntax to reference its source, for example {{ op://example.enclavr.io/secrets.example.enclavr.io/onboarding_status }}. Enclavr's OpenBao repository resolves whole-field scalar templates on reads; direct op:// reference values and document bodies retain their original form. Cycles, missing scalar sources, cross-namespace references and references to an entire secrets document are rejected. The OpenBao KV API itself does not evaluate these templates.

./enclavr namespace inspect checks scalar sharing across the accessible core items and reports logical field names instead of opaque import IDs. Equal copies do not count as a shared source. Ordinary core-item writes through the repository check the candidate against the other core items before mutation; an inaccessible required source fails closed. Existing conflicts must be reviewed on the namespace writer. No automatic field deletion or migration is performed. This check is not a transaction across separate KV documents and cannot police direct writes made outside Enclavr.

Duplicate JSON object keys, repeated field IDs/labels, and aliases that make a field ambiguous are rejected by the repository, including JSON document payloads. Item metadata and a secrets item's descriptive document are separate from its scalar runtime fields; their coexistence alone is not duplication. Imported field annotations pointing to the retired split-runtime item are also rejected. Remove only the stale annotation after verifying the canonical field; its value and identity stay intact.

An unreachable endpoint, failed secret lookup, missing checkout or missing checkpoint does not prove that a cluster is uninitialized. Do not initialize existing Raft storage, delete a volume, regenerate seal/TLS material, or select 1Password as a runtime fallback to make an error disappear.

Canonical runtime fields

All scalar runtime fields belong in the existing secrets.<namespace> item. Inspect the namespace before correcting a missing field. On the namespace writer, launch ./enclavr → Config → Repair namespace, review the canonical field changes, and confirm publication. Then run ./enclavr namespace inspect and ./enclavr config validate.

Retired split-runtime aliases are not fallback or migration sources. Before removing their remaining grants, verify that consumers reference canonical items and update each existing connection's item list on its declared owner. Preserve its credentials, namespace, cluster and TLS trust. Remove only the obsolete grants from existing ACLs and matching assets declarations, then verify canonical reads and an unchanged repeat. Otherwise a stale connection list can turn an obsolete item lookup into a permission failure.

An inaccessible item is not proof of absence. Preserve recoverable historical versions; this cleanup does not require a replacement document or credential.

Completed migration and ongoing access management

The mwc.enclavr.io migration completed on 2026-11-01. Its ten retired confidential grants and connection entries were removed, all affected owners passed canonical-read and denied-access checks, and an unchanged repeat passed. The accepted versions were assets 46, secrets 10 and users 3. This is execution evidence, not a runtime input; use namespace status and inspect for current state.

The one-time namespace cleanup command and its shell wrapper have been retired. Keep historical KV versions and recovery evidence. If another namespace still has retired grants or a pending cleanup checkpoint, complete that reviewed migration with the preceding release before using this release's access editor. Never discard a pending checkpoint to bypass acceptance.

For subsequent permission changes, launch ./enclavr → Config → OpenBao access (CLI: ./enclavr namespace access). Review the exact role, owner, connection paths and item permissions before confirming. Enabled item roles require their existing custodian connection; managed consumers whose runtime host differs from the custodian also require that target's already configured connection. Targets resolve their accepted path from their protected runtime configuration; no path, identity or credential is invented or copied.

Review checks local scope and authenticated identity before server changes. After ACL and full-assets publication, each connection must prove its exact effective capabilities and retained reads before its item list is synchronized. The workflow preserves credentials, cluster and CA pins and updates complete owner assets mirrors. Missing scope entries are reconciled only to reviewed grants; unrelated extra entries stop for inspection. A disabled role is authenticated after re-enabling. Disabling retains its connection file.

Every participating host needs the same updated code and verified SSH access from the writer. The existing managed entrypoint is used; SSH is not retried automatically. After a partial failure, use OpenBao access → Resume pending change. The checkpoint remains until owner verification completes, including when server publication already succeeded. A successful repeat preserves unchanged ACLs, assets and connection files. Namespace inspection from a scoped consumer can report inaccessible service items; verify them on the writer without broadening that consumer's grants.

The existing migration-verifier remains the DNS source reader. Its historical name is not grounds for renaming or deleting it. Administrator access, host enrollment, namespace repair, backup, restore and receipt verification remain supported. The internal document-structure and assets-templates operations are retained for explicit representation conversion, including older restored snapshots; they are not normal runtime fallback sources. The production-specific rustdesk-caddy rewrite has been removed; use reviewed Caddy publication instead.

Prepare an attended workspace

Use a trusted administrator terminal without command tracing or session recording. Keep directories 0700, files 0600, and avoid symlinks and writable ancestors. The example administrator directory is outside the checkout:

umask 077
ENCLAVR_NAMESPACE='<namespace>'
ENCLAVR_OPS="$HOME/.local/state/enclavr/openbao-admin"
ENCLAVR_NODE_ROOT="$HOME/.local/share/enclavr/openbao"
mkdir -p "$ENCLAVR_OPS"
chmod 700 "$ENCLAVR_OPS"

In JSON examples, /home/operator/.local/state/enclavr/openbao-admin means the actual absolute value of ENCLAVR_OPS. JSON does not expand shell variables or ~. On macOS use the selected account's /Users/... paths. Preserve existing administrator paths and checkpoints rather than starting a new empty directory during recovery.

Prepare protected request files with Vim. Use vim -n -i NONE and :set nobackup nowritebackup before handling sensitive files. Never put a token, share, key, resolved namespace document or private key in argv. Commands below pass protected input through files or stdin; responses that might contain credentials go to protected files.

Every Enclavr operation uses this plan/apply boundary:

./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/request.json"
./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/request.json" --apply

The plan validates request shape only: it reads no referenced credentials and contacts no endpoint. It is not a live readiness check. initialize and recover-provision additionally require --attended and an actual terminal. Stop after any failed command and inspect its value-free diagnostic; do not blindly run the following mutation. Retain partial-operation checkpoints for a supported retry.

Initialize once

1. Establish custody and topology

For an existing cluster, supply its original seal, TLS and recovery material in the repository's protected .env-mwc-openbao JSON object. Preserve the initialized cluster ID and seal identifier. Missing local files never authorize generating replacement keys. The operator retains an independent copy manually; Enclavr does not contact an external secret store for custody.

For a positively new cluster only, inspect and explicitly create local seal custody:

./enclavr internal openbao-key --namespace "$ENCLAVR_NAMESPACE"
./enclavr internal openbao-key --namespace "$ENCLAVR_NAMESPACE" --apply

Existing valid custody is preserved. The JSON must be a regular 0600 file under parents that are not writable by other accounts. Duplicate keys, malformed material or a pin mismatch stop the procedure. Do not distribute the combined operator file: preparation places only the minimal seal in each node bundle.

Write cluster.json, declaring exactly three or five unique production voters. Addresses below are illustrative Tailscale addresses; verify the real addresses and host identities independently. Node IDs are stable Raft IDs, not automatically the SSH hostname. Five-node deployments add two explicit node records in the same shape.

{
  "namespace": "<namespace>",
  "nodes": [
    {"id":"node-1","hostname":"node-1.<namespace>","address":"100.64.0.11","root":"/home/operator/.local/share/enclavr/openbao","image":"enclavr-openbao:2.7.1"},
    {"id":"node-2","hostname":"node-2.<namespace>","address":"100.64.0.12","root":"/home/operator/.local/share/enclavr/openbao","image":"enclavr-openbao:2.7.1"},
    {"id":"node-3","hostname":"node-3.<namespace>","address":"100.64.0.13","root":"/home/operator/.local/share/enclavr/openbao","image":"enclavr-openbao:2.7.1"}
  ]
}

Ensure the approved image is installed on every target; preparation does not pull it. Prepare complete node bundles:

./enclavr internal openbao-prepare --bootstrap-file "$PWD/.env-mwc-openbao" \
  --spec "$ENCLAVR_OPS/cluster.json" --output-dir "$ENCLAVR_OPS/prepared"
./enclavr internal openbao-prepare --bootstrap-file "$PWD/.env-mwc-openbao" \
  --spec "$ENCLAVR_OPS/cluster.json" --output-dir "$ENCLAVR_OPS/prepared" --apply

Preparation preserves the accepted CA, seal generation and node identities; a topology mismatch requires explicit reconciliation. Transfer only each node's own protected bundle over verified SSH. On that exact host, apply the selected node directory:

sudo bash src/scripts/install-openbao.sh \
  /absolute/private/prepared/node-1 "$ENCLAVR_NODE_ROOT" 0 0

The four arguments are prepared directory, deployed node root, mount UID and mount GID. Native Linux uses 0 0. Docker Desktop uses the invoking desktop account's "$(id -u)" "$(id -g)" instead. The installer retains the named Raft volume and audit history, and an unchanged healthy deployment is a no-op. It does not initialize OpenBao or issue identities. Retain pending activation markers after failure.

2. Initialize the first voter and verify all members

Verify TLS reachability of every declared endpoint using its accepted CA. Uninitialized status is expected only for this positively new cluster. Existing initialized endpoints or inconsistent cluster identities require investigation.

Initialization uses the accepted local custody file. It captures the generated recovery share durably, merges it into .env-mwc-openbao, verifies read-back, and only then scrubs the temporary share checkpoint. No external sign-in is required.

Save initialize.json:

{
  "operation":"initialize",
  "namespace":"<namespace>",
  "bootstrap_file":"/home/operator/core.enclavr.io/.env-mwc-openbao",
  "spec_file":"/home/operator/.local/state/enclavr/openbao-admin/cluster.json",
  "state_file":"/home/operator/.local/state/enclavr/openbao-admin/initialization.json"
}

./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/initialize.json"
./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/initialize.json" --apply --attended

Only the first endpoint receives initialization. The other nodes join that cluster through retry_join; never initialize every node separately. Enclavr uses one recovery share with threshold one, pins the resulting cluster ID, verifies local recovery custody and retains the initial root only in the protected administrator checkpoint. .env-mwc-openbao retains the recovery_share, recovery_sha256, seal_key, seal_key_id, cluster_id and accepted cluster_tls material.

Preserve initialization.json, any independent capture/intent files, and the original administrator workstation if an operation is interrupted. A lost response is not permission to repeat sys/init. The supported retry uses the same request and checkpoints. Share/capture material is scrubbed only after custody read-back succeeds.

3. Issue identities and accept consumers

For a single host consumer, launch ./enclavr and select Operations → Issue host identity. The form collects the required values, validates local checkpoint collisions, reviews the exact scope and requires y before issuing. No operation-request JSON is required. See host enrollment for the form workflow; the CLI examples below also support other service roles.

Save a provision.json request. This small example issues an observer and a snapshot-only identity for backup-host; add each actual consumer's complete explicit item list and separate writer, DNS, audit and Caddy-storage scopes before applying.

{
  "operation":"provision",
  "namespace":"<namespace>",
  "state_file":"/home/operator/.local/state/enclavr/openbao-admin/initialization.json",
  "output_directory":"/home/operator/.local/state/enclavr/openbao-admin/identities",
  "identities":[
    {"name":"cluster-observer","hostname":"node-1.<namespace>","cluster_health":true},
    {"name":"snapshot-collector","hostname":"backup-host","snapshot":true},
    {"name":"node-1-consumer","hostname":"node-1.<namespace>","items":["assets.<namespace>","secrets.<namespace>","users.<namespace>"]}
  ]
}

Plan and apply with openbao-ops. The core three-item consumer example is sufficient only if the actual host references no additional items. Include all personal/service references required by its validated plan. Keep <name>.state.json and issuance state with the administrator; deliver only <name>.connection.json to its intended host, mode 0600. Repeating the same provisioning request preserves accepted credentials; do not remove checkpoints to bypass a revoked identity or scope conflict.

For a new host, use the host enrollment workflow after issuing its dedicated connection. host enroll and remote enroll validate the current authoritative inventory without a migration receipt. Newly issued connections bind hostname; legacy accepted connections keep their existing bytes when provisioning repeats. The migration-specific accept-consumer operation below remains for consumers already listed in the retained receipt.

For an existing OpenBao namespace, preserve its accepted source receipt and consumer evidence. Each row describes properties in a protected request with operation and namespace. Legacy migrate, verify-migration, cutover, rollback and retire operations have been removed; do not invoke old runbooks to recover missing state.

For example, save and apply this request on node-1.<namespace>, after its complete connection and migration receipt are installed privately:

{
  "operation":"accept-consumer",
  "namespace":"<namespace>",
  "hostname":"node-1.<namespace>",
  "receipt_file":"/home/operator/.local/state/enclavr/openbao-admin/migration.json",
  "connection_file":"/home/operator/.local/state/enclavr-openbao-consumer/connection.json",
  "acceptance_file":"/home/operator/.local/state/enclavr/openbao-admin/node-1-acceptance.json"
}

The protected runtime .env selects openbao and the host's absolute accepted connection path. Restart only consumers whose inputs changed, then verify their actual reads. A receipt alone does not restart services.

Do not revoke root until all intended hosts, backups, audit delivery, an isolated restore and attended recovery have acceptance evidence. Preserve the scrubbed initialization checkpoint after revocation. Restore procedures do not select an external runtime provider.

Recover local files

First distinguish missing local inputs from missing cluster data:

docker inspect enclavr-openbao --format '{{range .Mounts}}{{println .Type .Name .Source "->" .Destination}}{{end}}'
docker volume inspect enclavr-openbao-raft --format '{{.Name}}'
docker inspect enclavr-openbao --format '{{.Image}} {{.State.Status}}'

Inspect paths and metadata, not environment values or full connection contents. Inventory the current and archived checkouts, administrator workstation and protected backups. Restore the same host's accepted connection, node bundle, receipts and checkpoints with their permissions; do not substitute another host's identity. Compare pinned namespace, cluster ID, CA, endpoint set and issuance state before use. A directory timestamp does not establish which credential/checkpoint is authoritative.

When a checkout moves, retain the live Raft volume, node root and audit directory. Update absolute references in .env, container bind sources, Compose working directories, systemd units/timers, launchd jobs, installed binary symlinks and protected operation requests. Preserve files newer than an archive unless authoritative state proves otherwise. Recreate only the affected container after checking its mounts and retaining the same image, named volumes and credentials. Never run docker compose down -v during migration.

The initialization checkpoint is not recreated by copying openbao-unseal.<namespace>. The seal item alone does not supply an administrator root, recovery share, custody digest or identity issuance checkpoint. For the checkpoint-based CLI operations, locate the original protected administrator state/backup if initialization.json is missing; do not fabricate a partial replacement. The TUI administrator workflow below uses the accepted connection and live OpenBao recovery authentication instead; it does not require or reconstruct that checkpoint.

After restoration, check the configured consumer with:

./enclavr namespace inspect --namespace "$ENCLAVR_NAMESPACE"
./enclavr config validate
./enclavr host plan

A Caddy publication error is a separate unmet prerequisite: publish the Caddyfile through the namespace writer. A storage identity is required only if that Caddyfile explicitly selects shared OpenBao certificate storage. Independent filesystem certificates require no storage AppRole or recovery workflow; see Update Caddy. Do not treat an archived Caddyfile as a published snapshot or substitute a local volume for a failed shared backend.

Recover scoped administration

Use this when the cluster has quorum and the initial root has been revoked. The revoked identity is the original bootstrap root token; a newly issued host consumer identity is used for enrollment, not unenrollment. The namespace writer's ordinary AppRole cannot create policies or AppRoles.

TUI administrator session

On the current namespace writer, launch ./enclavr and select Operations → Obtain administrator access. Open OpenBao recovery key share and enter only the complete secret value. If you saved the entire JSON record:

• Plain JSON: copy the value of recovery_share.
• Exported item: under details.sections[].fields[], find the field with "n": "recovery_share" and copy its v value.

Paste that value without surrounding quotes, spaces or line breaks, retaining any final =. Do not paste the whole JSON or a file path. The separate static_seal_key / seal_key and TLS CA private key are not inputs to this form; recovery_sha256 is a digest, and cluster_id / seal_key_id are identifiers.

Example only — not a usable recovery key share:

RVhBTVBMRS1PTkxZLU5PVC1BLVJFQUwtS0VZLTEyMzQ=

This illustrates Enclavr's single-share format: 44 Base64 characters encoding 32 bytes. It encodes the dummy text EXAMPLE-ONLY-NOT-A-REAL-KEY-1234. Use your original saved share. The TUI provides this guidance beside the concealed input; PgUp/PgDn scrolls the help on smaller terminals.

Enclavr discovers the deployed voter directory from the existing local Docker mounts and displays it read-only, together with the namespace and cluster ID from the accepted connection. No checkpoint path, JSON file or cross-host copy is needed. OpenBao authenticates the share against its existing recovery key; Enclavr pins the accepted cluster ID and TLS CA and rechecks ownership and paths before use. Press r to review, then y to confirm. Other hosts refuse the workflow and identify the current writer.

Enclavr checks cluster health and Docker network isolation, temporarily opens the loopback listener, exchanges generated root authority for a non-renewable token issued through the OpenBao administrator token role, revokes the generated root and closes the listener. That server role is initialized with a four-hour explicit maximum, orphan tokens and renewal disabled. Subsequent issuance uses the role's configured lifetime. Tokens and recovery input remain in process memory. The listener worker also closes on cancellation, pipe loss or its separate ten-minute recovery-operation timeout.

Operations → Issue host identity reuses this authority. If authentication is needed, it opens the same workflow while retaining the identity draft. Review and confirm issuance after returning. Operations → End administrator access revokes early. Ctrl+C cancels active work immediately and asks for confirmation before exit. There is no TUI expiry timer or session-duration setting. OpenBao enforces token expiry; the TUI stays open and privileged actions require fresh attended authentication. Exit mentions token revocation only when authority is held. The attended review includes setting namespace Enclavr AppRole token TTL, maximum TTL and explicit maximum TTL to four hours, preserving credentials, policies and restrictions. Cleanup errors remain visible and block privileged use until cleanup succeeds. A hard crash cannot guarantee immediate revocation; the temporary token still expires in OpenBao. The brief generated-root exchange and a host crash interrupting listener cleanup remain separate recovery risks.

Existing CLI recovery procedure

The following internal CLI procedure requires the original scrubbed initialization.json with pinned cluster/CA/endpoints and recovery digest, plus the matching operator-held share. It remains available for existing service scope operations. It uses protected files and its separate root checkpoint; the TUI session workflow above never calls it or persists administrator tokens.

1. Read attended custody into a protected recovery file

Use the operator-supplied .env-mwc-openbao file. If the ceremony runs elsewhere, prepare a protected JSON containing only version, namespace, cluster_id and recovery_share, and transport it over verified SSH. Do not send the combined custody file, seal or CA signing key to the ceremony node. The recovery reader checks the namespace, cluster and original initialization digest before sending any recovery request. Supply the absolute path as recovery_file; external administrator sessions are rejected. Keep a temporary share export's lifetime short.

2. Enable the temporary listener on one voter

Verify all voters are healthy and one restart preserves quorum. Check that the selected node's dedicated Compose network has no other containers. On that node, as an identity able to read its protected deployed bundle:

./enclavr internal openbao-recovery-listener-prepare \
  --source-dir "$ENCLAVR_NODE_ROOT" --output-dir "$ENCLAVR_OPS/recovery-enabled" --enable
./enclavr internal openbao-recovery-listener-prepare \
  --source-dir "$ENCLAVR_NODE_ROOT" --output-dir "$ENCLAVR_OPS/recovery-enabled" --enable --apply
sudo bash src/scripts/install-openbao.sh \
  "$ENCLAVR_OPS/recovery-enabled" "$ENCLAVR_NODE_ROOT" 0 0

Use desktop UID/GID instead of 0 0 on macOS. On Linux, root-owned source files may require running the preparation CLI with sudo and retaining protected ownership for its output. Verify docker port enclavr-openbao 8290/tcp reports only 127.0.0.1:8290; never publish recovery on a tailnet or public address. The ordinary listener remains on 8200 with legacy unauthenticated root generation disabled.

3. Provision, revoke and close

For the existing CLI procedure and other service scopes, save recover-provision.json using its actual protected paths:

{
  "operation":"recover-provision",
  "namespace":"<namespace>",
  "state_file":"/home/operator/.local/state/enclavr/openbao-admin/initialization.json",
  "recovery_file":"/home/operator/.local/state/enclavr/openbao-admin/recovery.json",
  "recovery_endpoint":"https://127.0.0.1:8290",
  "output_directory":"/home/operator/.local/state/enclavr/openbao-admin/identities",
  "identities":[
    {"name":"caddy-storage-node-1","hostname":"node-1.<namespace>","caddy_storage":true}
  ]
}

./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/recover-provision.json"
./enclavr internal openbao-ops --request-file "$ENCLAVR_OPS/recover-provision.json" --apply --attended

The operation checks the supplied share against the pinned digest, generates and checkpoints a temporary root, issues only the requested scopes, revokes the root, verifies denial and checks the issued connections again. An unchanged successful repeat creates no root. After a failure, preserve temporary-root.pending and identity checkpoints; retry the same request or resolve the specific failure before removing recovery material.

Immediately disable the listener using a fresh complete bundle:

./enclavr internal openbao-recovery-listener-prepare \
  --source-dir "$ENCLAVR_NODE_ROOT" --output-dir "$ENCLAVR_OPS/recovery-disabled" --disable --apply
sudo bash src/scripts/install-openbao.sh \
  "$ENCLAVR_OPS/recovery-disabled" "$ENCLAVR_NODE_ROOT" 0 0
docker port enclavr-openbao

Verify 8290 is absent, all voters are healthy, the new scope works, out-of-scope reads remain denied and temporary-root revocation is complete. Remove only the temporary recovery exports after those checks; retain scrubbed administrator/issuance checkpoints. OpenBao 2.6 changed the native operator generate-root command to the authenticated workflow; it is not a substitute for Enclavr's controlled temporary-listener ceremony. See the upstream legacy recovery API.

Backups on backup-host

Give backup-host only its accepted snapshot connection. Store snapshots in a private directory on persistent backup storage; keep operator recovery keys in protected repo-root .env-mwc-openbao custody, separately from snapshot archives. A protected administrator backup must separately retain cluster/node specifications, CA/TLS custody, scrubbed initialization and issuance checkpoints, migration/acceptance receipts, and necessary local operation configuration. These filesystem artifacts are not all contained in a Raft snapshot.

On backup-host, save backup.json:

{
  "operation":"backup",
  "namespace":"<namespace>",
  "connection_file":"/home/operator/.local/state/enclavr-openbao-backup/connection.json",
  "output_directory":"/srv/enclavr-openbao/snapshots"
}

./enclavr internal openbao-ops --request-file /absolute/private/backup.json
./enclavr internal openbao-ops --request-file /absolute/private/backup.json --apply

The caller must own the protected destination. The implementation pins its cluster in identity.json, validates archive structure/checksums before publication, retains 30 UTC daily dates and 180 calendar months, and records snapshot-status.json. A repeat on the same UTC day verifies the immutable existing snapshot; it does not create a newer recovery point. That distinction matters before an incident or a destructive maintenance window.

The built-in combined collect/scheduled collector currently has a deployment-specific backup-host allowlist and requires five peers. Merely substituting the literal hostname backup-host in a collector request will fail validation. The standalone backup operation above has no such hostname restriction. Generalizing the scheduled collector requires a separately reviewed implementation change; this documentation does not claim arbitrary-host audit scheduling is supported today. Existing accepted collectors retain seven calendar years of audit segments and acknowledge only verified durable copies. Their health checks require snapshots and every peer's audit delivery within 26 hours.

Retrieve the selected accepted archive over verified SSH without copying administrator keys onto backup-host:

scp 'backup-host:/srv/enclavr-openbao/snapshots/daily/<YYYY-MM-DD>.snap' \
  "$ENCLAVR_OPS/selected.snap"
chmod 600 "$ENCLAVR_OPS/selected.snap"

Use the configured restricted backup account where applicable. Record the selected timestamp, cluster identity and checksum against the backup inventory. Structural verification is not proof that the matching seal material can restore the snapshot; perform the isolated restore below.

Inspect health and authenticate an operator API session

Read a known accepted connection's CA and non-secret cluster pin locally. Do not print its RoleID/SecretID. Set the endpoint to an exact accepted origin, then check it before transmitting credentials:

ENCLAVR_CONNECTION="$ENCLAVR_OPS/identities/cluster-observer.connection.json"
ENCLAVR_ENDPOINT='https://<accepted-tailnet-ip>:8200'
jq -r '.ca_certificate' "$ENCLAVR_CONNECTION" > "$ENCLAVR_OPS/ca.pem"
curl --noproxy '*' --cacert "$ENCLAVR_OPS/ca.pem" --silent --show-error \
  "$ENCLAVR_ENDPOINT/v1/sys/health?standbyok=true" > "$ENCLAVR_OPS/health.json"
jq -e --slurpfile c "$ENCLAVR_CONNECTION" \
  '.initialized == true and .sealed == false and .cluster_id == $c[0].cluster_id' \
  "$ENCLAVR_OPS/health.json"
curl --noproxy '*' --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  "$ENCLAVR_ENDPOINT/v1/sys/leader" | jq '{is_self,leader_address,leader_cluster_address}'

Check every expected origin. Health alone does not verify membership or catch-up. An unavailable node must be accounted for in the quorum calculation. Select the observed active origin only after matching it to the accepted endpoint inventory and repeating its cluster check. Do not follow credential-bearing redirects to an unverified origin, disable TLS verification, or use ambient proxy/CA/token settings.

For manual upstream read-only membership inspection, authenticate the observer through protected files:

jq '{role_id,secret_id}' "$ENCLAVR_CONNECTION" > "$ENCLAVR_OPS/login-request.json"
curl --noproxy '*' --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  --request POST --data-binary @"$ENCLAVR_OPS/login-request.json" \
  "$ENCLAVR_ENDPOINT/v1/auth/approle/login" > "$ENCLAVR_OPS/login-response.json"
jq -er '.auth.client_token | select(type == "string" and length > 0) | "header = \"X-Vault-Token: \(.)\""' \
  "$ENCLAVR_OPS/login-response.json" > "$ENCLAVR_OPS/operator.curl"
curl --config "$ENCLAVR_OPS/operator.curl" --noproxy '*' \
  --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  "$ENCLAVR_ENDPOINT/v1/sys/storage/raft/configuration" \
  | jq '.data.config.servers | map({node_id,address,leader,voter})'

Compare the full expected voter set and cluster-port addresses. At the end of the session revoke the short-lived token using the same protected configuration:

curl --config "$ENCLAVR_OPS/operator.curl" --noproxy '*' \
  --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  --request POST "$ENCLAVR_ENDPOINT/v1/auth/token/revoke-self"

Verify a subsequent auth/token/lookup-self returns HTTP 403, then remove the temporary login request/response and curl configuration. Manual examples do not broaden the issued policy. The observer and namespace writer both lack cluster mutation authority.

Change leadership

Three voters need two for quorum; five need three. Before maintenance, verify the full voter set, unsealed health, a single agreed leader and caught-up Raft applied/committed indices. Also verify the current leader can retrieve its seal bootstrap before a restart. A running node's health alone does not establish that startup credentials still work.

With an explicitly authorized operator token having sudo/root authority on sys/step-down, place its header in the protected operator.curl file (never argv) and call the current active endpoint:

curl --config "$ENCLAVR_OPS/operator.curl" --noproxy '*' \
  --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  --request POST "$ENCLAVR_ENDPOINT/v1/sys/step-down"

The API releases leadership; it accepts no target-node parameter. Election chooses the next leader. Enclavr does not issue a general cluster-administrator AppRole by default, and its observer/writer cannot call this endpoint. Do not widen those roles to work around a 403. See the step-down API.

An attended host administrator can instead gracefully restart only the current leader, after confirming all other voters are healthy and quorum survives its absence:

docker exec enclavr-openbao /usr/local/bin/enclavr-openbao \
  --namespace "$ENCLAVR_NAMESPACE" --key-id '<accepted-seal-key-id>' --check-bootstrap
docker restart --time 60 enclavr-openbao

Run on the verified leader host. This may hand off leadership, but does not guarantee a particular successor. Inspect sys/leader on every node and the observer's Raft configuration afterward. Confirm the previous leader returns unsealed as a member and catches up. If a requested host did not become leader, report that outcome; do not cycle healthy nodes or degrade quorum to force it. There is no persistent preferred-leader setting in the Enclavr integration.

Recover quorum

Temporary member outage

Recover original nodes, addresses, connectivity on API 8200 / cluster 8201, accepted TLS/seal bootstrap and their existing volumes. Start a stopped required container with docker start enclavr-openbao; do not rebuild or reinitialize it. Bring back enough voters for the existing membership to form a majority, then recover the rest. Do not edit membership just because a host is temporarily unreachable.

With quorum maintained, a permanently lost member can be removed through the upstream sys/storage/raft/remove-peer API using explicit administrative authority and a protected JSON body such as {"server_id":"<lost-node-id>"}. Remove one only after confirming its exact identity and replacement plan. Configure replacement nodes from the accepted topology with fresh, separately retained volumes; their retry_join joins the existing leader. Never initialize a joining node. See the Raft API.

Permanent loss of a majority

Use manual membership recovery only when the missing majority cannot be restored. It can lose acknowledged state missing from survivors and can commit previously uncommitted log entries. Fence every excluded node so an old majority cannot restart alongside the recovered cluster. Stop all survivors, disable automatic restarts during the repair, and take an offline protected copy of each survivor's complete Raft volume. Keep originals and copies until acceptance.

On each survivor, record the existing image and restart policy, then stop it:

ENCLAVR_IMAGE=$(docker inspect enclavr-openbao --format '{{.Image}}')
docker inspect enclavr-openbao --format '{{.HostConfig.RestartPolicy.Name}}'
docker update --restart=no enclavr-openbao
docker stop --time 60 enclavr-openbao
docker run --rm --network none --read-only --cap-drop ALL \
  --security-opt no-new-privileges:true --entrypoint /bin/tar \
  --mount type=volume,src=enclavr-openbao-raft,dst=/data,readonly \
  "$ENCLAVR_IMAGE" -C /data -cpf - . > "$ENCLAVR_OPS/raft-before-repair.tar"

Verify the offline copy completed, can be listed and has a recorded checksum before proceeding. Use unique backup names per host and incident; never overwrite an earlier accepted copy. Keep node bundles/audit state too. Docker Desktop volumes live inside Docker's VM; do not assume a Linux host /var/lib/docker/volumes path.

Choose the trustworthy surviving set using available replication evidence. Write an identical protected peers.json for that exact set. An emergency single-survivor example is:

[
  {"id":"node-1","address":"100.64.0.11:8201","non_voter":false}
]

Use the real stable node ID and cluster address, not the API port or a guessed hostname. If retaining more survivors, include every retained node in the same array on every survivor. With all selected containers stopped, install into the named volume's raft/peers.json:

docker run --rm -i --network none --read-only --cap-drop ALL \
  --security-opt no-new-privileges:true --entrypoint /bin/bash \
  --mount type=volume,src=enclavr-openbao-raft,dst=/data \
  "$ENCLAVR_IMAGE" -euc \
  'test -d /data/raft; test ! -e /data/raft/peers.json; umask 077; cat > /data/raft/peers.json' \
  < "$ENCLAVR_OPS/peers.json"
docker start enclavr-openbao

Start all retained survivors after the same file has been installed on each. Preserve diagnostics if startup fails; do not repeatedly rewrite recovery membership. Check that the recovery file was consumed, the pinned cluster ID is unchanged, a leader exists and surviving replicas agree. Restore the recorded restart policy after successful recovery (docker update --restart=always enclavr-openbao for the canonical node).

Rebuild excluded nodes with fresh replacement volumes, keeping their old divergent volumes offline as evidence. Rejoin using the accepted configuration and matching seal. Return to the declared three/five-voter topology, verify replication, scoped reads/writes, DNS/Caddy and backups before declaring normal service. One-survivor recovery is an emergency transition, not an approved production topology. Upstream explains the data-loss semantics and required identical files in manual Raft recovery.

If no usable survivor remains, follow snapshot recovery. bao server -recovery is a separate raw-storage repair mode, not a substitute for Raft majority recovery; see upstream recovery mode.

Restore a snapshot

There is no enclavr ... restore operation. Use the upstream non-force snapshot restore endpoint, first in an isolated restore exercise. A snapshot rewinds documents, policy, identities and tokens to the selected time. Later credentials/receipts may no longer match; previously revoked credentials present in an older snapshot may reappear. Keep restored state fenced until those differences are reconciled.

You need the accepted snapshot from backup-host, its matching static_seal_key and seal_key_id from the operator's matching .env-mwc-openbao custody, accepted TLS material, and authority on the target. A snapshot-only connection cannot restore. A recovery share cannot decrypt a snapshot, and snapshot-force/-force does not recover a missing key. Preserve old key generations while their snapshots remain retained.

Isolated restore exercise

Use the isolated restore commands, which create a separate container with no network and no production mounts. They initialize only that positively empty scratch target, restore the snapshot using its matching static seal, and verify the original cluster identity and scoped data. The real production volume and runtime selectors stay untouched. Record actual results; a successful archive checksum alone is not recovery acceptance.

Restore the actual service

Choose the target deliberately. If a cluster still operates, stop application writers/consumers and take an additional current snapshot before rollback. If the whole cluster is lost, fence all old nodes, restore accepted node bundles and matching seal bootstrap on three/five replacement nodes, and use new empty volumes. Existing survivors use quorum recovery instead of new initialization.

For a total-loss replacement only: positively verify the new seed is uninitialized (GET /v1/sys/init returns initialized:false) and that all replacement volumes were newly created. A failed query is not absence. Initialize that new seed once through the native API into a protected response file, solely to obtain temporary restore authority:

curl --noproxy '*' --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  --request PUT --data '{"recovery_shares":1,"recovery_threshold":1}' \
  "$ENCLAVR_ENDPOINT/v1/sys/init" > "$ENCLAVR_OPS/replacement-init.json"
jq -er '.root_token | select(type == "string" and length > 0) | "header = \"X-Vault-Token: \(.)\""' \
  "$ENCLAVR_OPS/replacement-init.json" > "$ENCLAVR_OPS/operator.curl"

This is an attended disaster restore exception, not the normal Enclavr initializer or a new namespace. Preserve this response and stop on any uncertain result; do not repeat initialization. The other replacement nodes join the seed with matching seal/TLS, and must reach healthy quorum before restore. For an existing operational target, skip initialization and obtain explicit restore-capable administrative authority through the approved attended process instead.

On the verified active target, with the protected temporary/authorized header:

curl --config "$ENCLAVR_OPS/operator.curl" --noproxy '*' \
  --cacert "$ENCLAVR_OPS/ca.pem" --fail --silent --show-error \
  --request POST --data-binary @"$ENCLAVR_OPS/selected.snap" \
  "$ENCLAVR_ENDPOINT/v1/sys/storage/raft/snapshot"

If seal verification fails, stop and locate the matching generation; do not add -force. After a transport interruption, inspect target identity/state before retrying. The target's temporary initialization credentials cease to be authoritative after restoration; use credentials present in the snapshot or the matching attended recovery-share ceremony. Verify the restored original cluster ID and every member, reconcile revived/revoked identities, and compare full documents/versions against the selected recovery point.

Complete acceptance before enabling consumers: all intended voters unsealed and caught up; exactly one leader; expected membership/addresses; scoped consumer and writer checks; migration/receipt consistency; current TLS; successful DNS/Caddy adoption; fresh off-node snapshot and audit delivery; temporary administrative authority revoked and recovery listeners removed. Retain the pre-restore backups and evidence. Reaccept changed consumer connections explicitly—do not hand-edit pins to suppress a mismatch. The upstream snapshot API defines restore and force semantics.