Isolated OpenBao snapshot restore

Use this attended exercise before accepting backups or planning a real rollback. It uses the existing Enclavr OpenBao image and the upstream API through its bundled CLI. The scratch container has no network, no published ports, no production volume and no 1Password credentials. Its Raft, audit, key and token state are memory-backed and disappear when stopped. This is a restore test, not a replacement production node.

Return to the operations runbook for restoring actual service and quorum. Never run these initialization commands against an existing production container. They are scoped to the new container named enclavr-openbao-restore throughout.

Prepare protected inputs

Use a trusted host with Docker, Bash, jq and xxd, an accepted local OpenBao runtime image, and enough memory for the snapshot plus a running server. Disable shell tracing and stop after a failed command. Keep each exercise's evidence separately; do not overwrite an existing scratch directory or container.

umask 077
ENCLAVR_NAMESPACE='<namespace>'
ENCLAVR_CLUSTER_ID='<cluster_id>'
ENCLAVR_RESTORE="$HOME/.local/state/enclavr/openbao-restore-<incident-id>"
mkdir -m 700 "$ENCLAVR_RESTORE"
mkdir -m 700 "$ENCLAVR_RESTORE/config" "$ENCLAVR_RESTORE/input"
ENCLAVR_IMAGE=$(docker image inspect enclavr-openbao:2.7.1 --format '{{.Id}}')

Select the actual accepted image instead if the installation has advanced; do not pull or downgrade a release during recovery. Record its immutable image ID. Retrieve a selected accepted archive from backup-host:

scp 'backup-host:/srv/enclavr-openbao/snapshots/daily/<YYYY-MM-DD>.snap' \
  "$ENCLAVR_RESTORE/input/selected.snap"
chmod 600 "$ENCLAVR_RESTORE/input/selected.snap"
sha256sum "$ENCLAVR_RESTORE/input/selected.snap" > "$ENCLAVR_RESTORE/snapshot.sha256"

On macOS use shasum -a 256 if sha256sum is unavailable. Compare the checksum/date/cluster to the protected backup inventory. Copy only an accepted node's ca.pem, server.pem and server.key into this exercise's config/ over protected transport. Its server certificate must be valid for 127.0.0.1; verify its CA and validity before proceeding. Do not mount a whole node bundle or bootstrap file.

Use the existing operator-owned custody JSON validated by the attended operations procedure. It must be a regular 0600 file under protected parents, with the accepted namespace, initialized cluster ID, and seal generation matching the snapshot. Recover a missing backup manually outside Enclavr before continuing; do not authenticate an external secret store or generate replacement material. Set the absolute path to that existing file:

ENCLAVR_CUSTODY='/absolute/private/.env-mwc-openbao'
jq -e --arg namespace "$ENCLAVR_NAMESPACE" --arg cluster "$ENCLAVR_CLUSTER_ID" \
  '.version == 1 and .namespace == $namespace and .cluster_id == $cluster' \
  "$ENCLAVR_CUSTODY" >/dev/null
jq -erj '.seal_key | select(type == "string" and test("^[0-9a-fA-F]{64}$"))' \
  "$ENCLAVR_CUSTODY" > "$ENCLAVR_RESTORE/seal-key.hex"
jq -erj '.seal_key_id | select(type == "string" and length > 0)' \
  "$ENCLAVR_CUSTODY" > "$ENCLAVR_RESTORE/seal-key-id.input"
jq -erj '.recovery_share | select(type == "string" and length > 0)' \
  "$ENCLAVR_CUSTODY" > "$ENCLAVR_RESTORE/recovery-share.input"
jq -Rse 'test("^[0-9a-fA-F]{64}$")' "$ENCLAVR_RESTORE/seal-key.hex"

A failed read or mismatch stops the exercise. The recovery share is used later to prove administration of the restored copy; the seal key is needed first to decrypt the snapshot. Keep the complete operator custody file outside the scratch container; deliver only these selected inputs.

Create the isolated server configuration. This has no retry_join, a separate node ID and only container-loopback listeners. The temporary legacy recovery endpoint is enabled only inside this network-free container:

jq -n --rawfile keyid "$ENCLAVR_RESTORE/seal-key-id.input" '
{
  ui:false,
  disable_mlock:true,
  api_addr:"https://127.0.0.1:8200",
  cluster_addr:"https://127.0.0.1:8201",
  storage:{raft:{path:"/var/lib/enclavr-openbao",node_id:"restore-only"}},
  listener:{tcp:{
    address:"127.0.0.1:8200",cluster_address:"127.0.0.1:8201",
    tls_cert_file:"/restore-config/server.pem",tls_key_file:"/restore-config/server.key",
    tls_min_version:"tls13",disable_unauthed_generate_root_endpoints:false
  }},
  seal:{static:{current_key_id:$keyid,current_key:"file:///run/enclavr-openbao/current.key"}},
  audit:[{file:{enclavr:{description:"Isolated restore exercise",options:{
    file_path:"/var/log/enclavr-openbao/audit.log",mode:"0600",log_raw:"false"
  }}}}]
}' > "$ENCLAVR_RESTORE/config/server.json"
chmod 600 "$ENCLAVR_RESTORE"/config/*

Start a positively empty target

Size the memory/tmpfs limits for the restored database; the following limits are examples, not a guarantee that a large archive fits. --memory-swap equal to --memory prevents container swap. Docker must be able to read the private input directories (use the accepted desktop account on macOS).

docker run -d --name enclavr-openbao-restore --network none --read-only \
  --cap-drop ALL --security-opt no-new-privileges:true \
  --user "$(id -u):$(id -g)" \
  --memory 4g --memory-swap 4g \
  --tmpfs "/run/enclavr-openbao:rw,nosuid,nodev,noexec,mode=0700,uid=$(id -u),gid=$(id -g),size=16m" \
  --tmpfs "/var/lib/enclavr-openbao:rw,nosuid,nodev,noexec,mode=0700,uid=$(id -u),gid=$(id -g),size=2g" \
  --tmpfs "/var/log/enclavr-openbao:rw,nosuid,nodev,noexec,mode=0700,uid=$(id -u),gid=$(id -g),size=256m" \
  --mount "type=bind,src=$ENCLAVR_RESTORE/config,dst=/restore-config,readonly" \
  --mount "type=bind,src=$ENCLAVR_RESTORE/input,dst=/restore-input,readonly" \
  --env BAO_ADDR=https://127.0.0.1:8200 --env BAO_CACERT=/restore-config/ca.pem \
  --env BAO_DISABLE_REDIRECTS=true \
  --entrypoint /bin/sleep "$ENCLAVR_IMAGE" infinity
xxd -r -p "$ENCLAVR_RESTORE/seal-key.hex" \
  | docker exec -i enclavr-openbao-restore /bin/bash -euc \
    'umask 077; cat > /run/enclavr-openbao/current.key; test "$(wc -c < /run/enclavr-openbao/current.key)" -eq 32'
docker exec -d enclavr-openbao-restore /bin/bash -euc \
  'umask 077; exec bao server -config=/restore-config/server.json > /run/enclavr-openbao/server.log 2>&1'
docker exec enclavr-openbao-restore bao status -format=json \
  > "$ENCLAVR_RESTORE/before-init.json"

Wait for the server to answer. Native bao status returns exit 2 while sealed/uninitialized; that expected status is not permission to ignore other failures. Inspect the protected JSON and require initialized:false. The running container itself is not proof the detached server started successfully.

jq -e '.initialized == false' "$ENCLAVR_RESTORE/before-init.json"
docker exec enclavr-openbao-restore bao operator init \
  -recovery-shares=1 -recovery-threshold=1 -format=json \
  > "$ENCLAVR_RESTORE/scratch-init.json"

Initialize only once, preserving the response. If it is lost or uncertain, stop this exercise and investigate; do not repeat initialization against an initialized volume. Wait for bao status to report unsealed before proceeding.

Load the scratch root into the bundled CLI's default token file via stdin. The image already uses /run/enclavr-openbao as its home; this file remains in its private tmpfs. No root is put in arguments or the environment:

jq -er '.root_token | select(type == "string" and length > 0)' \
  "$ENCLAVR_RESTORE/scratch-init.json" \
  | docker exec -i enclavr-openbao-restore /bin/bash -euc \
    'umask 077; cat > /run/enclavr-openbao/.vault-token'
docker exec enclavr-openbao-restore bao operator raft snapshot restore \
  /restore-input/selected.snap

Do not add -force. A key mismatch needs the correct custody generation. This native command restores only the isolated target. See the snapshot restore command and token-file behavior.

Verify restored identity and attended recovery

Wait for the restored server to become unsealed, then require the original cluster ID:

docker exec enclavr-openbao-restore bao status -format=json \
  > "$ENCLAVR_RESTORE/after-restore.json"
jq -e --arg id "$ENCLAVR_CLUSTER_ID" \
  '.initialized == true and .sealed == false and .cluster_id == $id' \
  "$ENCLAVR_RESTORE/after-restore.json"
docker exec enclavr-openbao-restore rm /run/enclavr-openbao/.vault-token

The scratch initialization root is superseded by the restored state. The next commands use the matching original recovery share to prove attended recovery in this isolated copy. They use the legacy HTTP paths explicitly; OpenBao 2.6's native operator generate-root command uses a different authenticated endpoint.

First inspect for an existing ceremony and require started:false. Never cancel another operator's in-progress ceremony automatically:

docker exec enclavr-openbao-restore bao read -format=json sys/generate-root/attempt \
  > "$ENCLAVR_RESTORE/generation-before.json"
jq -e '(.data // .).started == false' "$ENCLAVR_RESTORE/generation-before.json"
docker exec enclavr-openbao-restore bao write -f -format=json sys/generate-root/attempt \
  > "$ENCLAVR_RESTORE/generation-attempt.json"
jq -e '(.data // .) | .required == 1 and (.nonce | length > 0) and (.otp | length > 0)' \
  "$ENCLAVR_RESTORE/generation-attempt.json"
jq --rawfile share "$ENCLAVR_RESTORE/recovery-share.input" \
  '(.data // .) | {nonce:.nonce,key:$share}' \
  "$ENCLAVR_RESTORE/generation-attempt.json" > "$ENCLAVR_RESTORE/generation-update.json"
docker exec -i enclavr-openbao-restore bao write -format=json sys/generate-root/update - \
  < "$ENCLAVR_RESTORE/generation-update.json" > "$ENCLAVR_RESTORE/generation-result.json"
jq -e '(.data // .) | .complete == true and (.encoded_token | length > 0)' \
  "$ENCLAVR_RESTORE/generation-result.json"
jq --slurpfile attempt "$ENCLAVR_RESTORE/generation-attempt.json" \
  '(.data // .) | {encoded_token:.encoded_token,otp:($attempt[0] | (.data // .).otp)}' \
  "$ENCLAVR_RESTORE/generation-result.json" > "$ENCLAVR_RESTORE/decode-request.json"
docker exec -i enclavr-openbao-restore bao write -format=json sys/decode-token - \
  < "$ENCLAVR_RESTORE/decode-request.json" > "$ENCLAVR_RESTORE/decoded-root.json"
jq -er '.data.token | select(type == "string" and length > 0)' \
  "$ENCLAVR_RESTORE/decoded-root.json" \
  | docker exec -i enclavr-openbao-restore /bin/bash -euc \
    'umask 077; cat > /run/enclavr-openbao/.vault-token'

All responses and payloads remain protected. The legacy generation API and decode API define these operations. This manual drill does not change Enclavr's production recover-provision contract.

Read the restored namespace items only into protected files, then compare complete payloads and versions against the selected backup's acceptance evidence. For example:

docker exec enclavr-openbao-restore bao read -format=json \
  "enclavr/data/$ENCLAVR_NAMESPACE/items/assets.$ENCLAVR_NAMESPACE" \
  > "$ENCLAVR_RESTORE/restored-assets.json"
docker exec enclavr-openbao-restore bao operator raft list-peers -format=json \
  > "$ENCLAVR_RESTORE/restored-peers.json"

Check secrets/users and all expected personal/service documents similarly without displaying them. Also exercise a snapshot-present scoped identity against this container only and prove out-of-scope denial. The isolated member inventory is a scratch topology; it does not prove production quorum or fresh audit collection.

Revoke the generated root only in the restored copy, verify lookup is denied, and preserve value-free evidence of the result:

docker exec enclavr-openbao-restore bao token revoke -self
docker exec enclavr-openbao-restore bao token lookup -format=json \
  > "$ENCLAVR_RESTORE/revoked-lookup.json" 2> "$ENCLAVR_RESTORE/revoked-lookup.error"

The lookup must fail with HTTP 403; a network error is not proof of revocation. Keep the container while investigating failures. After acceptance, stop and remove only the scratch container:

docker stop --time 60 enclavr-openbao-restore
docker rm enclavr-openbao-restore

Stopping discards its tmpfs database, key, audit and token files. Remove temporary host exports of custody, roots, generation payloads and restored secret documents according to protected-material handling; keep a value-free report with snapshot checksum/time, image ID, original cluster identity match, document/permission checks and revocation result. Leave the source backup and administrator custody intact. An isolated success must not be reported as an actual production restore or quorum-recovery exercise.