… on-failure debug

CI runs on 1786e69 and 38bf611 both showed Grafana healthcheck failing
("container mintkey-grafana-1 is unhealthy") at `docker compose up -d
--wait --timeout 120`. Local `docker compose config` (via root shim)
resolves Grafana volume paths to correct absolute paths under
infra/observability/, so the path resolution should work — but CI
behaves differently.

Two instrumentations to discriminate root cause:

1. Explicit `-f infra/compose/docker-compose.yml` on the start + teardown
   steps. Bypasses the root shim's `include:` directive (which might
   have subtle behavior differences with `--wait` in some Compose
   versions). Also bumps `--timeout` 120s→180s in case Grafana boots
   slowly in CI (Keycloak depends_on=service_healthy; Grafana
   depends_on=prometheus:service_healthy).

2. NEW debug step that runs on failure(): dumps `docker compose ps -a`
   and the last 100 lines of grafana logs (plus 50 keycloak + 30
   prometheus) so the NEXT failing run shows WHY grafana is unhealthy.

If the explicit -f form makes Playwright pass, the root cause is the
include: shim. If it still fails, the new debug step will reveal the
actual Grafana startup error in the CI environment.

Trivy + Dep Review will still fail (by-design); other Mintkey CI fixes
are queued for the new run.

…vices, errors surface

Three restore-correctness bugs surfaced when validating that
backup → down -v → restore actually achieves the user's success
criterion ("after a restore I do not need to change anything").

Bug #4: dev-backup.sh:367 — `pg_dump` was called without `--clean
--if-exists`. The resulting dump had no DROP statements. When
dev-restore.sh piped it into psql against a non-empty schema (e.g.,
a freshly-seeded one), every INSERT failed with duplicate-key /
already-exists errors. Fix: add `--clean --if-exists` so the dump
emits `DROP INDEX IF EXISTS` + `DROP TABLE IF EXISTS` at the head.

Bug #5: dev-restore.sh:441 — `psql ... 2>/dev/null` silenced ALL
postgres errors. A 100%-broken restore would report "✅ pg_dump
restored to mintkey database" and the operator would not know.
Fix: remove `2>/dev/null`; add `--set ON_ERROR_STOP=on` so psql
exits non-zero on the FIRST error (rather than racing through
the dump leaving the DB partly-applied).

Bug #6: dev-restore.sh — did NOT stop services that hold open
postgres connections (keycloak, admin-api, mcp-server, broker,
kong-syncer, admin-ui). With the new `--clean` dump, the DROP
TABLE statements would block on row locks held by those services.
Fix: `docker compose stop` the connection-holders before psql,
then `docker compose up -d --wait` after.

Verification (this commit):
- Took a backup with the fix; decrypted+gunzipped the dump:
    DROP TABLE statements:  ~110
    DROP INDEX statements:  ~300
    CREATE TABLE statements: ~106
    INSERT INTO statements: ~5  (just the seed-job-default rows)
  → Bug #4 confirmed fixed.

End-to-end round-trip test (backup → down -v → restore → verify
agent+service intact) is the next step; user will create an
agent + service first so there's real data to round-trip.

Counts now of session-discovered backup/restore bugs:
  #1 ✅ fixed (51e7077) pg_dump silently skipped — Service field
  #2 ✅ fixed (a47a4e6) vault-adapter detection — same class
  #3 ✅ fixed (49d519a) host-file admin_password drift
  #3a ✅ fixed (77ab040) Makefile colon-pattern blocked make admin-password
  #4 ✅ fixed (this commit) pg_dump --clean
  #5 ✅ fixed (this commit) psql errors surfaced
  #6 ✅ fixed (this commit) restore stops dependent services

…pter + proxy-plugin

Round-trip validation surfaced 2 more bugs after #4 #5 #6 were fixed.

Bug #7: dev-backup.sh:367 — pg_dump was called with `--no-privileges`,
which strips ALL `GRANT` / `REVOKE` statements from the dump. After
restore (DROP TABLE + CREATE TABLE), `mintkey_app` and
`mintkey_subscriber` had zero privileges on every table → every query
from admin-api / mcp-server / kong-syncer / proxy-plugin failed with
"permission denied for table services|agents|...".

Fix: drop `--no-privileges`. The dump now emits the same GRANTs that
liquibase migration 009-roles.yaml normally creates (212 GRANTs in a
fresh dump vs 0 before). Verified post-restore: agent key → 200 OK
without manual GRANT intervention.

Bug #8: dev-restore.sh — the service stop-list before the postgres
restore did not include `vault-adapter` or `proxy-plugin`. Two
distinct failures result:

  - vault-adapter holds an open SQLite file handle on
    /var/lib/vault/vault.db inside the vault_data volume. When that
    volume is replaced from the backup tarball, the kernel keeps
    serving the OLD inode to vault-adapter's file descriptor. Net
    effect: `GetCredential` returns "not found" for every credential
    even though the restored sqlite file is correct on disk. Proxy
    call → HTTP 502 "vault error".

  - proxy-plugin caches DEKs in-memory keyed by service_id. If
    service IDs survive the restore (they do — same UUIDs in the
    dump), the cache may shortcut to a stale "not found" or to an
    encrypted-DEK that the restored KEK can no longer decrypt.

Fix: add both to the stop-list. They get cleanly restarted by the
post-restore `docker compose up -d --wait`.

End-to-end verification with both fixes (round-trip test):
  - backup → docker compose down -v → up → dev-restore.sh
  - Zero manual SQL, zero manual restarts needed
  - Agent key works, request_token issues JWT, proxy call to GitHub
    API returns 200 authenticated as ciprianiacobescu
  - make admin-password returns the PRE-backup password
  - postgres row counts match baseline exactly (1/1/1/1/1/1/8)

Counts of session-discovered backup/restore bugs (all fixed):
  #1 ✅ pg_dump silently skipped — Service field (51e7077)
  #2 ✅ vault-adapter detection — same class (a47a4e6)
  #3 ✅ host-file admin_password drift (49d519a)
  #3a ✅ Makefile colon-pattern blocked `make admin-password` (77ab040)
  #4 ✅ pg_dump --clean --if-exists (a1dc9e3)
  #5 ✅ psql errors surfaced (a1dc9e3)
  #6 ✅ restore stops dependent services (a1dc9e3)
  #7 ✅ this commit — pg_dump includes GRANTs
  #8 ✅ this commit — restore stops vault-adapter + proxy-plugin

…lthchecks

Round-trip validation v3 surfaced 2 more bugs after #4–#8 were fixed.

Bug #9: dev-restore.sh processed manifest entries in order. The pg_dump
iteration stopped services, applied the dump, restarted them. Then the
volume iterations (vault_data, vault_kek, bootstrap_secrets) restored
tarballs WHILE vault-adapter / proxy-plugin were already running with
stale file handles into those volumes. Net effect: post-restore the
volumes were correct on disk, but vault-adapter served "GetCredential
not found" from the old inode → proxy → HTTP 502 "vault error".

Fix: after the manifest loop completes (so ALL volumes + the pg_dump
are restored), restart every data-dependent service ONCE. This drops
all stale handles + in-memory caches regardless of manifest order.

Bug #10: `docker compose restart` is fire-and-forget. It returns as
soon as the docker daemon kicks the containers, NOT when they're
actually responsive. End-to-end test showed admin-ui returning fast
enough to confuse the verifier into "Connection reset by peer" while
uvicorn was still bootstrapping in mcp-server / admin-api.

Fix: chain `docker compose up -d --wait --timeout 180` after the
restart. up -d is a no-op for already-running containers, but --wait
blocks until every service's compose healthcheck passes (or times out).
Callers can now hit /v1/tools/list_services immediately after the
script returns and get a real response, not Connection-reset.

Round-trip test v3 (with #7 #8 #9 #10):
  backup → docker compose down -v → up → dev-restore.sh → verify
  WITHOUT ANY manual SQL, manual GRANTs, manual restarts, or sleeps:
    [1/4] list_services     HTTP 200  ✅
    [2/4] request_token     HTTP 200  ✅
    [3/4] proxy → GitHub    HTTP 200  ✅  authenticated as ciprianiacobescu
    [4/4] admin password    yAPal2y2J... ✅ matches pre-backup

Success criterion met: "after restore I do not need to change anything."

Final tally of session-discovered backup/restore bugs (all fixed):
  #1 ✅ pg_dump silently skipped — Service field (51e7077)
  #2 ✅ vault-adapter detection — same class (a47a4e6)
  #3 ✅ host-file admin_password drift (49d519a)
  #3a ✅ Makefile colon-pattern blocked make admin-password (77ab040)
  #4 ✅ pg_dump --clean --if-exists (a1dc9e3)
  #5 ✅ psql errors surfaced (a1dc9e3)
  #6 ✅ restore stops dependent services (a1dc9e3)
  #7 ✅ pg_dump includes GRANTs (2fba9c3)
  #8 ✅ restore stops vault-adapter + proxy-plugin (2fba9c3)
  #9 ✅ this commit — post-loop service restart, manifest-order independent
  #10 ✅ this commit — restart waits for healthchecks