gateway_credentials at-rest encryption + master-key custody

Status: Active

Tracking: RIG-2863 (parent RIG-1715)

Addendum to the frozen record compass-server-llm-gateway (§Credential storage and rotation, L312–373). Scope: encryption-at-rest for the gateway_credentials value store ONLY. This record does not re-open the store’s shape, the CAS version discipline, the scope column, the RPC-vs-Postgres-direct wiring, or the stack-token channel — those stay as the frozen record decided. Ships as its own PR; frozen on merge; the credential-store build proceeds against it. One deliberate scope expansion, Matt-ruled (D6/T0): server-only secrets get a physically SEPARATE server_secrets store (mechanism C1) — a new table plus a second resolver instance — so the master key and the existing PEM/webhook/Linear server secrets are structurally undeliverable to agent containers (the table split PLUS the F1 membership guard on the user declare/set path — the split alone is necessary but not sufficient; see D6). C1 needs no public proto ENUM change: it REMOVES the SECRET_DELIVERY_SERVER_ONLY proto/store/secrets enum additions (and the FetchSecrets filter) the earlier delivery-kind mechanism required, and leaves the container FetchSecrets path byte-for-byte unchanged. It DOES add two admin-gated methods to SecretsService (proto/compass/v1/compass.proto, the public codegen lane) — an additive, non-breaking service-method addition, not a change to an existing wire enum.

Problem / Intent

The gateway record adds gateway_credentials, the FIRST value-persisting store in compass — deliberately breaking the names-only half of the secrets invariant. The invariant it breaks is stated at go/internal/secrets/secrets.go:20-22:

“Values live only in the provider and this process’s memory during a resolve; they are never persisted by Compass and never logged.”

The frozen record specifies the new store’s shape (“api_key and OAuth-shaped payloads (access/refresh/expiry), a monotonic version per row supplying the CAS substrate, and a scope column”, design.md:324-326) but says nothing about encryption at rest. A security red-team flagged this HIGH (CWE-311/312/522/532): a plaintext value column makes every pg_dump, read replica, backup, and operator SELECT a bulk all-tenant credential exfiltration path. The record already concedes the blast radius is total at the process level (design.md:333-337):

“Blast radius, stated: a compromised gateway (holding one stack token) can read every tenant’s provider credentials — the isolation boundary is the per-tenant pool scoping enforced server-side on the credential-list surface (below), not the TS process, which by design holds them all.”

That concession covers a compromised gateway process; it does not license a plaintext database. This addendum adds the missing at-rest-encryption and master-key-custody decision before the store is built.

Approach

Matt has ruled the core approach; this section records it as decisions.

D1 — Application-layer AES-256-GCM envelope encryption

The credential value payload (the full api_key / OAuth access+refresh+expiry JSON) is encrypted application-side before it reaches Postgres: AES-256-GCM, a fresh random 96-bit nonce per encryption (per write, never reused), authenticated. The gateway_credentials row stores ONLY value_ciphertext + value_nonce (+ key_version, D4); no plaintext value column exists in any migration version — the store is born encrypted. A DB dump, replica, or SELECT alone yields nothing usable.

Why the fresh-nonce discipline is absolute, quantified: reusing a 96-bit GCM nonce under one key forfeits both confidentiality AND authenticity — a two-message catastrophe (keystream XOR + authentication-key recovery), not a gradual weakening. Random 96-bit nonces are collision-safe only to ~2^32 encryptions per key (NIST SP 800-38D). The write rate is not one-per-credential: every hourly OAuth refresh write-back is a fresh encryption (“OAuth access/refresh tokens are refreshed by the gateway on the hour”, design.md:322-323). At plausible scale (10^3–10^4 credentials, hourly) that is ~10^7–10^8 encryptions/year — comfortably under 2^32, so v1’s single live key has years of headroom; key rotation (OQ-1) is the nonce-budget release valve when scale grows, which is itself part of why OQ-1’s deferral is safe.

This is the first at-rest crypto in the Go tree — verified by grep this session: no crypto/aes / cipher.NewGCM usage exists anywhere under go/ (existing crypto/rand uses are IDs/tokens/certs only, e.g. go/internal/store/ids.go:4). The envelope helper is therefore a new (small, justified) package: no existing abstraction carries symmetric at-rest crypto, and the seam must be reusable if a later store ever needs the same discipline.

D2 — Master key auto-provisioned into the existing SecretSpec provider

The master key never touches the DB and never requires a human step (rule://no-human-clicks). Every running Compass already has exactly one configured SecretSpec provider — the seam described at go/internal/secrets/secrets.go:10-13:

“this package reads that registry, generates the SecretSpec manifest the resolver resolves against, calls SecretSpec to resolve the actual values from the configured provider (keyring/1Password/Vault/…)”

On boot, Compass resolves the declared master-key secret (GATEWAY_CREDENTIALS_MASTER_KEY). If absent, it generates a fresh 256-bit key from crypto/rand and — serialized against concurrent booters through a Postgres advisory lock (T2) — provisions it:

• writes the value into the provider via secrets.Resolver.Set (go/internal/secrets/resolver.go:219, func (r *SpecResolver) Set(ctx context.Context, name, value string) error — “Set writes value into the provider for name via the pinned CLI, feeding the value on stdin (never argv, so it is not visible in the host process list)”, resolver.go:206-207);
• registers the name in the SEPARATE server_secrets store (D6) via the server-internal DeclareServerSecret (T0) — a mirror of store.DeclareSecret (go/internal/store/secrets.go:82, func (s *Store) DeclareSecret(ctx context.Context, actor AccountID, name string, delivery SecretDelivery, kind SecretKind, provider, host string) error — “It stores NO value — the value lives in the SecretSpec provider”, secrets.go:74-75) MINUS the delivery/kind routing parameters, which do not exist for server secrets (they are never container-delivered and never reach the T5 materializer) — never into the user secrets table, whose every row rides the container-delivery manifest;
• re-resolves and byte-compares before the key is ever used to encrypt (the read-back verify, T2).

Thereafter the key is resolved at boot — through the SERVER-SECRET resolver instance (D6), the second SpecResolver reading server_secrets — and held in process memory only, following the existing declared-secret boot-resolve pattern the Forge App PEM uses (newDeclaredSecretResolver, go/server/serve.go:1469: “resolves the declared server_only secret NAME to its raw value bytes on each call”, serve.go:1463-1464 — a “server_only” that is convention-only prose today; D6 makes it a physically separate store). Unlike the webhook secret’s per-request resolve, the master key is resolved once at startup and cached for the process lifetime (a decrypt happens on every credential read; a provider round-trip per decrypt would be the same amplification cachedWebhookSecret exists to prevent, serve.go:1484-1491).

D3 — Managed plane: KMS as just another provider URI, zero core change

The managed plane points the ONE declared master-key secret at a KMS-backed SecretSpec provider URI. The core stays cloud-agnostic; there is no KMS SDK, no cloud-conditional code path, and no self-host/managed fork in the crypto code. KMS-grade custody is a deployment configuration, not a code change.

D4 — Key versioning column from day one

Each row carries a key_version (smallint, starts at 1) identifying which master-key generation encrypted it. Whether v1 ships active rotation is an open question (OQ-1), but the column is load-bearing NOW: adding it later means a schema migration plus a backfill under ambiguity about which key encrypted which row. Cheap at creation, expensive retrofit.

D5 — Redaction discipline: Stringer alone is not enough

Breaking the never-persisted half of the invariant does not touch the never-logged half — but the established Stringer pattern (ResolvedSecret.String()/GoString(), go/internal/secrets/secrets.go:150-156:

func (s ResolvedSecret) String() string { return fmt.Sprintf("ResolvedSecret{name: %q, kind: %d, delivery: %d, value: <redacted>}", …) } func (s ResolvedSecret) GoString() string { return s.String() }

) covers only fmt-verb formatting — and slog under a TextHandler, which formats via the fmt path. It does NOT cover a slog JSONHandler (reflection over exported fields, ignores String()), json.Marshal, or direct field access — and T4’s decrypted-payload type is exactly the OAuth JSON shape that gets marshaled toward the gateway RPC, so it will have serializable fields. A one-line handler swap must not silently un-redact the fleet.

Therefore the value-bearing types this record adds MUST either:

• keep secret fields UNEXPORTED with accessor methods (the envelope.Key posture: unexported [32]byte, reflection-proof), or — where the payload shape needs exported fields — implement slog.LogValuer AND a redacting MarshalJSON alongside the String()/GoString() pair; and
• never serve as the RPC serialization type: the gateway-bound response is built by an explicit proto/DTO conversion, never json.Marshal of the payload type itself.

The store never logs a row’s value fields, and error paths wrap without embedding plaintext or key material.

D6 — A physically separate server_secrets store (mechanism C1, Matt-ruled)

The threat D2 must defeat: the declared-secrets registry is inject-all by design — “NO per-agent grant column (the MVP injects the whole store into every agent; per-agent scoping is a named FUTURE seam)” (go/internal/store/migrations/0001_init.sql:391-393). FetchSecrets resolves the WHOLE registry with no filter — resolved, err := h.resolver.Resolve(ctx, "runner fetch") then for _, s := range resolved { out = append(out, resolvedSecretToProto(s)) } (go/internal/runnerhub/handler.go:303-311) — and the Runner materializes every resolved value into the container pre-exec (resolved, err := h.link.FetchSecretsByContainer(ctx, name) → h.materializer.Install(ctx, handle.ID(), handle.HomeDir(), handle.WorkspaceUID(), resolved), go/internal/runner/host.go:378-383). Declared into the secrets table at all, the master key — which decrypts every tenant credential — would land on disk inside every untrusted agent sandbox, defeating the encryption entirely.

Matt ruled: a separate store, not a delivery flag. Server-only secrets get their OWN physically separate table, server_secrets, so they can NEVER get mixed up with user secrets that are delivered to agent containers. Container delivery becomes default-CLOSED by construction, not default-open-minus-a-filter.

Why the table boundary IS the delivery boundary: the delivery surface is the resolver’s MANIFEST. SpecResolver reads its declared set through the declarations interface — DeclaredSecrets(ctx context.Context) ([]store.SecretDeclaration, error) (go/internal/secrets/resolver.go:29-31; the store declarations struct field, resolver.go:57) — and buildManifest “renders the SecretSpec manifest TOML for a declared set: one [project] block and one [profiles.<profile>] block with every declared name as a required key” (resolver.go:99-101; the function, resolver.go:105-128); Resolve can only return names present in that manifest. Today ONE resolver instance (resolver := secrets.NewSpecResolver(st, secretsStateDir(cfg)), go/server/serve.go:528) serves BOTH the container path (FetchSecrets → Resolve, handler.go:303-311) and the boot consumers (PEM/webhook via newDeclaredSecretResolver → Resolve, serve.go:1469-1481). C1 splits the READ: a store view over server_secrets (ServerDeclaredSecrets, T0) feeds a SECOND SpecResolver instance. The container resolver keeps reading secrets; its manifest never contains a server-secret name, so a server secret is undeliverable to containers via the container manifest — there is nothing to filter. That structural property holds only while no name is ever present in BOTH tables: because the two tables share a keyspace and each name is a per-table PK with no cross-table exclusion, the same both-tables end state is reachable in EITHER declaration order — an authenticatedOpen SetSecret minting a shadow secrets row under a server-secret name, OR an admin SetServerSecret declaring a server secret under a name already live in secrets — and either gets that name container-delivered. The F1 membership guard is therefore SYMMETRIC (T0 — enforced on every path that can create a row in either table: reject a server_secrets name on the user declare/set path, and reject a secrets name on the admin server-secret path) and the boot reconcile is self-healing (below). The table split alone is necessary but not sufficient. Together the symmetric guard and the reconcile close every SEQUENTIAL declaration order. Each guard is a membership SELECT against the other table followed by an INSERT into this one, and the underlying declare/set/rollback trio it rides on “is not atomic and assumes no concurrent same-name writer” (secrets_service.go:88-91) — the same single-writer MVP assumption the existing SetSecret path already carries — so a concurrent same-name SetSecret/SetServerSecret pair remains the one residual both-tables window; the boot reconcile heals it on the next boot (see the full-scan note below). Closing that window atomically (a shared per-name advisory lock across both write paths) is deferred to the multi-writer work the cited comment already flags, tracked with the per-tenant defer (RIG-3237); this record does not widen the single-writer contract.

C1 keeps the SAME SecretSpec profile for both instances — the shared project is manifestProject = "compass" (resolver.go:19) and the profile defaultProfile = "default" (resolver.go:23; WithProfile exists, resolver.go:78, but C1 does not use it) — so the provider keyspace is shared and moving an existing secret between the two tables is a DB-row move with the provider value untouched (load-bearing for OQ-4).

C1 REMOVES the public proto ENUM change the previous delivery-kind mechanism required: no SECRET_DELIVERY_SERVER_ONLY proto enum value, no SecretDeliveryServerOnly store enum, no secrets-package DeliveryServerOnly, no widened migration CHECK, and no FetchSecrets delivery filter. The container FetchSecrets path is byte-for-byte UNCHANGED. The delivery enum gains no new value in any of the four representations — two live delivery kinds throughout (plus the proto3 UNSPECIFIED zero sentinel, which is not a delivery kind): delivery SMALLINT NOT NULL CHECK (delivery IN (0, 1)) (0001_init.sql:399); SecretDeliveryFile SecretDelivery = 0 / SecretDeliveryEnv SecretDelivery = 1 (go/internal/store/secrets.go:18-25); DeliveryFile / DeliveryEnv (go/internal/secrets/secrets.go:39-46); SECRET_DELIVERY_UNSPECIFIED = 0; SECRET_DELIVERY_FILE = 1; SECRET_DELIVERY_ENV = 2; (proto/compass/v1/compass.proto:160-164).

Writes into server_secrets go through a NEW admin-gated SetServerSecret/DeleteServerSecret RPC (T0) — mirroring the user SetSecret declare-then-Set flow (go/server/secrets_service.go:92) minus delivery/kind, classified adminOnly in classifyProcedure (go/internal/auth/admin_gate.go:47, :27; “An unrecognized path (ok=false) is treated as adminOnly — fail closed, never admit an unknown method as open”, admin_gate.go:44-46). Today’s user-facing SetSecret cannot declare a row as a server secret — secretRoutingFromProto admits only File/Env delivery (secrets_service.go:275-284), with no server-only delivery value — so operators declare server secrets through the new RPC. It CAN, however, mint an ordinary secrets row under a server-secret NAME (delivery=File/Env, an arbitrary name), which is exactly why the F1 membership guard is mandatory on this user path and not merely on the admin RPC.

This retroactively fixes a pre-existing exposure: the App PEM, webhook signing, and Linear OAuth secrets are server_only by convention only (“Declared server_only secret NAME holding the PRIMARY App PEM private key”, go/cmd/compass-server/main.go:414-417; webhook :418-421; Linear :435-450; boot-resolved by name via newDeclaredSecretResolver, serve.go:1469-1481) and today ride the same inject-all path into every agent container. Their declared rows MOVE from secrets into server_secrets in this PR chain (OQ-4, RESOLVED) — a DB-row move only, since the profile is shared.

D7 — One seal/open path for api_key and OAuth payloads

Both stored payload shapes — an OAuth token bundle and a bare api_key — are sealed and opened through the SAME envelope path (D1); there is no plaintext branch and no per-shape code fork. Both are secrets of the same sensitivity: a stored api_key is as long-lived and as disclosure-critical as an OAuth refresh token, and a split path would leave the most static secret class in plaintext for zero benefit. T3’s single encrypted column and T4’s single wiring path encode this. (Matt-ruled; resolves OQ-3.)

Alternatives considered

The core choice (envelope encryption + auto-provisioned key in the existing provider) is Matt-ruled and not re-opened; the custody fork and the server-secret containment-mechanism fork are recorded here because the rejected branches are the ones a future reader will reach for first.

• Mechanism A: a real SERVER_ONLY delivery kind + FetchSecrets filter — REJECTED (superseded by Matt’s C1 ruling). The previously folded design: mint SERVER_ONLY across the four delivery representations (migration CHECK widen, store enum, secrets package, a PUBLIC proto enum value) plus a delivery filter in the FetchSecrets handler skipping SERVER_ONLY rows before the append (handler.go:307-310). Rejected because it is default-OPEN minus a subtractive filter: server and user secrets share one table and one resolver manifest, and one edit — a dropped filter clause, a mis-mapped enum arm — re-exposes the master key to every container. It also costs a public proto ENUM widening — a change to an existing wire contract — where C1’s proto delta is two additive SecretsService methods.
• Mechanism C2: separate table + a separate SecretSpec PROFILE — considered, DEFERRED. The same server_secrets table, but the server resolver pinned to its own profile (WithProfile, resolver.go:78) so even the provider keyspace is isolated. Fullest isolation, but migrating the existing PEM/webhook/Linear secrets would require RE-PROVISIONING their provider values under the new profile — a provider-write migration over live deployments. C1’s manifest separation already delivers the container-undeliverable property with a DB-row-only move; the profile split remains available later if provider-keyspace isolation is ever needed. Named residual under C1: the shared profile leaves the provider keyspace a shared mutable surface reachable by the user SetSecret path, so all SEVEN server-secret names’ provider values (the master key plus the six migrated names: primary and reviewer App PEM, the GitHub webhook secret, and the three Linear secrets — client id, client secret, and webhook) are guarded by the F1 membership check (T0, both paths — reject any server_secrets name) RATHER than structurally; the boot read-back verify additionally covers the master key. C2 would make that isolation structural for all of them. Acceptable with the guard in place; the follow-up (per-tenant credential at-rest isolation + gateway-topology exposure) is where C2 is reconsidered (RIG-3237).
• pgcrypto / key-in-DB — REJECTED. pgp_sym_encrypt or a key stored in a Postgres table/GUC puts the key inside the same blast radius as the ciphertext: one pg_dump carries both, reducing the encryption to obfuscation. The red-team requirement is specifically that the key lives OUTSIDE the DB.
• Require an external cloud KMS — REJECTED for the OSS core. A generic self-hoster has no AWS/GCP KMS and must not need one; they have already configured exactly one SecretSpec provider (keyring / 1Password / Vault / env) to run Compass at all. Mandating KMS either forks the code (self-host vs managed paths) or gates self-hosting on a cloud account. The chosen approach subsumes this alternative: the managed plane gets KMS-backed custody by pointing the declared secret at a KMS provider URI (D3) — KMS becomes deployment config, not a code requirement.
• Plaintext now + filed follow-up — REJECTED. Ships a known HIGH hole and creates a migration burden (encrypt-in-place over live rows) that the build-it-encrypted-first ordering avoids entirely. Matt ruled the encryption record lands BEFORE the store is built.

Plan

Global Constraints

• Go server under go/ (gateway itself is TS/Bun, but this record’s code is entirely Go server + store side); existing lint/test discipline applies.
• AES-256-GCM only; nonces from crypto/rand, 96-bit, fresh per encryption, never counter-derived; key is 256-bit from crypto/rand.
• The master key NEVER appears in the DB, in logs, in argv (Set feeds stdin, resolver.go:206-207), or in error strings.
• Auto-provisioning is zero-human-step (rule://no-human-clicks): first boot generates, stores, and declares the key with no operator action.
• The names-only invariant (secrets.go:20-22) is preserved for EVERYTHING except gateway_credentials values; both declared-name registries (secrets AND the new server_secrets) stay names-only — the master key’s declaration row is names-only like any other. server_key_state is not a declared-name registry and holds no value: a salted digest of the master key plus its salt, which discloses nothing about a 256-bit random key — the names-only invariant is untouched by it.
• Every value-bearing type redacts under %s/%v/%#v (D5).
• The master key lives in the separate server_secrets store (D6): its name never appears in the container resolver’s manifest, so it is NEVER materialized into an agent container — a structural property of the split store, not a filter to maintain.
• Every Seal binds AAD = the row’s stable identity (T4); ciphertexts are not portable between rows.

T0 — The server_secrets store: table, resolver split, admin RPC (prerequisite)

Implements D6 (mechanism C1). PREREQUISITE of T2 — the key cannot be declared into a store that does not exist.

• Interfaces:
  • Migration (a NEW migration file, never an edit to 0001): CREATE TABLE server_secrets — names-only, mirroring secrets (0001_init.sql:393-425) MINUS the container-delivery routing: name TEXT PRIMARY KEY (same env-var-name grammar, validated at the store door); declared_by TEXT REFERENCES accounts (id) ON DELETE RESTRICT, NULLABLE — NULL = server-provisioned (the master key; contrast secrets.declared_by, which is NOT NULL REFERENCES accounts (id) ON DELETE RESTRICT, 0001_init.sql:410 — justification below); created_at/updated_at. NO delivery column (server secrets are never container-delivered — that IS the point) and NO kind column (they never reach the T5 materializer).
  • Key-state table (same migration): a single-row server_key_state table (id SMALLINT PRIMARY KEY DEFAULT 1 CHECK (id = 1), key_version SMALLINT NOT NULL DEFAULT 1, key_fingerprint BYTEA NOT NULL, fingerprint_salt BYTEA NOT NULL, updated_at) holds the T2/T5 key-swap tripwire’s non-secret salted digest. It is NOT a declared-name registry, so the names-only invariant (§Global Constraints) is untouched — a salted digest of a key is not that key’s VALUE. Same-migration GRANT SELECT, INSERT, UPDATE ON server_key_state TO compass_app, compass_system (the grant is not inherited — see the server_secrets grant rationale below). It is likewise bucket-A infrastructure (deployment-global, no tenant_id, RLS NOT enabled) and joins server_secrets in the bucketA allow-list (rls_pgtest_test.go:543) so the self-auditing RLS guard test stays a conscious gate.
  • Grants + RLS (mandatory — the new-table integration the shipped multi-tenancy regime, RIG-3106, requires): the same new migration issues GRANT SELECT, INSERT, UPDATE, DELETE ON server_secrets TO compass_app, compass_system. This is NOT inherited: 0001’s grant is a one-time ON ALL TABLES IN SCHEMA snapshot (0001_init.sql:896) and there is no ALTER DEFAULT PRIVILEGES anywhere in the migrations, so a table created by a later migration receives no grant and every server-side resolve would fail permission denied for table server_secrets (fails closed, but breaks the T2 boot path). RLS posture: server_secrets is declared bucket-A infrastructure (Matt-ruled) — NO tenant_id column, RLS NOT enabled: the master key and the PEM/webhook/Linear secrets are deployment-global (the master key decrypts EVERY tenant’s credentials), so there is no tenant to scope by. T0 adds server_secrets to the bucketA allow-list in go/internal/store/rls_pgtest_test.go:543 so the self-auditing RLS guard test stays a CONSCIOUS gate (that test keys off tenant_id, so a bucket-A table is otherwise invisible to it — the allow-list edit is what records the deliberate exemption). Live cross-tenant EXPOSURE (one shared gateway process holding all tenants’ creds) is a gateway-topology property of the parent record (design.md:333-337), out of scope here and tracked as a follow-up (RIG-3237).
  • declared_by nullability, justified: the boot provisioner declares the master key with no human actor. NULL = server-provisioned is honest provenance; attributing the row to the bootstrap-admin account (available at that point in boot, serve.go:358-362) would falsify the audit trail and couple key provisioning to account-bootstrap ordering. Operator-declared rows (the new RPC) always carry the caller’s account id; ON DELETE RESTRICT still protects them.
  • Store: DeclareServerSecret(ctx, actor, name) / DeleteServerSecretDeclaration(ctx, actor, name) mirroring DeclareSecret / DeleteSecretDeclaration (store/secrets.go:82, :160) minus delivery/kind/provider/host (actor nullable-empty for the server-provisioned path). DeclareServerSecret AND its tx-bound DeclareServerSecretTx variant BOTH carry the MIRROR of the store.DeclareSecret membership guard (one shared unexported check both call, so the two can never drift) — it rejects any name STILL present in secrets at the store door, so no writer (the admin RPC, or any future second writer that bypasses it) can create a server_secrets shadow row while the name is live in secrets; the boot reconcile satisfies the guard by deleting the source secrets row FIRST, inside the ONE store.WithTx(store.WithSystemRole(ctx), …) transaction the Row-migration bullet specifies, before its DeclareServerSecretTx insert — so F1 stays held at BOTH doors, not only at the RPC. It ALSO adds the read ServerDeclaredSecrets(ctx) — a thin store view whose DeclaredSecrets(ctx context.Context) ([]store.SecretDeclaration, error) method (the declarations interface shape, resolver.go:29-31) reads server_secrets, mapping rows to store.SecretDeclaration with generic kind / zero delivery (the resolver uses only the name to build its manifest), so NewSpecResolver is reused UNCHANGED.
  • Resolver split: a SECOND SpecResolver instance (the SERVER resolver) constructed over that view — SAME profile (defaultProfile = "default", resolver.go:23) and project (manifestProject = "compass", resolver.go:19), its own manifest state dir. The container resolver’s CONSTRUCTION at serve.go:528 is unchanged — no filter, nothing to filter — but its boot CONSUMERS must be re-pointed: once T0’s reconcile moves the six server-secret names out of secrets, every boot consumer that still resolves those names through the serve.go:528 instance breaks, because that instance’s manifest no longer declares them. THREE failure modes verified in the tree: (a) HARD BOOT FAILURE — validateForgeSecret (serve.go:1539-1549) returns forge secret %q not declared for an absent name and is called on the primary App PEM (:1013) and the App webhook secret (:1016) — both inside buildBoardWebhookWiring under buildForgeReadWiring, propagated to failStartup, so every GitHub-App deployment fails its first post-upgrade boot; (b) SILENT SECURITY DEGRADATION — forgeSecretDeclared (serve.go:1746-1757) returns (false, nil) for an absent name rather than an error, and its caller buildLinearWebhookWiring maps that to a legitimate off-state — if !declared { return nil, nil } (serve.go:1082-1083) — so the Linear webhook ingress silently unmounts, and buildLinearTokenSource (serve.go:1699-1717) drops the Linear write + notify lanes the same way (return nil, nil at :1716); (c) SILENT CAPABILITY LOSS — wireForgeWriteCaller (:587) resolves the forge-write set and gates on forgeWritesEnabled (:1583, over forgeWriteAppsConfigured :270-275): with BOTH PEM names absent from the container resolver’s set the predicate reads both-absent, warnPartialForgeWriteSecrets returns SILENTLY (:1826-1827, havePrimary == haveReviewer), wireForgeWriteCaller returns nil (:1585), hub.SetForgeCaller is NEVER called, and the reviewer-PEM validateForgeSecret at :1638 is UNREACHABLE — so agent forge WRITES fail-closed to CodeUnavailable fleet-wide with no error at boot. (Ordering note: (c) is MASKED by (a) on a FULL no-repoint T0 — forgeWriteAppsConfigured gates havePrimary on App.AppID != 0 (serve.go:272), which is exactly boardIngestionEnabled() (serve.go:224-226), so any writes-enabled deployment hard-fails at the primary-App-PEM validateForgeSecret (serve.go:1013) before reaching wireForgeWriteCaller (:587). (c) is the PARTIAL-repoint mode: it becomes live the moment the read path is repointed and the write path is not — which is precisely why the write lane needs its own positive assertion rather than inheriting the read lane’s.) So T0 has an explicit CONSUMER-REPOINT deliverable: thread the SERVER resolver instance (not the serve.go:528 container instance) into buildForgeReadWiring (serve.go:567), buildLinearWebhookWiring, buildLinearTokenSource, wireForgeWriteCaller (:587) and buildForgeWriteService, and into the newDeclaredSecretResolver / newCachedWebhookSecret closures (serve.go:1025/1039/1085/1644) that mint the App token + verify webhook HMAC. buildLinearWebhookWiring is called from INSIDE buildDoors (serve.go:788), whose single resolver parameter (:677) ALSO feeds buildNetworkServer (:803) → runnerhub.NewMountedHandler (network_door.go:313) — the container FetchSecrets path that MUST keep reading secrets; so buildDoors takes BOTH resolver instances (or receives a pre-built linearWebhookHandler from Serve), because swapping its one resolver argument would repoint the container manifest at server_secrets and deliver every server secret into every agent container, inverting D6. Ordering: the reconcile MUST run BEFORE buildForgeReadWiring (serve.go:567) so the names are in server_secrets before any forge consumer resolves them.
  • RPC: SetServerSecret/DeleteServerSecret on the secrets service — mirrors the SetSecret declare-then-Set flow and its rollback discipline (secrets_service.go:92-145) minus delivery/kind, targeting server_secrets; admin-gated (adminOnly in classifyProcedure, admin_gate.go:47). Carries the reserved-name guard: rejects GATEWAY_CREDENTIALS_MASTER_KEY (or the reserved prefix GATEWAY_CREDENTIALS_, so future master-key-family names are reserved before they are declared) with an actionable error — rotation is OQ-1 machinery, never a raw overwrite. It ALSO carries the MIRROR membership guard: it rejects any name already present in secrets (checked before the server_secrets insert and before resolver.Set) with an actionable error naming the conflicting user declaration — the F1 invariant is order-free (no name is ever live in both tables) and must hold in BOTH declaration orders, so the secrets→server_secrets direction is guarded here just as the server_secrets→secrets direction is guarded on the user path below.
  • User-path server-secret guard (F1 — mandatory, NOT admin-RPC-only): C1 shares the provider keyspace (§D2 read-back verify), so the authenticatedOpen user secretsService.SetSecret/DeleteSecret path (any authenticated account, admin_gate.go:122-125) can overwrite a server secret’s provider value AND — because after OQ-4 the six PEM/webhook/Linear names live in server_secrets, not secrets — create a FRESH shadow row in secrets under that name (no primary-key conflict), which the inject-all path then delivers into every container. The reserved-name check alone is INSUFFICIENT: those six names are arbitrary per-deployment config (main.go:414-450) and carry no reserved prefix. So the guard is a MEMBERSHIP test, not a name/prefix match: SetSecret/DeleteSecret (and DeclareSecret at the store door, so the shadow row can never be created at all) MUST reject any name present in server_secrets — checked BEFORE resolver.Set/Delete (secrets_service.go:124/208) — in addition to the reserved master-key name GATEWAY_CREDENTIALS_MASTER_KEY and prefix GATEWAY_CREDENTIALS_. A T0 deliverable on the existing user path, not only the new admin RPC.
  • Row migration — a BOOT-TIME config-driven reconcile, NOT an in-migration DML step (F2): the server-only secret NAMES are per-deployment config the embedded SQL migration (//go:embed migrations/*.sql, store.go:24-25) cannot know. So the move runs at boot in the server wiring (where the config IS available): for each configured server-secret NAME, if a row exists in secrets, move it — inside ONE store.WithTx(store.WithSystemRole(ctx), …) (coordination.go:368; armTx arms the BYPASSRLS system role for the whole tx, tenant_tx.go:199-215), DELETE the secrets row FIRST, then INSERT the server_secrets row, so the store-door membership guard sees the delete already applied and admits the insert. Both statements run through tx-bound DeleteSecretDeclarationTx/DeclareServerSecretTx(ctx, tx, …) variants (the shipped …Tx convention, coordination.go:86/168, dm.go:49/97), with DeclareServerSecretTx carrying the SAME store-door membership guard as its plain DeclareServerSecret counterpart via the shared check (the delete side carries no membership guard — the reconcile heal arm below depends on being able to delete a secrets row whose name is already live in server_secrets), so the DELETE-FIRST ordering is what admits the insert — NOT the plain-signature methods, each of which executes through scopedDBTX as its OWN implicit transaction (one pool.SendBatch per statement, tenant_tx.go:64-70, :80-92), which would make the two operations two transactions and break the atomicity below (atomic: one transaction commits or rolls back both, so no crash window strands the declaration in neither table) — INCLUDING when a server_secrets row for that name already exists, in which case the pass deletes the secrets shadow row and logs the reconciled collision. Leaving a both-tables name in place would strand it on the inject-all path forever, so the reconcile heals the collision rather than skipping it. The heal is NOT limited to the configured names: after moving the configured set, the pass also scans server_secrets for any name that STILL has a live secrets row (an operator-declared server secret raced into both tables, which carries no reserved prefix and is not in cfg.Forge.resolved()) and deletes that shadow row too, so every both-tables state self-heals regardless of provenance. Idempotent, serialized under the SAME pg_advisory_xact_lock constant key T2 uses, acquired with the SAME bounded discipline (pg_try_advisory_xact_lock in a bounded retry loop, or a session lock_timeout) so a contended boot fails closed with a diagnosable error naming the contended lock rather than parking — in its OWN transaction, taken and released before the T2 provisioning transaction, so the two never overlap; the reconcile’s only ordering constraint is that it precede buildForgeReadWiring (serve.go:567). No provider write (the shared profile makes it a pure row move). The name source is the RESOLVED config cfg.Forge.resolved() (serve.go:232-246) — the SAME accessor every live consumer uses (buildLinearTokenSource :1700, buildBoardWebhookWiring :1002, buildForgeWriteService :1624), NOT the raw flag/env layer: two of the names carry CODE DEFAULTS applied after that layer (defaultForgeLinearClientIDSecretName = "LINEAR_FORGE_CLIENT_ID" / defaultForgeLinearClientSecretName = "LINEAR_FORGE_CLIENT_SECRET", serve.go:202-203), so a reconcile reading the raw main.go flag/env layer would see "" for a default-named Linear pair and SILENTLY SKIP those two rows, leaving them on the inject-all path forever — the exact exposure the reconcile exists to close. The complete set is SIX names, not three: the PRIMARY App PEM (appKeySecret, main.go:414-417), the webhook secret (appWebhook, main.go:418-421), the REVIEWER App PEM (reviewerAppKeySecret, main.go:430; consumed serve.go:1644), and the THREE Linear secrets (client id, client secret, webhook — resolved names per cfg.Forge.resolved(), provenance main.go:435-450). Closes the pre-existing inject-all exposure (D6) for every configured name the reconcile actually runs for. Same PR chain: OQ-4, RESOLVED.
  • Reconcile execution context (F2 follow-on — the relocation’s RLS consequence): the SOURCE table secrets is FORCE ROW LEVEL SECURITY (it is in the tenant_tables array, 0001_init.sql:912; the DO loop at :922-923 issues ENABLE/FORCE ROW LEVEL SECURITY). The old in-migration approach ran on the raw owner pool BEFORE any policy existed (store.go:89-90, “migrate() … runs on the raw pool as the owner, before any policy exists to fight”); the boot-time reconcile runs AFTER migration, and if it took the ordinary tenant-scoped store path (SET LOCAL ROLE compass_app + set_config('compass.tenant_id', …), tenant_tx.go:140-141) — a non-owner, non-BYPASSRLS role — it would be confined to the bootstrap tenant’s secrets rows. secrets.name is a GLOBAL primary key (0001_init.sql:396, name TEXT PRIMARY KEY; tenant_id is a plain non-key column at :413), so there is at most ONE row per configured server-secret name in the whole deployment — but that row may have been declared under a NON-bootstrap tenant, in which case a compass_app-scoped reconcile cannot see it and silently skips it, leaving it on the inject-all path forever. So the reconcile MUST run under store.WithSystemRole(ctx) (tenant_tx.go:41-50), which arms SET LOCAL ROLE compass_system with no tenant GUC (armQueue, tenant_tx.go:136-138; armTx, :199-215, shared by beginTenantTx, :185-195; compass_system is NOLOGIN BYPASSRLS, 0001_init.sql:879, idempotently re-asserted :883), seeing every tenant’s rows in one pass. This adds a cross-tenant, request-path-free boot step to the BYPASSRLS surface DL-315 governs. DL-315 names four cross-tenant loops (delivery-cursor sweep, deliver-ack advance, reattach recovery, lag-resync); the shipped tree arms the system role at THREE call sites — go/internal/delivery/consumer.go (Consumer.Run, :316; covering the sweep/resync/recovery arms since the loop threads one ctx and never re-roots it, :315), go/internal/runnerhub/hub.go (deliver-ack advance, :753), and go/internal/runnerhub/hub.go (forgeNotificationAck, :814 — an arm DL-315’s four names do NOT list — though WithSystemRole’s own doc comment already names it (tenant_tx.go:41-47), so THIS staleness is in DL-315’s ledger prose, not in the code’s documentation — but note T0 itself then widens that same code doc’s ONLY-claim (tenant_tx.go:36-38, :43-45) to admit the boot reconcile as a fifth site, so both surfaces are corrected in this PR chain, scheduled as a T0 deliverable above). Symbol names anchor these; the line numbers are a convenience that drifts. DL-315’s load-bearing “granted ONLY to those named background workers and NEVER on the request path” clause (DECISIONS.md:113) is UNCHANGED — this is a boot step, never the request path — but its named entrypoint set is widened, recorded (OQ-5, Matt-ruled) as its own ledger row DL-326 that Refines DL-315. (Note: a raw-owner-pool reconcile would move every tenant’s rows only if the owner connection is a superuser — which the tree ASSUMES but does not enforce: 0001_init.sql:848-851 states the property that a superuser owner bypasses even FORCE, and rls_pgtest_test.go:281-284 records the same assumption in a comment (“the pgtest harness (like production) connects as a SUPERUSER owner”), which is why that test deliberately does not try to prove FORCE. An assumption stated in two comments is not an enforced invariant; the reconcile takes the explicit BYPASSRLS path rather than inheriting an owner-privilege escape.)
• Consumes: nothing from T1-T5 (pure prerequisite).
• Tests: a secret declared in server_secrets is ABSENT from a FetchSecrets response (because it lives in the other table — no filter involved) while secrets rows in the same boot ARE present; the server resolver instance RESOLVES it; the container resolver’s generated manifest never contains its name; the new RPC rejects a non-admin caller; the reserved master-key name is rejected on SetServerSecret/DeleteServerSecret AND on the user-path SetSecret/DeleteSecret (a non-admin authenticated user calling SetSecret with the reserved name is rejected and the provider value is unchanged — F1); the server resolver can READ server_secrets through the normal compass_app store path (the GRANT is present — F3); all six migrated names (primary + reviewer App PEM, webhook, and the three Linear secrets) resolve through the server resolver and are gone from FetchSecrets; and a configured server-secret row declared under a NON-bootstrap tenant is found and moved by one reconcile pass and is absent from FetchSecrets afterward (red without WithSystemRole: the compass_app-scoped reconcile sees zero rows and skips it silently). A deployment with the Linear pair declared under the DEFAULT names (LINEAR_FORGE_CLIENT_ID / LINEAR_FORGE_CLIENT_SECRET) and NO flag/env set is likewise found and moved by one reconcile pass and is absent from FetchSecrets afterward (red if the reconcile reads the raw flag/env layer instead of resolved()). CONSUMER-REPOINT positive assertions (the (b)/(c) silent modes need them, since an absent name is indistinguishable from a legitimate off-state): boot a server with all six names configured and, after one reconcile pass, assert the GitHub App token source mints, the GitHub and Linear webhook handlers are MOUNTED (non-nil), the Linear token source is non-nil, AND the forge WRITE caller is MOUNTED (hub.SetForgeCaller called / RelayForgeCall does not fail-close to CodeUnavailable) — i.e. the forge read AND write lanes still wire off the server resolver, not just that the names left secrets. MEMBERSHIP-GUARD assertion (F1): a non-admin authenticated caller invoking SetSecret with a configured server-secret NAME (not only the reserved master-key name) is rejected, no secrets shadow row is created, and the provider value is unchanged. MIRROR-GUARD assertion (F1, admin path): an admin calling SetServerSecret with a name already present in secrets is rejected before the server_secrets insert and before resolver.Set, no server_secrets row is created, and the provider value is unchanged. RECONCILE-HEAL assertion: a name live in BOTH tables at boot is healed by one reconcile pass — the secrets shadow row is deleted, the collision is logged, and the name is absent from FetchSecrets afterward (red if the reconcile skips a name that already has a server_secrets row). ORDERING assertion: a reconcile that INSERTs the server_secrets row before deleting the secrets row reds against the store-door membership guard, and both statements commit or roll back together — a crash between them leaves the name in secrets, never stranded in neither table (red if the DELETE and INSERT run as two transactions rather than one store.WithTx).

T1 — Envelope-crypto helper package

New package go/internal/secrets/envelope (child of the secrets seam it serves; no import cycle — it depends on nothing in secrets).

• Interfaces:
  • type Key struct{ /* unexported [32]byte */ } — redacts under %s/%v/%#v (String()/GoString() → "envelope.Key{<redacted>}").
  • func NewKey() (Key, error) — 32 bytes from crypto/rand.
  • func KeyFromBytes(b []byte) (Key, error) — errors unless len == 32.
  • func (k Key) Seal(plaintext, aad []byte) (ciphertext, nonce []byte, err error) — AES-256-GCM, fresh 96-bit random nonce per call; aad is authenticated, not encrypted (binds the ciphertext to its row identity — T4).
  • func (k Key) Open(ciphertext, nonce, aad []byte) ([]byte, error) — GCM auth failure (tamper OR aad mismatch) returns an error naming no plaintext/key material.
  • Key encoding for provider storage: base64(std) of the 32 raw bytes (SecretSpec values are strings; Set rejects empty, resolver.go:225-227).
• Consumes: crypto/aes, crypto/cipher, crypto/rand only.
• Tests: round-trip; tamper (flip a ciphertext/nonce byte → error); Open under a different aad → error; nonce uniqueness across calls; redaction of Key under all three verbs; KeyFromBytes length validation.

T2 — Master-key boot-provision seam

Boot-time resolve-or-provision, in the server wiring next to the existing declared-secret consumers (go/server/serve.go). DEPENDS ON T0 (the server_secrets store and its resolver instance must exist before the key is declared into it) and T1.

• Interfaces:
  • func provisionGatewayMasterKey(ctx context.Context, resolver secrets.Resolver, st *store.Store) (envelope.Key, error) — resolver is the SERVER-SECRET resolver instance (T0). Resolve GATEWAY_CREDENTIALS_MASTER_KEY through it; on absence: envelope.NewKey() → resolver.Set(ctx, name, encodedKey) (resolver.go:219; the value rides stdin, never argv, resolver.go:206-207) → st.DeclareServerSecret(ctx, "", name) with declared_by = NULL (server-provisioned; T0’s nullable FK). No delivery, no kind — those columns do not exist on server_secrets.
  • Concurrency — advisory-lock serialized (mandatory): the whole resolve→generate→Set→Declare sequence runs under a Postgres advisory lock (pg_advisory_xact_lock on a constant key) — Postgres is the one store all instances share. This replaces the draft’s Set-then-Declare with tolerated ErrConflict, which was a check-then-set race: two concurrently booting instances both resolve-absent and both Set (last writer wins in the provider); the loser’s Declare hits ErrConflict, is tolerated, and that instance proceeds to encrypt with a key the provider no longer holds — silently undecryptable rows, discovered at read time. The write path this builds on explicitly disclaims concurrent safety: “The declare/set/rollback trio is not atomic and assumes no concurrent same-name writer (the single-Runner MVP: SetSecret is user-driven CLI)” (go/server/secrets_service.go:88-91). Ordering inside the lock stays Set-before-Declare: the inverse leaves a crash-window orphan declaration, and “an orphaned declaration is required=true in the resolve manifest and would poison EVERY live session’s FetchSecrets” (secrets_service.go:86-88) — under C1 the blast radius shifts but stays severe: an orphaned server_secrets declaration is required=true in the SERVER resolver’s manifest and would fail every server-side resolve (master key, PEM, webhook, Linear). A crash between Set and Declare converges on the next boot (the undeclared name does not resolve, so the provisioner re-generates, re-Sets, and Declares — no row was ever encrypted under the orphaned value).
  • Bounded critical section (mandatory — a stuck provider must not wedge the fleet): the provider round-trips inside the lock inherit only the caller’s ctx, which at boot is long-lived — but the two halves bound DIFFERENTLY. SpecResolver.Set IS ctx-bounded: it shells out via exec.CommandContext(ctx, r.cli, …) (resolver.go:233), so a ctx deadline genuinely kills it. SpecResolver.Resolve is NOT: it threads ctx only into DeclaredSecrets (resolver.go:136); the actual provider round-trip is b.Load() (resolver.go:165), whose SDK signature carries NO ctx (func (b *Builder) Load() (*Resolved, error), secretspec-go v0.15.0 secretspec.go:245) and which blocks in an uncancellable FFI call (nativeResolve → C.secretspec_resolve, binding_cgo.go:30 / binding_purego.go:118). A hung provider (1Password awaiting biometric approval, an unreachable Vault, a half-open TCP) would otherwise hold the transaction-scoped lock indefinitely, and because the key is a shared constant EVERY other booting instance blocks on it — one stuck provider becomes a fleet-wide boot wedge. So the provisioner (1) derives a ctx with an explicit timeout (mirror the 30s &http.Client{Timeout: 30 * time.Second} precedent that already bounds the Linear boot mint at serve.go:1731) and, because the Resolve-side call cannot be cancelled, RUNS THE ctx-LESS Resolve ON ITS OWN GOROUTINE AND SELECTS ON THAT CTX (Set is ctx-bounded and stays on the parent, inside the lock) — so the provisioner returns a diagnosable bounded startup error (naming the hung provider) and its transaction is rolled back, RELEASING the xact-scoped advisory lock, while the orphaned FFI goroutine is knowingly leaked for the process’s remaining boot-failing lifetime (acceptable: the boot is aborting anyway). The PARENT goroutine owns the transaction (pgx.Tx is not concurrency-safe): it takes the advisory lock, offloads ONLY the ctx-less provider READ (Resolve) and NEVER a provider write, and on the timeout branch performs the Rollback itself and discards the buffered result without acting on it. A Set reached after the parent’s Rollback would land OUTSIDE the released advisory lock and could overwrite a key another booter has already provisioned and begun sealing rows under — reintroducing the silently-undecryptable-rows failure the lock exists to prevent — so the offloaded goroutine performs no Set; the parent runs the (ctx-bounded) Set itself, in-lock, only on the success branch. The offloaded call reports through a BUFFERED (cap-1) channel so the orphaned FFI goroutine can complete its send and exit rather than blocking forever on an abandoned receiver (bounding the leak on a crash-looping boot). And (2) acquires the lock with pg_try_advisory_xact_lock in a bounded retry loop (or sets a session lock_timeout) so a booter that cannot get the lock fails closed with a diagnosable startup error naming the contended provisioning lock rather than parking forever. Test (T2): a provider that never returns yields a bounded, diagnosable boot failure whose transaction (and advisory lock) is released even though the provider call itself cannot be cancelled, and a second instance blocked on the lock also fails bounded rather than hanging.
  • Read-back verify, every boot: after provisioning AND on every subsequent boot, re-resolve the name and byte-compare against the key the process is about to encrypt with; on mismatch, refuse to serve gateway-credential writes (fail closed). This re-resolve is the SAME uncancellable Load (resolver.go:165) and uses the SAME bounded-offload path as the provisioning resolve (timeout ctx + own goroutine + buffered cap-1 channel), so on a steady-state boot — key already provisioned, nothing to serialize — a hung provider still yields a bounded, diagnosable startup error rather than a parked process. This is necessary because the provider keyspace is a shared mutable surface: C1 pins BOTH resolver instances to the same SecretSpec project + profile (manifestProject = "compass", defaultProfile = "default", resolver.go:19/23), so the master key’s provider VALUE is reachable by any writer of that keyspace — an operator’s out-of-band secretspec set, AND (absent the guard below) the user-driven SetSecret RPC. The verify is a tripwire, not a boundary: on a plain boot the resolved value IS “the key the process is about to encrypt with”, so a byte-compare cannot by itself distinguish “my key” from “a swapped key”. T5 strengthens it by binding a non-secret key fingerprint (a salted digest, persisted in the T0 server_key_state row — key_fingerprint/fingerprint_salt, not a declared-name registry so the names-only invariant is untouched) so a swapped provider value is DETECTED at boot rather than silently adopted.
  • Reserved-name guard (F1 — on EVERY provider-writing path): C1 splits the DECLARATION registries (two tables) but NOT the provider keyspace (one shared profile, above). So the isolation C1 buys is that server secrets are structurally undeliverable to CONTAINERS (the manifest separation, §Mechanism C1) — it does NOT make the master key’s provider value unreachable from the user path. The user SetSecret/DeleteSecret RPC (authenticatedOpen, any authenticated account — admin_gate.go:122-125) declares into secrets but then calls resolver.Set, which shells secretspec set <NAME> --profile default (resolver.go:258-267) against the SHARED keyspace — so a user calling SetSecret with name GATEWAY_CREDENTIALS_MASTER_KEY would OVERWRITE the master key’s provider value (the running process keeps its cached key, but the next boot adopts the attacker-chosen key → every existing row fails GCM auth, every new row is sealed under a known key: the exact bulk-disclosure D1 prevents). Therefore the reserved-name guard is MANDATORY on the user secretsService.SetSecret/DeleteSecret path (reject the reserved name GATEWAY_CREDENTIALS_MASTER_KEY and prefix GATEWAY_CREDENTIALS_ BEFORE resolver.Set/Delete), a T0/T2 deliverable, AND on the new SetServerSecret/DeleteServerSecret RPC. Both are tested red-green (a non-admin user calling SetSecret with the reserved name is rejected and the provider value is unchanged). Rotation is OQ-1’s machinery, never a raw overwrite through either surface. The name-keyed global user delete (“a row is keyed by name alone, not (actor, name)”, go/internal/store/secrets.go:150-153) is why the guard covers the delete path too, once a real provider hard-delete lands.
  • Nil-resolver deployment: a server built with no secrets surface is legitimate today (“resolver may be nil on a server built with no secrets surface (FetchSecrets then fails CodeFailedPrecondition rather than panicking)”, go/internal/runnerhub/handler.go:52-54; agents still start, go/internal/runner/host.go:367-370). On such a server the gateway credential store is NOT constructed; enabling the gateway is a configuration error naming the missing secrets surface; a no-gateway boot proceeds unchanged.
  • Resolve path mirrors newDeclaredSecretResolver (serve.go:1469) but is pointed at the SERVER-SECRET resolver instance and invoked once at boot; the decoded envelope.Key is held in memory for the process lifetime.
  • Boot fails closed: a resolve/provision fault is a startup error, never a fall-back-to-plaintext.
• Consumes: T0’s server_secrets store + server resolver instance, T1 envelope, secrets.Resolver, store.DeclareServerSecret.
• Produces: the process-lifetime envelope.Key handed to T4.
• Tests: fresh-boot provisions (Set + Declare called, key usable); second-boot resolves without Set; two-writer interleaving (concurrent provisioners converge on ONE key both read back identically); Set-succeeded/Declare-crashed reboot converges; read-back mismatch → fail closed; reserved-name SetServerSecret/DeleteServerSecret → actionable reject; nil-resolver + gateway enabled → configuration error naming the missing surface; nil-resolver without gateway → boot proceeds unchanged; provider fault → boot error.

T3 — Schema: ciphertext columns on gateway_credentials

The gateway_credentials migration (owned by the store build the frozen record plans) carries, for the value payload:

• Interfaces: columns value_ciphertext BYTEA NOT NULL, value_nonce BYTEA NOT NULL, key_version SMALLINT NOT NULL DEFAULT 1. NO plaintext value column exists in any migration version — the store is born encrypted (per the rejected plaintext-first alternative).
• The row’s non-secret metadata (provider, scope, owner_user_id, version CAS counter, expiry timestamp if queried-on) stays plaintext-queryable; ONLY the credential value payload is inside the envelope. Expiry inside vs. beside the ciphertext is settled by the store build; default: beside (the gateway lists refreshable rows by expiry without decrypting).
• Dependency edge, explicit: this migration task DEPENDS ON T1+T2 merged (same PR chain or a stated blocking dependency in the store build’s tracker) — the encrypt-before-any-row ordering must be structural, not a prose sentence. gateway_credentials exists nowhere under go/ today (grep this session, zero matches), so the ordering is currently satisfiable with no existing-rows hazard. T5’s ciphertext-at-rest assertion is the named CI tripwire that reds if a plaintext value column ever appears.
• Tests: migration applies; NOT NULL enforced.

T4 — Wire crypto into the CredentialStore read/write path

Every write path (initial credential save, gateway OAuth-refresh write-back per design.md:367-371) seals before INSERT/UPDATE; every read path opens after SELECT. This sits server-side under the RPC surface the frozen record recommends (design.md:348-358), so the TS gateway never sees the key or the crypto — it receives plaintext credentials over the stack-token-authenticated RPC exactly as the frozen record already specifies.

• Interfaces: the store’s credential accessors take/return the decrypted payload type (per D5: secret fields unexported with accessors, or slog.LogValuer + redacting MarshalJSON alongside String/GoString); the gateway-bound RPC response is built by an explicit proto/DTO conversion, never by marshaling the payload type. The envelope.Key is a construction-time dependency of the store/service wrapper, not a per-call parameter. AAD binds row identity: every Seal/Open passes aad = row primary key + key_version — the STABLE row identity, never the CAS version counter (which increments per write). Without AAD an attacker with DB write access can swap two rows’ ciphertext+nonce pairs and both still authenticate — a cross-tenant credential substitution crossing exactly the boundary the frozen record names (“the isolation boundary is the per-tenant pool scoping enforced server-side”, design.md:333-337). Writes stamp the current key_version; reads select the key by the row’s key_version (v1: single live key — a mismatched version is a diagnosable error naming the expected/found version numbers, never key material; see OQ-1).
• Consumes: T1, T2’s key, T3’s columns.
• Tests: see T5.

T5 — Tests: ciphertext-at-rest + redaction assertions

• Interfaces: (test-only)
  • Ciphertext-at-rest assertion (the named CI tripwire, T3): write a credential through the store, then read the raw row via SQL — assert the known plaintext substring (e.g. the api key literal) appears NOWHERE in any column; assert value_ciphertext != plaintext and decrypt-with-key round-trips. This is the guard that reds if a plaintext value column ever appears.
  • Row-swap assertion (AAD): swap two rows’ ciphertext+nonce pairs at the SQL level — Open MUST fail for both rows (the AAD binds row identity).
  • Redaction assertion (beyond fmt verbs): format the decrypted-payload type and envelope.Key under %s, %v, %+v, %#v; json.Marshal the payload type; log it through a slog.JSONHandler capture — assert no secret bytes appear in ANY of these. The fmt verbs alone mirror the existing pattern’s intent (secrets.go:155-156) but do not cover the reflection paths (D5).
  • Key-swap tripwire (F1): persist a non-secret key fingerprint (a salted digest of the master key, written to the T0 server_key_state row — key_fingerprint + fingerprint_salt, alongside key_version) at provision time; the boot read-back verify recomputes it and refuses to serve on mismatch. Test: swap the provider value out-of-band, reboot → boot fails closed (the fingerprint distinguishes “a key” from “my key”, which a bare byte-compare cannot).
  • Refresh write-back path: refreshed OAuth tokens land re-sealed with a fresh nonce (nonce differs from the previous row state).
  • Wrong-key / tampered-row read → error, no partial plaintext; a key_version-mismatch error is distinguishable from a GCM auth failure (names the versions, never key material).

Tasks

• T0 — server_secrets store (mechanism C1): new migration (table + single-row server_key_state table for the key-swap tripwire digest + GRANT SELECT/INSERT/UPDATE/DELETE on server_secrets and SELECT/INSERT/UPDATE on server_key_state to compass_app/compass_system + bucketA allow-list edit in rls_pgtest_test.go), ServerDeclaredSecrets store view + second SpecResolver instance (same profile), admin-gated SetServerSecret/DeleteServerSecret RPC carrying the MIRROR membership guard (reject any name already present in secrets, checked before the server_secrets insert and before resolver.Set, AND at the store.DeclareServerSecret / DeclareServerSecretTx store door (both variants, one shared check) so no writer can create a server_secrets shadow row while the name is live in secrets — the boot reconcile satisfies it by deleting the source secrets row first inside its single store.WithTx(store.WithSystemRole(ctx), …) transaction) AND the matching user-path half on secretsService.SetSecret/DeleteSecret and store.DeclareSecret — together the SYMMETRIC F1 MEMBERSHIP test, each direction rejecting any name present in the OTHER table (the six configured names carry no reserved prefix, so a name/prefix match is insufficient) PLUS the reserved GATEWAY_CREDENTIALS_MASTER_KEY name and GATEWAY_CREDENTIALS_ prefix, both checked before resolver.Set/Delete, and a BOOT-TIME config-driven reconcile (NOT in-migration DML) moving all SIX configured server-secret names (primary + reviewer App PEM, webhook, three Linear) from secrets to server_secrets via the tx-bound DeleteSecretDeclarationTx/DeclareServerSecretTx pair (both NEW — no …Tx sibling of DeleteSecretDeclaration exists today, store/secrets.go:160), run under store.WithSystemRole (BYPASSRLS — the compass_app-scoped path is confined to the bootstrap tenant, so a row declared under another tenant is otherwise silently skipped) — PLUS the doc-comment repairs T0’s own changes require: store.WithSystemRole’s ONLY-claim (tenant_tx.go:36-38, :43-45) widens from the four background-loop entrypoints to ALSO cover this boot server-secret reconcile (a FIFTH, non-loop site; per DL-326); the five “three/3 procedures authenticatedOpen” SecretsService comments (admin_gate.go:116-121, network_door.go:288-290, secrets_service.go:6, serve.go:555, serve.go:749-750) are restated as three authenticatedOpen + two adminOnly once the two admin methods land; the adminOnly type doc (admin_gate.go:22-26) widens its “privileged CompassService agent-session RPCs (and token issuance)” enumeration to admit the two SecretsService server-secret writes; and the SecretsService service comment (proto/compass/v1/compass.proto:175-179) is restated to cover the two admin-gated server-secret methods and to note their authz is door-side (adminOnly in classifyProcedure), not handler-side — this one propagates through moon run compass-proto:gen into two checked-in TS gen trees — packages/compass-client/src/gen (the PUBLIC lane, buf.gen.yaml) and packages/compass-agent/src/gen (the internal-only agent lane, buf.gen.agent-ts.yaml, proto/moon.yml:35-41) — which the drift gate does NOT catch as staleness on its own (it regenerates and diffs, so stale comment text regenerates identically and passes). (Opportunistic in the same edit, flagged pre-existing not T0-caused: admin_gate.go:40-41’s “every generated CompassService and CommsService procedure” phrasing is already stale — the switch covers SecretsService too — as is the identical phrasing at classify_exhaustive_test.go:45-46; correct both in the same pass, but neither gates T0.) Red-green tests: absent-from-FetchSecrets + server-resolver-resolves + user-path-guard + admin-path-mirror-guard (a SetServerSecret on a name already in secrets is rejected) + reconcile-heals-both-tables-collision (one pass deletes the secrets shadow row and the name leaves FetchSecrets) + reconcile-orders-delete-before-insert (a reconcile that INSERTs the server_secrets row before deleting the secrets row reds against the store-door membership guard) + non-bootstrap-tenant-reconcile. PREREQUISITE of T2. Proto delta: two additive SecretsService methods, no enum change — the checked-in public gen trees are drift-gated, so this needs the compass.proto edit + moon run compass-proto:gen, and both new procedure paths MUST be added to classifyProcedure as adminOnly or classify_exhaustive_test reds CI (go/internal/auth/classify_exhaustive_test.go).
• T1 — go/internal/secrets/envelope: Key/NewKey/KeyFromBytes/Seal/Open (AAD-carrying) + redaction + unit tests
• T2 — boot resolve-or-provision seam via the server-secret resolver (advisory-lock serialized, read-back verify + key-fingerprint tripwire, nil-resolver gating, fail-closed boot; the F1 guards (membership + reserved-name) are T0’s, on BOTH the admin SetServerSecret/DeleteServerSecret RPC AND the authenticatedOpen user SetSecret/DeleteSecret path). DEPENDS ON T0 + T1.
• T3 — gateway_credentials migration columns: value_ciphertext, value_nonce, key_version; no plaintext column ever. DEPENDS ON T1+T2 merged (the store build’s migration blocks on them).
• T4 — seal/open wiring in the CredentialStore read/write + refresh write-back paths; AAD = row PK + key_version; key as construction-time dependency
• T5 — ciphertext-at-rest assertion (CI tripwire), row-swap AAD test, redaction incl. json.Marshal + slog JSONHandler, tamper/wrong-key tests, and the key-swap tripwire (persist a salted key fingerprint at provision; boot recompute + fail-closed on mismatch).

Open Questions

• OQ-1 (deferrable, recommendation: defer): master-key rotation + row re-encryption. How does the master key rotate and how are existing rows re-encrypted? Recommendation: the key_version column lands at v1 (D4 — load-bearing now because retrofitting it is a migration under ambiguity), but active rotation machinery — lazy re-encrypt on next write plus a one-shot re-encrypt sweep — is DEFERRABLE to a follow-up record; v1 runs a single live key version and treats a version mismatch as an error. Why deferral is SAFE, not merely convenient: (1) provider key loss makes every row unrecoverable, but the blast radius is BOUNDED — gateway credentials are re-obtainable from their owners (re-enter the API key, re-run the OAuth grant; the store holds api_key and OAuth-shaped payloads a user configures, parent design.md:324-326), so the worst case is a fleet re-authentication event, never permanent data loss; (2) rotation is also the nonce-budget release valve, and D1’s write-rate arithmetic shows years of headroom under the 2^32 bound. Carried into T4/T5 regardless of deferral: a wrong-key / key_version-mismatch decrypt error MUST be operator-distinguishable from row corruption — the error names the expected/found key_version numbers, never key material.
• OQ-2 — RESOLVED (Matt; mechanism updated by the C1 ruling): master-key containment. The master key is declared into the physically separate server_secrets store that D6/T0 build, boot-resolved through the server-secret resolver instance (mirroring the Forge App PEM’s serve.go:1469 pattern), and never materialized into an agent container — its name is not in the container resolver’s manifest at all. An earlier fold resolved this with a minted SERVER_ONLY delivery kind + FetchSecrets filter; Matt’s C1 ruling replaced that with the separate store (see Alternatives, mechanism A) and REMOVED the public proto ENUM change it required (the C1 admin RPC is an additive service-method delta, not a wire enum change).
• OQ-3 — RESOLVED (Matt: yes, D7): api_key rows are encrypted identically to OAuth rows. One seal/open code path for both payload shapes; both are secrets of the same sensitivity (a stored api_key is as long-lived and as disclosure-critical as an OAuth refresh token), and a split path would leave the most static secret class in plaintext for zero benefit. T3’s single-column schema and T4’s single wiring path encode this — there is no plaintext branch.
• OQ-4 — RESOLVED (Matt: same PR chain, mechanism C1): the PEM/webhook/Linear server-secret rows move to server_secrets in THIS record’s PR chain, closing the live pre-existing exposure (they ride the inject-all path into every agent container today, D6). NOTE: C1 makes the move cheap — because both resolver instances share one SecretSpec profile (defaultProfile, resolver.go:23), this is a DB-row move only: provider values stay put and nothing is re-provisioned. New operator declarations go through the admin-gated SetServerSecret RPC (T0); the move for the ALREADY-declared names is T0’s boot-time config-driven reconcile (per configured NAME, not an in-migration DML — the names are per-deployment config the embedded migration cannot know).
• OQ-5 — RESOLVED (Matt: a new ledger row): how to RECORD the DL-315 BYPASSRLS-allow-list widening. T0’s boot server-secret reconcile adds a cross-tenant, request-path-free BYPASSRLS entrypoint (store.WithSystemRole) beyond the set DL-315 governs — and the review found DL-315’s own enumeration is already stale (the shipped hub.go:814 forge-notification-ack arm is a BYPASSRLS site DL-315’s four names omit). DL-315’s load-bearing NEVER-on-the-request-path clause is untouched; only its named-entrypoint set widens. Matt ruled: its OWN ledger row that Refines DL-315 (matching the ledger’s dense precedent for scope-amendment rows — DL-316/DL-317/DL-293/ DL-072/DL-212 — and fixing the discoverability gap: an engineer auditing “which sites may arm BYPASSRLS?” reads DL-315, not this at-rest-encryption record). Landed as DL-326 in this PR (this PR’s encryption row is DL-325; the sibling SubjectService PR takes DL-327), restating the true shipped entrypoint set (three call sites incl. the forge-notification-ack arm) + the reconcile; DL-315 stays Active.