Skip to content

Firecall v2 one-shot implementation plan

2026-09-03 Cloudflare ownership amendment: The platform owner pre-creates the dedicated tunnel and three Access applications in the shared account. Their sessions last six hours. Terraform consumes the tunnel ID and three audiences. It never creates, imports, updates, or deletes those four objects. The owner stores the connector token directly in Key Vault. Resource-scoped Cloudflare permissions select only Firecall resources. This amendment supersedes later text that specifies account-wide tunnel or Access mutation, one-hour application sessions, Terraform-created applications, or Terraform retrieval of the connector token.

2026-09-03 development identity simplification: The development apply identity (ARM_CLIENT_ID_APPLY) runs the development infrastructure workflows. Five user-assigned managed identities in rg-firecall-control-dev-eastus, federated to their AKS ServiceAccounts, are the only Firecall identities. No Entra application exists for developers or workflows, no sp-firecall-trust application is seeded, and development activation does not use the firecall-live environment. .github/workflows/firecall-dev-terraform.yml plans and applies the control, AKS pool, and Cloudflare edge roots with saved plans bound to the plan run, stack, commit, lock digest, and plan digest. .github/workflows/firecall-dev-images.yml builds, scans, publishes, and signs the runtime containers. The owner stores the connector token, the API token, the audiences, the HMAC secrets, and the signing keys in the Key Vault after the control root creates it. Approval gates stay for staging and production. This amendment supersedes the trust-root seed steps, the Entra flexible-credential activation for development, and any text that requires firecall-live before development activation.

Date: 2026-08-24

2026-09-10 development VM build: firecall-dev-vm.yml prepares a credentialless, run-bound artifact with six VM programs and fresh scanner databases. Its second job verifies the artifact before using the existing development deployment identity. The pinned Packer path retains final offline scanning, closed executable checks, qualification, and cleanup. Only the development scanner contract also accepts this exact workflow identity; the existing privileged scanner consumers keep their original workflow allowlist. No production gate or session profile is enabled by this workflow. See the workspace runbook for live evidence requirements.

Status: Accepted for implementation on 2026-08-25

Scope: Development only

Implementation state: Groups A through C, the portal, workspace, and authenticated HTTP preview origin exist as reviewed source. Live activation remains disabled.

Development identity amendment, 2026-09-02: Cloudflare Access email one-time PIN replaces the proposed Entra, Conditional Access, MFA-claim, PIM, and human-approval gates for development users. The exact openmined.org email domain is the development allowlist. Normal development session create, read, optional write selection, use, close, and expiry need no human approval. Azure workload applications can still use Entra service principals. Retained PIM, Conditional Access, and Entra human-login files are inactive reference. Staging and production remain deferred and require separate stronger identity and reviewer controls.

The portal source has a requester and platform-owner session-close control. Its browser proof covers authorization visibility, confirmation, idempotent retry, pending state, accepted state, and error recovery.

1. Executive decision

Build Firecall v2 as a browser debug workspace.

Do not build it as a styled ttyd page.

The workspace will provide:

  • A Cloudflare-authenticated portal.
  • Self-service session creation.
  • Durable build progress.
  • A browser terminal.
  • Shared tmux sessions.
  • k9s and exact debug commands.
  • Workload, log, and event views.
  • Prefilled SigNoz links.
  • Safe co-debug links.
  • Bounded HTTP web previews.
  • Automatic close and expiry.

The ephemeral box remains the Kubernetes credential boundary.

The browser, broker, terminal proxy, and Cloudflare connector receive no Kubernetes credential.

The existing GitHub Actions and SSH system remains available during the canary.

2. Product names

The repository currently uses three conflicting version names.

Use these names from this plan forward:

Name Meaning State
Firecall Classic Current GitHub Actions, Terraform, VM, and SSH flow Live
Firecall v1 draft Unlanded portal design and stale ADR branch Superseded
Firecall v2 Portal and browser workspace in this plan Active implementation
Firecall v2.1 Deferred editor, transfer, and advanced collaboration work Deferred

Firecall v2 absorbs and supersedes the unlanded v1 design.

3. Verified baseline

This plan uses live evidence from 2026-08-24.

3.1 What works now

Firecall Classic works end to end.

Recent successful runs include:

  • Open for sergey.nikitin@openmined.org: run 32741021375.
  • Open for ionesio@openmined.org: run 32742041033.
  • Close for ionesio@openmined.org: run 32742632928.
  • Open for stephen@openmined.org: run 32743597268.
  • Close for stephen@openmined.org: run 32746555384.
  • Scheduled sweep: run 32749874303.

Azure contained no Firecall resource group after the last close.

3.2 Current Classic design

Classic has these properties:

  • A human dispatches .github/workflows/firecall.yml.
  • scripts/ci/firecall.sh serializes operations across three Terraform state slots.
  • The exact state keys are azure/firecall/dev.tfstate, azure/firecall/dev-slot2.tfstate, and azure/firecall/dev-slot3.tfstate.
  • Up to three Classic sessions can exist.
  • Each new Classic session uses rg-firecall-dev-<session_id>-eastus.
  • A pre-concurrency session can still use rg-firecall-dev-eastus. Close, sweep, migration queries, and zero proof retain that selector until live readback proves it absent.
  • The requester supplies an SSH public key.
  • The VM receives one user-assigned managed identity.
  • The identity receives Cluster User access.
  • The identity receives Writer access for profile namespaces.
  • The VM installs tools through cloud-init.
  • A scheduled workflow destroys expired sessions.

The default profile currently grants Writer access to:

  • sf-aigw.
  • sf-fusion.
  • sf-scoreboard.

Writer access includes Kubernetes Secret access.

Writer access can also use each namespace's ServiceAccounts.

A developer can copy Secret values and ServiceAccount tokens outside Firecall.

Deleting the box identity cannot revoke copied Secret values.

A copied bound ServiceAccount token can remain valid until its own expiry or bound-object invalidation.

Firecall revokes its access path, not information that a developer already copied.

A Writer can also leave a Pod or controller that continuously refreshes a bound ServiceAccount token.

That namespace mutation can keep authority indefinitely until an application owner removes it.

Firecall does not automatically revert developer Kubernetes mutations at close.

The v2 portal must show this complete scope before creation.

3.3 What does not exist

No reusable v1 product code exists.

The shared infrastructure worktree contains unrelated in-progress Claude work.

This plan uses an isolated worktree and does not modify that work.

The firecall-v1 branch contains current Classic code only.

The firecall-v1-s1-adr branch contains documentation only.

The private OpenMined/firecall-control repository exists, but it is empty.

It has no default branch, workflows, settings, or protection.

Its GitHub default OIDC subject prefix is repo:OpenMined/firecall-control.

Numeric owner and repository IDs remain separate OIDC claims.

The existing infrastructure repository keeps subject prefix repo:OpenMined/infrastructure.

AKS has no Firecall namespace or Argo CD application.

No Firecall Cloudflare apex, tunnel, or Access application exists.

3.4 Current network facts

AKS uses loadBalancer outbound networking.

Its current effective outbound address is 52.234.174.24.

The address is a Standard static Azure public IP.

V2 does not use that public address for proxy-to-box traffic.

The live AKS VNet is 10.224.0.0/12.

Its node subnet is 10.224.0.0/16.

Its overlay Pod CIDR is 10.244.0.0/16.

Its Service CIDR is 10.0.0.0/16.

The AKS VNet has no current peering.

A subscription-wide VNet review found no use of 10.240.0.0/24.

Reserve that non-overlapping range for the standing Firecall VNet.

The dev AKS API is public at aks-dev-eastus-uax0v3k5.hcp.eastus.azmk8s.io.

AKS API authorized IP ranges are not enabled.

The session box reaches that API through its own explicit public address.

3.5 Current cluster capability

Live kubectl version reports AKS Kubernetes v1.35.6 on 2026-08-24.

This version supports the stable admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy API.

The live API serves both ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding.

AKS Advanced Container Networking Services security features are disabled.

Do not depend on Cilium FQDN or layer-seven policy features.

AKS OIDC and workload identity are enabled.

The Azure Key Vault CSI add-on is disabled.

The repository already deploys External Secrets with workload identity.

3.6 Current cluster capacity

The live user node is one Standard_D4as_v5 node.

It has 3860m allocatable CPU and approximately 11.5 GiB allocatable memory.

Current requests consume 3370m CPU and approximately 8.7 GiB memory.

Live use was 71 percent CPU and 77 percent memory during this review.

This is not safe headroom for the Firecall control plane.

Do not place Firecall on the tenant user node or the untrusted preview node.

The live aks-system node uses Standard_D2as_v5.

It has 1900m allocatable CPU and 5,928,364 KiB allocatable memory.

Current requests consume 171m CPU and approximately 1,679 MiB memory.

Two binding-validator replicas add only 50m CPU and 128 MiB memory requests.

This measured system-pool capacity is sufficient for that validator baseline.

Add one tainted, fixed-size Firecall node pool before activation.

East US BS-family quota is 10 vCPUs with 4 currently used.

East US Standard DSv5-family reports limit 0 and current use 24. It cannot add a node.

East US DDSv5-family quota is 350 vCPUs with 0 currently used.

East US Total Regional vCPU quota is 1,581 vCPUs with 181 currently used.

East US Standard public-address quota is 100 with 39 currently used.

The running Classic vm-firecall-dev accounts for 2 vCPUs.

Other live subscription use accounts for the remaining 2 vCPUs.

One Standard_D2ds_v5 Firecall node needs two DDSv5-family vCPUs.

Three session boxes need 6 vCPUs.

One AKS upgrade surge node needs another two DDSv5-family vCPUs.

Coexistence needs eight BS-family vCPUs: two other, two Classic, and four for two v2 boxes.

Post-cutover needs eight BS-family vCPUs: two other and six for three v2 boxes.

The fixed node and one surge need four DDSv5-family vCPUs.

One concurrent image builder increases that DDSv5-family requirement to six vCPUs.

Both phases need 12 Total Regional vCPUs when the surge exists.

A concurrent image builder increases that Total Regional requirement to 14 vCPUs.

Do not add three v2 sessions while also counting the retired Classic VM.

The live 10-vCPU BS-family quota supports the approved limits without a quota increase.

Keep the coexistence capacity at two while Classic can open.

Set capacity to three only after the signed Classic zero proof passes.

Every capacity check must also pass the Total Regional vCPU quota.

The regional quota has ample current headroom, but another workload can consume it before activation.

East US Standard IPv4 quota is 100 addresses with 39 currently used.

Three session addresses fit that quota with 58 addresses of remaining headroom.

4. Product outcome

A developer will need only a browser.

The developer flow will be:

  1. Open the Firecall portal.
  2. Complete Cloudflare Access email one-time PIN authentication.
  3. Select one or more reviewed service profiles.
  4. Keep read, or select write for each service that needs it.
  5. Review every namespace and effective access mode.
  6. Select the time limit.
  7. Enter a reason.
  8. Create the session.
  9. Watch real build stages.
  10. Open the workspace after readiness succeeds.
  11. Use the terminal, k9s, logs, events, and SigNoz links.
  12. Share the join link when co-debugging is necessary.
  13. Close the session early, or let it expire.

The developer will not run:

  • az login.
  • az aks get-credentials.
  • kubelogin.
  • firecall-connect.
  • A kubeconfig download command.

5. Scope

5.1 Firecall v2 scope

V2 includes:

  • Development sessions only.
  • A maximum of three v2 sessions after cutover.
  • Cloudflare Access email one-time PIN authentication.
  • Exact @openmined.org identity checks.
  • No Entra, Conditional Access, PIM, or human approval in the development session path.
  • Read access by default for every selected service.
  • Optional write access for each selected service.
  • Requester and platform-owner session lists.
  • Requester and platform-owner close controls.
  • A revocable random join capability in the URL fragment.
  • Browser terminal access.
  • Shared tmux state.
  • Named terminal tabs.
  • k9s launchers.
  • A command palette.
  • Workload views.
  • Current and previous container logs.
  • Live log streams.
  • Kubernetes events.
  • Fixed structured workload details without arbitrary object YAML.
  • Prefilled SigNoz links for logs, traces, and metrics.
  • Bounded HTTP and WebSocket previews.
  • Optional SSH under an advanced section.
  • Durable lifecycle status.
  • Automatic failed-open cleanup.
  • Scheduled expiry cleanup.
  • Complete requester and operator runbooks.
  • Security, failure, load, browser, and accessibility tests.

Firecall web previews expose selected HTTP workloads through an active Firecall session.

They are not tenant pull-request preview environments.

Firecall does not replace the existing pull-request-scoped Kubernetes access flow.

5.2 Deferred to v2.1

V2.1 can add:

  • VS Code or code-server.
  • Browser file upload.
  • Arbitrary browser file download.
  • Raw TCP browser tunnels.
  • Saved home directories.
  • Per-person shell permissions.
  • Command-level human attribution.
  • Terminal recording.
  • Spectator mode.
  • Session extension.
  • Graphical Kubernetes mutations.
  • AI command generation.
  • Pull-request preview and Kargo integration.
  • Staging and production sessions, with separate Access applications, stronger corporate identity, MFA, and required environment reviewers.

6. Non-negotiable properties

  1. Firecall remains development-only.
  2. The browser receives no kubeconfig or Kubernetes token.
  3. The broker receives no Kubernetes credential.
  4. The terminal proxy receives no Kubernetes credential.
  5. Only the ephemeral box receives Kubernetes access.
  6. The control runner cannot apply infrastructure repository Terraform.
  7. Pull-request code receives no Firecall Azure credential.
  8. A session uses one independent Terraform state.
  9. Capacity uses an atomic reservation.
  10. An unreadable state blocks actions.
  11. ready requires live terminal and Kubernetes probes.
  12. A failed open cleans all resources before it releases capacity.
  13. rollback_failed continues to consume capacity.
  14. Close denies browser access before resource destruction starts.
  15. The proxy closes streams at Access-token expiry.
  16. The proxy closes streams at session expiry.
  17. The box also stops terminal access at session expiry.
  18. SSH remains disabled by default.
  19. Optional SSH requires one public key and one /32 source address.
  20. Firecall holds no global SigNoz credential.
  21. Firecall records no terminal input or output.
  22. Firecall does not claim command-level attribution.
  23. Classic remains available until the v2 canary passes.
  24. Read is the default for each selected service.
  25. Write can apply only to a selected service.
  26. Write wins when selected services share one namespace.

7. Personas and authorization

Action Requester Verified OpenMined collaborator Platform owner Service identity
Create a session Yes Yes Yes No
List own sessions Yes Yes Yes No
List all sessions No No Yes No
View detailed portal status Yes No Yes No
View safe workspace facts through join URL Yes Yes Yes No
Join through a shared URL Yes Yes Yes No
Close own session Yes No Yes No
Close another session No No Yes No
Select reviewed services Yes Yes Yes No
Select write for a selected service Yes Yes Yes No
Select a custom namespace No No No No
Select a custom Azure role No No No No
Select a custom VM size No No No No

Cloudflare can change sub after an administrator removes and adds the same person again.

Do not use sub as a durable human identity.

Use the normalized Cloudflare-verified development email as the durable identity input.

Require one ASCII email address with one @ separator. Lowercase it for canonical storage and comparison. Require the domain after the separator to equal openmined.org exactly.

Permit only letters, digits, dots, underscores, and hyphens in the local part. Reject plus addresses and routing forms. Require the developer's primary corporate address.

Derive subject_key exactly as:

hex(SHA-256(UTF8("firecall-dev-email-subject-v1\\0" || canonical_email)))

The quoted \\0 is one zero byte.

Derive one short-lived Access-token binding exactly as:

access_subject_key = hex(SHA-256(UTF8("firecall-cloudflare-sub-v1\\0" || sub)))

Use the complete lowercase 64-character subject_key for authorization, ownership, capacity, rate limits, invitations, revocations, and audit correlation.

The broker and proxy derive both keys only after full JWT validation.

Bind every CSRF token, acknowledgement cookie, handoff, and open browser stream to both keys.

Keep access_subject_key only in request memory and signed short-lived browser artifacts.

Never use access_subject_key for durable ownership, limits, invitations, owner records, or offboarding markers.

They keep the raw sub only in request memory and clear it with the request.

Never write the raw sub to Git, Blob storage, status, audit, logs, metrics, traces, workflow inputs, or signed envelopes.

The canonical email is development identity and audit metadata. Escape the original display value before HTML output.

Keep one canonical platform-owner file at platform/firecall/contracts/platform-owners.json.

It contains a schema version, a strictly increasing revision, and sorted unique exact subject_key values.

The protected owner-management script derives the stable key locally from one validated canonical @openmined.org email.

The script prints only the derived key and its first 12 characters as the display fingerprint.

Do not infer platform ownership from an email prefix, domain membership, raw subject, group name, or repository role. Require the exact derived key in the reviewed owner file.

CI canonicalizes the file and computes its SHA-256 revision digest.

GitOps creates one immutable firecall-platform-owners-<digest_12> ConfigMap and one mutable selector ConfigMap.

The selector contains only the complete digest, immutable ConfigMap name, and source commit.

Argo updates the selector only after the immutable ConfigMap exists.

The broker and proxy watch the selector, fetch the named ConfigMap, verify its digest, and atomically replace their complete in-memory list.

They keep no partially parsed or merged list.

Each owner-authorized broker-to-proxy request carries the complete loaded digest.

The proxy rejects that request when its current digest differs.

Each owner stream rechecks membership, selector resource version, and digest at most every five seconds.

A missing, malformed, rolled-back, unreadable, or digest-mismatched selector denies all owner authority.

It does not deny requester close or reduction-only cleanup.

Startup readiness fails when a configured Deployment digest, selector digest, and immutable ConfigMap digest differ.

Removal must reach both ready broker and proxy replica sets within five seconds.

Qualification injects watch loss, a stale selector, a revision rollback, and a split broker-proxy revision.

8. User interface

8.1 Portal origin

Use this origin:

https://portal.FIRECALL_APEX

Use these routes:

Route View Access
/ Session dashboard Verified human
/sessions/new New session form Verified human
/sessions/<id> Session state and controls Requester or owner
/help Quick start and runbooks Verified human

The permanent header shows:

  • Firecall name.
  • A DEV ONLY badge.
  • Current capacity.
  • Help.
  • Signed-in email.

The dashboard shows:

  • New session.
  • The user's active sessions.
  • Recent closed and failed sessions.
  • State.
  • Profile.
  • Namespace scope.
  • Requester.
  • Expiry.
  • Workspace action.
  • Close action where permitted.

Normal users see only their sessions.

Platform owners see all sessions.

8.2 New session form

The form contains:

  • Service profile cards with read and write choices.
  • Environment.
  • Complete effective namespace access list.
  • Plain access explanation.
  • Default and maximum time limits.
  • Estimated maximum session cost.
  • Required reason.
  • Optional SSH section.
  • Scope confirmation.

The cost card shows the selected duration, pinned rate date, base estimate, and exclusions.

It states that data transfer and a failed cleanup can add cost.

The first-session card shows four steps:

  1. Choose scope.
  2. Open the workspace.
  3. Use k9s or logs.
  4. Close the session when the work is complete.

The optional SSH section contains:

  • One public key.
  • The verified Cloudflare source IPv4 address.
  • The calculated key fingerprint.
  • A short-lived key proof challenge.
  • A fingerprint confirmation checkbox.

Do not let the requester enter or change the source address.

Disable optional SSH when Cloudflare reports only IPv6.

Show the fixed ssh-keygen -Y sign proof command for the submitted key.

The broker verifies that signature before it accepts the key.

Accept only Ed25519 or RSA public keys of at least 3072 bits.

Use one shared strict OpenSSH public-key parser in the broker and runner.

Accept one line with an algorithm, one base64 key blob, and an optional comment.

Reject authorized-key options, OpenSSH certificates, another key, leading or trailing whitespace, control bytes, and embedded line breaks.

Parse the binary key blob and require Ed25519 with 32 key bytes or RSA with at least 3072 bits.

Reserialize it as exactly <algorithm> <canonical-base64> and discard the optional comment before signing.

Calculate the displayed fingerprint from that canonical parsed key.

Require source_cidr to be one canonical public IPv4 address with an explicit /32 prefix.

Derive it only from Cloudflare's validated source address and append /32.

The form gets the email from the verified identity.

It does not ask the user to enter an email.

The reason is 1 to 200 visible characters.

Reject control characters after Unicode normalization.

Show this warning beside the reason field:

Do not put a password, token, key, or other secret here. Firecall stores this reason in operational records.

The resolved scope warning always says:

Reader warning: Read is the default. Reader can view most objects. Reader cannot view Secrets, roles, or role bindings.

When the scope contains write access, it also says:

Writer warning: Writer can read Secrets, run Pods as any ServiceAccount, and change resources in these write namespaces: . Firecall does not revert those changes.

The form states that write wins when read and write services share a namespace. It requires no human approval for a development write selection.

The build page shows completed stages and elapsed time.

It does not show a fake percentage.

Stages are:

  • Request verified.
  • Capacity reserved.
  • Debug box created.
  • Tools initialized.
  • Cluster access verified.
  • Terminal verified.
  • Ready.

A proved-clean build failure offers Retry as new session.

The retry creates a new idempotency key and a new session identifier.

8.3 Workspace origin

Use one random origin for each session:

https://<session_id>.FIRECALL_APEX

Use these routes:

Route View
/terminal Shared terminal tabs
/workloads Pods and workload controllers
/logs Current and previous logs
/events Kubernetes events
/previews Bounded web previews
/session Scope, state, active connections, and audit limits

The root route redirects to /terminal.

The top bar always shows:

  • Short session identifier.
  • DEV badge.
  • Selected namespace.
  • Profile.
  • State.
  • Expiry countdown.
  • Connection state.
  • Active workspace connection count.
  • Copy join link for the requester or platform owner.
  • Manage session for the requester or platform owner.
  • Portal home for a collaborator.

The workspace layout is:

┌ Firecall / f7c2a1  DEV  sf-aigw ▼  Ready  1h42m  Connected  3 connections ┐
│ [Copy join link]                                    [Manage session] │
├────────────┬─────────────────────────────────────────────────────────┤
│ Terminal   │ Shell 1 │ k9s: sf-aigw │ Logs: aigateway │ +          │
│ Workloads  ├─────────────────────────────────────────────────────────┤
│ Logs       │                                                         │
│ Events     │                browser terminal                         │
│ Previews   │                                                         │
│ Session    │                                                         │
├────────────┴─────────────────────────────────────────────────────────┤
│ aks-dev · sf-aigw · shared session · session-level audit only       │
└──────────────────────────────────────────────────────────────────────┘

8.3.1 Previews view

The Previews view shows one card for each preview.

Each card shows:

  • Display name.
  • Namespace, target kind, target name, port, and protocol.
  • Starting, Ready, Error, Idle expired, or Stopped state.
  • Remaining idle time.
  • A safe error message.
  • Open, Copy URL, and Stop actions when applicable.
  • A Cookies blocked badge.

Every acknowledged collaborator can create or stop a preview.

The stop action names the target and requires confirmation.

The create form shows target restrictions before submission.

It also shows this warning:

Firecall removes cookies and blocks Set-Cookie. Login and service-worker flows will not work.

Application Authorization forwarding is off by default.

Enabling it requires a separate confirmation for that preview.

The agent measures idle time from the last upstream request or WebSocket activity.

The card updates the idle countdown without extending it.

The agent stores only safe preview metadata on the ephemeral box disk.

A clean agent shutdown first stops every preview process.

It then writes one restart_allowed=true marker for each stopped preview.

Restore consumes each marker by persisting restart_allowed=false before it starts any process.

It revalidates each active target before process start and again after kubectl reports readiness.

A failed restore changes the card to Error and offers Create again.

8.4 Co-debug behavior

A verified OpenMined human can join through the random URL.

The shared URL has this form:

https://<session_id>.FIRECALL_APEX/join#<random_join_capability>

The URL fragment does not enter HTTP requests or Referrer headers.

The front end removes the fragment with history.replaceState before any API request.

It sends the capability only to the same-origin join-exchange route.

The capability contains 192 random bits.

Storage contains only its SHA-256 hash and invitation identifier.

The first successful exchange atomically binds the invitation to that exact subject key.

A different subject key cannot reuse the same invitation.

The requester mints one link for each collaborator.

Store all invitation records for one session in one leased index.json document.

Permit at most eight active invitations and 32 total minted invitations per session.

An invitation expires after two hours or at session expiry, whichever occurs first.

Mint permits five attempts per subject key and session in ten minutes.

It also permits 20 attempts for the complete session in one hour.

Join exchange permits 20 attempts per subject key and session in ten minutes.

It also permits 100 attempts for the complete session in ten minutes.

Use Blob-leased counters so both proxy replicas enforce the same limits.

The invitation index is at most 64 KiB.

Mint, first binding, individual revocation, and revoke-all use its lease and ETag.

A failed or unknown capability exchange creates no invitation object.

The proxy requires an active invitation for the session, profile digest, and expiry.

The session identifier alone never authorizes a collaborator.

The requester and platform owner can enter without a join capability.

Other verified humans need a valid join capability before they see session scope.

The portal does not list another user's session.

A collaborator cannot close the session.

The join flow has two separate steps.

First, the proxy authorizes the human and returns the complete scope.

It uses status ownership for the requester or platform owner.

It uses the join capability for a collaborator.

A successful authorization exchange sets a five-minute __Host-firecall_join_pending cookie with Path=/.

That signed cookie binds subject key, session, profile digest, authorization kind, authorization identifier, owner-list revision, and expiry.

Second, every human must explicitly confirm the complete scope.

The confirmation shows:

  • Requester.
  • Request reason.
  • Every selected service and its read or write mode.
  • Every namespace and its effective read or write mode.
  • Exact expiry.
  • Shared shell authority.
  • Reader limits for Secrets, roles, and role bindings.
  • Writer access to Secrets and ServiceAccounts, when selected.
  • Write precedence when services share a namespace.
  • Permission to end shared tabs and previews.

Confirmation creates a signed host-only __Host-firecall_join_ack cookie.

It binds subject key, session identifier, profile digest, authorization kind, authorization identifier, owner-list revision, and expiry.

A collaborator acknowledgement also binds the exact invitation identifier.

The proxy deletes the pending cookie after successful confirmation.

The proxy requires it for every terminal, workload, log, event, preview, and presence route.

The requester and platform owner confirm the same scope on a new browser.

Every human confirms again after the profile digest changes.

Cloudflare authentication alone never satisfies this acknowledgement.

Only the requester or platform owner can mint a new join capability in the interface.

Invitation mutation also requires a current scope acknowledgement.

They can revoke one invitation or all invitations immediately.

For every workspace HTTP request, the proxy rechecks the bound authorization kind before it contacts the agent.

For every workspace stream, it repeats that check at most every five seconds.

Requester authorization rechecks the exact session owner subject key.

Platform-owner authorization rechecks current membership and the exact reviewed owner-list revision.

Invitation authorization rechecks the exact invitation identifier and bound subject key.

An owner-list revision change invalidates every prior platform-owner pending or acknowledgement cookie.

An unreadable owner list or status object fails closed.

The proxy closes only streams whose requester, owner, or invitation authorization no longer passes.

An authenticated 404 means that an invitation is absent and unusable.

Any other invitation read failure denies new collaborator access and closes its streams within five seconds.

An authenticated 404 for the session close marker means that no marker exists.

Any other close-marker read failure denies new data access and closes all session streams within five seconds.

Never use a cached invitation or close-marker absence beyond that five-second interval.

A terminal session invalidates every invitation immediately.

The sweep deletes its invitation index no later than 30 minutes after terminal status.

The workspace shows this warning:

This is a shared terminal. All collaborators can type. Audit records identify the session, not each command author.

V2 shows an active workspace connection count.

The count means active workspace connections.

It is not a count of identified people.

Each open workspace page creates one dedicated presence WebSocket.

The box agent counts these connections for the session.

Permit at most two presence connections for one actor and eight for the complete session.

When seven connections exist, reserve the eighth slot for an actor with no current presence connection.

The agent enforces both limits across all proxy replicas by the opaque actor handle.

Each proxy also enforces two local presence connections for one actor as a defensive limit.

Release a presence slot on a clean close or after 30 seconds without a valid heartbeat.

Presence, terminal, log, and preview budgets are independent.

A noisy actor cannot consume another actor's terminal or log budget.

Terminal, log, and preview streams do not change this count.

V2 does not show collaborator names.

Names need a shared presence store and belong in v2.1.

8.5 Reconnect behavior

A browser refresh attaches to the existing tmux state.

A short network loss uses bounded automatic retries.

An idle timeout shows a Reconnect button.

An Access-token expiry starts authentication again.

The browser returns to the same workspace route after authentication.

A closed or expired session never reconnects.

The workspace warns at 15 minutes and five minutes before expiry.

After exact expiry, the page shows Access expired · cleanup pending.

It shows this derived state even when durable cleanup status still says ready.

Each terminal tab binds to one namespace when the tab starts.

Labels use forms such as Shell: sf-aigw and k9s: sf-fusion.

Changing the top selector never changes an existing terminal context.

9. Architecture

flowchart LR
    user[Developer browser]
    access[Cloudflare Access]
    tunnel[Dedicated cloudflared tunnel]
    broker[Broker and portal]
    proxy[Terminal proxy and workspace]
    egress[Credentialless HTTPS egress gateway]
    probe[Readiness probe worker]
    gh[OpenMined/firecall-control]
    runner[GitHub Actions runner]
    status[(Request, status, capacity-ledger, and revocation blobs)]
    probes[(Readiness probe blobs)]
    state[(Per-session Terraform state)]
    audit[(Immutable audit blobs)]
    azure[Azure session resources]
    box[Ephemeral debug box]
    agent[firecall-agent]
    sessiond[firecall-sessiond root supervisor]
    aks[AKS dev API]
    signoz[SigNoz UI]

    user --> access --> tunnel
    tunnel --> broker
    tunnel --> proxy
    broker --> egress
    proxy --> egress
    probe --> egress
    egress -->|approved public HTTPS| gh
    egress -->|caller TLS and identity| status
    egress -->|caller TLS and identity| probes
    gh --> runner
    runner -->|OIDC| azure
    runner --> status
    runner --> state
    runner --> audit
    azure --> box
    probe -->|internal nonce probe| proxy
    broker --> audit
    proxy --> audit
    proxy -->|mTLS, pinned server| agent
    agent -->|box managed identity| aks
    agent -->|bounded local Unix protocol| sessiond
    agent --- box
    sessiond --- box
    user -->|new tab, prefilled filter| signoz

9.1 Control plane

The control plane contains:

  • The portal and broker.
  • The isolated control repository.
  • GitHub Actions workflows.
  • Per-session Terraform.
  • Capacity reservations.
  • Durable lifecycle status.
  • Durable close revocations and immutable audit events.

9.2 Data plane

The data plane contains:

  • The terminal proxy.
  • The readiness probe worker.
  • The ephemeral box.
  • The box-local agent.
  • The box managed identity.
  • The AKS API.

The broker enters the data plane only through one reduction-only internal revocation route.

It cannot reach a debug box or any normal proxy route.

The proxy never receives a Kubernetes credential.

10. Components

10.1 Broker

Build the broker in Go 1.27.0.

Pin the exact supported patch in CI and container builds.

The broker will:

  • Serve the portal API.
  • Serve embedded portal assets.
  • Validate Cloudflare JWTs on every request.
  • Enforce the exact human identity policy.
  • Enforce Origin and CSRF checks.
  • Read and filter profiles.
  • Read status with Azure workload identity.
  • Create signed request envelopes.
  • Dispatch control workflows through one GitHub App.
  • Use the returned workflow run identifier.
  • Enforce exact cross-replica subject rate limits through Blob leases.
  • Filter session lists by requester_subject_key.
  • Authorize close requests.
  • Create close-revocation markers before authorized close dispatches.
  • Deliver authenticated immediate-revocation operations through the fixed internal proxy route.
  • Dispatch one idempotent close when an active session reaches expiry.
  • Append broker audit events.
  • Return safe error codes.

The broker will not:

  • Receive a kubeconfig.
  • Read Terraform state.
  • Write Azure status.
  • Create Azure resources.
  • Reach a debug box.
  • Hold a proxy client key.
  • Hold a tunnel token.

10.2 Terminal proxy

Build the terminal proxy in Go 1.27.0.

The proxy will:

  • Serve embedded workspace assets.
  • Validate the terminal Cloudflare audience.
  • Validate the exact one-label session hostname.
  • Validate the browser Origin.
  • Read the trusted session status.
  • Require the box address inside the exact Firecall private subnet.
  • Use fixed destination port 8443 only.
  • Serve a trusted lifecycle page for every known session state.
  • Require ready only for data-plane APIs and streams.
  • Require an unexpired session for data-plane APIs and streams.
  • Pin the box certificate fingerprint.
  • Use mutual TLS to the box agent.
  • Proxy terminal, workload, log, event, and preview streams.
  • Proxy one dedicated presence stream for each workspace page.
  • Manage hashed, revocable invitations for requesters and platform owners.
  • Require a scope acknowledgement on every data-plane route.
  • Consume Firecall and Cloudflare cookies at the proxy and never forward them to the agent.
  • Reject upstream cookies and redirects on terminal and structured-agent routes.
  • Apply the isolated preview cookie and redirect policy on preview routes.
  • Enforce connection and message limits.
  • Recheck status every 15 seconds.
  • Recheck close-revocation markers every five seconds.
  • Recheck each collaborator invitation every five seconds.
  • Close connections on unreadable status.
  • Deny new data connections and close all session streams when a close-revocation read is unreadable.
  • Deny and close collaborator streams when their invitation read is unreadable or absent.
  • Close connections at JWT expiry.
  • Close connections at session expiry.
  • Accept only the broker revoker identity on the internal session-revocation route.
  • Reread the exact global identity marker before it contacts a box.
  • Store one create-once delivery receipt after the box agent confirms local revocation.

The proxy will not:

  • Dispatch a workflow.
  • Read Terraform state.
  • Receive a kubeconfig.
  • Query Kubernetes directly.
  • Hold a GitHub credential.
  • Hold a broker signing key.

10.2.1 Immediate offboarding delivery

Expose one cluster-internal proxy Service for POST /.internal/v1/session-revocations.

Do not add this route to a Cloudflare tunnel, public Service, workspace Service, or preview Service.

Network policy permits only the broker Pods to reach this Service.

The listener requires mutual TLS with URI identity spiffe://firecall/broker-revoker and the selected exact SPKI.

The broker sends one canonical signed request per selected session.

The request binds schema, session ID, operation ID, random 256-bit nonce, subject key, marker path, marker ETag, marker digest, issue time, and expiry.

Set request expiry to at most 60 seconds after issue time.

The broker-revoker private key signs the request and also authenticates mutual TLS.

The proxy rejects an extra field, duplicate key, noncanonical value, wrong key, stale time, wrong path, or mismatched TLS identity.

It rereads the exact global identity marker without a list operation.

It verifies the path, subject key, ETag, digest, session binding, and active offboarding operation.

It then calls the fixed agent revoke operation through the existing pinned mutual-TLS connection.

The agent accepts no command, process identifier, signal, path, user, or timeout from this request.

The agent asks root firecall-sessiond to stop sshd and kill the complete interactive user slice.

After agent acknowledgement, the proxy creates this immutable delivery object with If-None-Match: *:

firecall-revocations/deliveries/<session_id>/<operation_id>.json

The object binds every request field, proxy replica, agent receipt digest, result, and completion time.

It contains no raw subject, email, reason, token, command, terminal data, or private address.

The proxy signs the receipt with its selected proxy-client key.

Both proxy replicas can read only the exact derived delivery path for idempotency.

The same operation and nonce return the same receipt.

A different nonce, marker, subject key, session, or request digest for that operation fails.

The broker verifies the proxy chain, URI identity, selected SPKI, request binding, agent receipt, and stored-object ETag.

It retries both ready proxy replicas until one valid receipt arrives.

The durable offboarding operation stores the receipt digest and resumes delivery after a broker or proxy restart.

This path needs no browser, presence stream, terminal stream, or SSH connection.

If both proxy replicas are unavailable, the global marker still denies new Firecall access.

Status then says Access revoked · local delivery pending and emits a high-priority alert.

The broker retries until local delivery succeeds or Azure cleanup proves the box no longer exists.

Do not claim five-second local process termination during a complete proxy outage.

Close dispatch and cleanup continue while delivery retries.

10.3 Controlled HTTPS egress gateway

Run two replicas of a small credentialless HTTPS CONNECT gateway.

CONNECT is the HTTP method that creates an encrypted tunnel to one destination.

The gateway will:

  • Expose one fixed listener for each approved Firecall component.
  • Bind each listener to one component-specific destination allowlist.
  • Accept each listener only from that component's protected Pod label.
  • Accept only TCP destination port 443.
  • Accept only exact reviewed DNS names.
  • Canonicalize names through strict ASCII and IDNA processing before comparison.
  • Reject user information, IP literals, wildcard input, extra separators, and ambiguous ports.
  • Resolve each approved name through cluster DNS.
  • Reject loopback, link-local, multicast, private, documentation, and other reserved results.
  • Revalidate every resolved address before each dial.
  • Buffer and parse the first complete TLS ClientHello before forwarding application bytes.
  • Require exactly one visible SNI host equal to the canonical CONNECT hostname.
  • Reject absent, duplicate, malformed, IP, wildcard, or mismatched SNI values.
  • Reject the TLS encrypted-client-hello extension because it hides the authoritative SNI.
  • Require the client to offer TLS 1.3 and the server to select TLS 1.3.
  • Reject TLS 1.2 and earlier so encrypted renegotiation cannot change the destination identity.
  • Apply bounded DNS, dial, idle, connection, and byte limits.
  • Log only the component, approved hostname, result, bytes, and safe reason.
  • Tunnel TLS without decrypting it.

The calling component remains the TLS client.

It validates the real destination certificate and hostname end to end.

The gateway validates the visible destination name but does not replace certificate validation.

It forwards bytes only after the ClientHello and ServerHello checks succeed.

The gateway cannot see an encrypted HTTP Host or HTTP/2 :authority value.

It also cannot force a compromised caller to validate the server certificate.

Therefore, this is an approved resolved-IP, visible-SNI, and TLS-version boundary.

It is not an exact application-layer hostname boundary.

A multi-tenant endpoint can route an allowed-SNI connection through a different encrypted authority.

Any approved endpoint can also receive a covert channel in encrypted request data or timing.

Record both paths as accepted residuals for each credentialed caller.

The gateway receives:

  • No ServiceAccount token.
  • No Azure workload identity.
  • No Key Vault secret.
  • No GitHub or Cloudflare credential.
  • No storage credential.

Use exact allowlists for GitHub, Entra, Key Vault, Azure Storage, Azure management, and Cloudflare JWKS endpoints.

Keep the reviewed allowlists in component configuration and gateway configuration.

Fail CI when those two configurations differ.

Use separate ClusterIP Services and listener ports for broker, proxy, readiness worker, and certificate rotator.

Standard NetworkPolicy binds each protected source Pod label to only its assigned listener port.

A validating admission policy binds each protected label to one exact namespace, ServiceAccount, and digest-pinned workload.

The admission policy rejects a label on every other Pod.

The gateway does not infer a ServiceAccount from a TCP connection.

It does not call the Kubernetes API to identify a caller.

Use these exact gateway limits:

  • Two seconds for DNS resolution.
  • Five seconds for the destination TCP dial.
  • 64 KiB and three seconds for the complete ClientHello.
  • 64 KiB and five seconds for the complete ServerHello.
  • 32 active connections for one source Pod.
  • 128 active connections for one component listener.
  • 512 active connections for one gateway replica.
  • 512 MiB transferred in both directions for one connection.
  • Sixty seconds without transferred bytes before idle close.
  • Thirty minutes maximum connection lifetime.
  • 64 KiB buffered in each direction before backpressure.

Acquire source, listener, and replica counters before DNS or dial.

Release every counter on denial, close, timeout, cancellation, or replica shutdown.

The gateway is not a general browser or shell proxy.

The session box does not use it.

10.3.1 DNS policy proxy

Run two credentialless DNS policy-proxy replicas.

Give each credentialed Firecall Pod dnsPolicy: None and no search suffix.

Set only the fixed DNS policy-proxy Service address as its nameserver.

Use a separate listener and exact query allowlist for each component.

Permit only required internal Service names and query types.

The credentialless HTTPS egress gateway resolves its reviewed public names through the cluster resolver.

Give cloudflared one separate listener for its exact internal origins and reviewed Cloudflare discovery names.

The proxy parses each DNS request and creates a new upstream request.

It never forwards the caller's packet, transaction identifier, EDNS data, padding, or extra sections.

It accepts and discards one empty EDNS(0) capability record from the Go resolver. The record must have the root owner, type OPT, payload size 512 through 4096, version zero, no flags, and no option data. All other additional records fail. This does not enable EDNS forwarding or change the reviewed name/type inventory.

It randomizes the upstream identifier and returns only normalized approved records.

Reject an arbitrary label, search-suffix expansion, unsupported type, IP literal, compression loop, and fragmented request.

Apply the same policy to UDP and TCP DNS.

Permit A and AAAA by default.

Permit another type only for one exact reviewed cloudflared query.

Set bounded response size, TTL, cache, rate, and concurrent-query limits.

Accept exactly one DNS question and a maximum 253-byte canonical query name.

Limit the complete UDP or TCP query to 1,232 bytes and the normalized response to 4,096 bytes.

Clamp positive TTL values to 30 through 300 seconds and negative cache entries to 30 seconds.

Keep at most 1,024 entries for one component listener.

Permit 100 queries each second for one source Pod, with a burst of 200.

Permit 32 outstanding queries for one source Pod and 256 for one listener.

Use a two-second upstream deadline and a five-second TCP idle deadline.

Acquire rate, concurrency, and response-buffer capacity before the upstream query.

Release that capacity on response, rejection, timeout, cancellation, or replica shutdown.

The proxy has no ServiceAccount token, workload identity, storage access, or secret.

DNS timing remains a low-bandwidth covert channel to an approved resolver path.

Rate limits and exact names bound that residual but do not remove it.

10.4 Readiness probe worker

Run a separate firecall-probe Deployment and ServiceAccount.

The worker reads minimal nonce-bound objects from firecall-probe-requests only.

It calls a dedicated internal proxy probe Service with mutual TLS.

The proxy tests its real mutual-TLS route and one disposable PTY.

The proxy signs the complete result with its current proxy client key.

The result binds session, nonce, box private address, server fingerprint, proxy SPKI, protocol versions, PTY challenge, and time.

The worker writes that signed result to firecall-probe-results.

The worker cannot write status, state, the capacity ledger, revocations, or audit retention policy.

The worker cannot read full lifecycle status.

It cannot create a valid result without the proxy signature.

10.5 Box-local firecall-agent

Add one small Go agent to the immutable VM image.

The agent is necessary for the rich workspace.

Raw ttyd cannot safely provide structured resources, tabs, logs, events, and previews.

The agent will:

  • Listen on TLS port 8443.
  • Require the proxy client certificate.
  • Check the exact proxy certificate identity.
  • Use a server key generated on the box.
  • Manage persistent terminal tabs through tmux.
  • Spawn PTYs as the unprivileged firecall user.
  • Use a root-owned loopback Kubernetes API proxy.
  • Enforce the profile namespace list.
  • Read selected Kubernetes resources.
  • Stream logs and events.
  • Manage bounded port forwards.
  • Stop all streams at local expiry.
  • Expose readiness probes.
  • Count active workspace presence connections for the session.
  • Enforce total terminal, log, presence, and preview limits across all proxy replicas.

The agent will not:

  • Receive a Cloudflare JWT.
  • Receive a browser cookie.
  • Receive a GitHub credential.
  • Receive a Terraform state credential.
  • Receive a Key Vault credential.
  • Record terminal input or output.

Protect the private agent listener before mutual-TLS authentication.

Enable kernel SYN cookies and use bounded SYN and accept backlogs.

Set one global pre-authentication connection cap.

Set lower concurrent and rate caps for each source address.

Reserve a separate authenticated-connection budget for the valid proxy path.

Apply a short TLS-handshake deadline and a maximum pre-authentication byte count.

Apply header, frame, and idle read deadlines before authentication completes.

Close an invalid client certificate without application processing.

Limit TLS handshakes in progress to 64 globally and eight for one source address.

Permit 20 new handshakes each second per source, with a burst of 20.

Permit 100 new handshakes each second globally, with a burst of 100.

Limit pre-authentication bytes to 64 KiB and the TLS handshake to five seconds.

Close an unauthenticated connection after ten idle seconds.

After authentication, permit 128 connections globally and 32 from one verified proxy replica.

Handshake counters release only after denial, authenticated promotion, close, timeout, or shutdown.

Promote a connection to the larger session limits only after full mutual-TLS validation.

The complete AKS node subnet can still exhaust shared NIC, kernel, or CPU capacity.

Treat that shared-node-subnet denial-of-service path as an accepted availability residual.

Harden the agent systemd service with:

  • NoNewPrivileges=yes.
  • ProtectSystem=strict.
  • ProtectHome=read-only.
  • PrivateDevices=yes.
  • ProtectKernelTunables=yes.
  • ProtectKernelModules=yes.
  • ProtectControlGroups=yes.
  • RestrictSUIDSGID=yes.
  • An empty capability bounding set.
  • Exact writable paths under /run/firecall-agent and /var/lib/firecall-agent only.

Run a separate root supervisor named firecall-sessiond.

The agent communicates with it through /run/firecall-sessiond/control.sock only.

The supervisor verifies the caller UID through Unix SO_PEERCRED.

It accepts only these bounded operations:

  • Create the shared tmux server.
  • Create one shell or k9s window for one allowed namespace.
  • Attach one PTY to one existing window.
  • Resize one attached PTY.
  • End one selected window.
  • Revoke the complete configured session through one idempotent fixed helper.

The request contains only operation, opaque window identifier, terminal size, launcher kind, and allowed namespace.

The supervisor reads allowed namespaces, absolute binary paths, environment, and kubeconfig paths from root-owned configuration.

It rejects caller-supplied commands, arguments, paths, environment, working directories, file descriptors, and user identifiers.

It never invokes a shell to construct a command.

The revoke operation accepts no command, path, service name, process identifier, or network value.

It verifies the configured session identifier, writes one root-owned revoked marker, and invokes only /usr/local/libexec/firecall-revoke.

That fixed helper stops sshd, the kube proxy, agent streams, sessiond children, and every interactive firecall process.

The local expiry timer invokes the same helper.

It uses fixed execve argument arrays.

It clears supplementary groups and drops to the firecall UID and GID before each child exec.

It removes every capability before the child exec.

It passes only the PTY master to the agent through SCM_RIGHTS.

The supervisor receives no network request and records no PTY bytes.

Harden it with NoNewPrivileges, strict filesystem protection, and only CAP_SETUID, CAP_SETGID, and CAP_CHOWN while launching.

Allow writes only to the Firecall runtime directory, bounded home filesystem, and bounded temporary mounts.

Do not give the agent or interactive user a sudo, setuid, Polkit, or systemd-management path.

The interactive user cannot authenticate to the local agent endpoint.

Run a second root-owned service named firecall-kube-proxy.

This service binds only 127.0.0.1:9443.

The root boot helper generates at least 256 random bits for one per-session local bearer token.

It generates the token on the box after first boot and never accepts token material from cloud-init.

It writes fixed root-owned proxy files and separate read-only kubeconfigs for the agent and firecall user.

It never writes the token to Terraform, custom data, Run Command output, status, logs, audit, or a process argument.

The proxy accepts that token only through one inbound Authorization: Bearer header.

That local token has no authority outside the box.

The service obtains the box managed-identity token from Azure Instance Metadata Service.

It adds that Azure token only to the upstream AKS request.

It uses one configured upstream origin: https://<verified-aks-api-host>:443.

It accepts only origin-form request targets under /api, /apis, /openapi, and /version.

It rejects absolute-form targets, authority-form targets, CONNECT, invalid paths, unexpected protocol upgrades, and any configured upstream change.

It ignores the inbound Host value and writes the fixed verified AKS host.

It never follows an upstream redirect and returns a fixed local error for every 3xx response.

Before upstream dispatch, it removes inbound authorization, proxy, forwarding, hop-by-hop, cookie, and Kubernetes impersonation headers.

It rejects duplicate security headers, conflicting length headers, ambiguous transfer encoding, folded headers, invalid framing, and request trailers.

It then adds one Azure authorization header and the fixed upstream host.

It removes upstream authorization, proxy-authentication, forwarding, Set-Cookie, and hop-by-hop response headers.

Allow only HTTP methods needed by the Kubernetes API.

Allow only a WebSocket upgrade on an exact exec, attach, or portforward subresource path.

Reject every other upgrade and never tunnel an arbitrary TCP destination.

Limit request headers to 32 KiB, response headers to 64 KiB, and nonstreaming request bodies to 16 MiB.

Apply a ten-second connect deadline, a 30-second nonstreaming header deadline, and the local session expiry to every stream.

Apply bounded concurrent requests, streams, response buffers, and per-user backpressure.

Permit 20 accepted Kubernetes requests each second, with a burst of 40.

Permit 32 concurrent nonstreaming requests and 12 concurrent upgrade streams.

Within those 12 streams, permit four each for exec, attach, and portforward.

Limit one nonstreaming response buffer to 1 MiB and all response buffers to 8 MiB.

Keep the 16 MiB request-body limit and reject excess bytes during streaming.

Close a nonstreaming request after five minutes.

Close an upgrade after 15 idle minutes or at the exact session expiry, whichever occurs first.

Acquire request, stream, and buffer capacity before IMDS or AKS access.

Release it on completion, rejection, client close, upstream close, timeout, cancellation, expiry, or service shutdown.

It supports normal HTTP requests and Kubernetes upgrade streams for exec, attach, and port-forward.

It validates the exact AKS API hostname and certificate chain.

It never returns the Azure token through an API, file, process environment, error, or log.

Root-owned nftables rules deny 169.254.169.254 to the firecall UID and every interactive cgroup.

Only the fixed dedicated kube-proxy service UID and systemd cgroup can reach the managed-identity token endpoint.

The interactive user cannot change these rules or inspect the kube-proxy process.

The agent and interactive tools use the loopback kubeconfig.

Azure RBAC remains the namespace authorization boundary behind that proxy.

The local expiry timer stops the kube proxy before it stops the terminal.

10.5.1 Protected AKS connection contract

Keep the public cluster trust contract at platform/firecall/contracts/aks-connection.json.

Keep the reviewed AKS CA bundle at platform/firecall/trust/aks-dev-ca.pem.

The JSON contains exactly:

  • Schema version.
  • Complete AKS resource ID.
  • Entra tenant ID.
  • Canonical API server hostname.
  • API server port 443.
  • Expected server DNS name.
  • CA PEM path and SHA-256 digest.
  • Kubernetes version.
  • Token resource audience 6dae42f8-4368-4678-94ff-3960e28e3630.
  • ARM systemData.createdAt and connection-contract generation.
  • Live retrieval time and evidence digest.
  • Managed-identity selection mode explicit_client_id.

The CA certificate and identifiers are public metadata.

The contract contains no token, client secret, private key, kubeconfig bearer value, or admin credential.

Protect both files with CODEOWNERS and required CI.

Bind their digests into the control profile, immutable image evidence, signed request, and allowed-image statement.

The VM entry in platform/firecall/release-evidence/required-subjects.json is the allowed-image statement.

All four authorities bind the same connection-contract generation, connection-contract SHA-256 digest, and CA-bundle SHA-256 digest.

Before every open, the runner reads the live AKS resource and user credential metadata.

It compares resource ID, tenant, host, port, DNS name, CA bytes, CA digest, Kubernetes version, audience, and ARM creation time.

Any difference blocks reservation and starts the connection-contract rotation path.

Use two explicit Terraform phases inside the one leased session operation.

The identity phase creates only the session managed identity and its deterministic AKS role assignments.

It creates no VM, NIC, disk, public address, or NSG.

The runner reads the resulting client ID, object ID, resource ID, and assignment IDs from exact Terraform outputs.

The fixed Go helper then generates one ephemeral Ed25519 provisioning key pair in locked process memory.

It calculates the canonical SHA-256 fingerprint of that public key before it constructs the bootstrap payload.

It keeps the private key only in locked runner memory until the compute apply ends.

It then creates one RFC 8785 canonical session-bootstrap payload.

The payload has this closed schema:

  • Schema identifier firecall.session-bootstrap.v1.
  • Session ID and request ID.
  • Signed-request SHA-256 digest.
  • Exact protected control commit and privileged-open workflow commit.
  • Profile name and SHA-256 digest.
  • Sorted unique allowed namespaces.
  • UTC reservation time and exact expiry.
  • Managed-identity client ID, object ID, and complete resource ID.
  • Complete deterministic AKS role-assignment IDs.
  • Static AKS connection-contract generation and SHA-256 digest.
  • Static AKS CA-bundle SHA-256 digest.
  • Exact immutable VM image version ID and evidence digest.
  • Selected proxy-client leaf SPKI SHA-256 digest.
  • Pinned client-root bundle SHA-256 digest, root-lock SHA-256 digest, and root-lock generation.
  • SSH mode, canonical requester public key, fingerprint, source IPv4 /32, and proof digest when SSH is enabled.
  • Empty SSH fields when SSH is disabled.
  • Ephemeral provisioning-admin public-key fingerprint.
  • One random 128-bit bootstrap nonce.

It contains no email, subject key, join capability, private key, token, secret, command, script, or arbitrary environment value.

The optional requester SSH public key is public configuration and appears only in its typed SSH object.

The runner keyless-signs the exact canonical payload through Sigstore from privileged-open.yml on protected control main.

The offline Sigstore bundle binds the GitHub issuer, repository, numeric repository owner, workflow path, workflow commit, source commit, payload digest, and transparency-log time.

Wrap the base64url payload and Sigstore bundle in one strict JSON envelope.

Limit the canonical payload to 16 KiB and the complete decoded envelope to 32 KiB.

Reject an envelope whose base64 form exceeds 48 KiB before Terraform planning.

The runner writes the typed compute input only through that fixed Go serializer to one mode-0600 temporary .auto.tfvars.json file.

Do not use shell interpolation, a command-line variable, an environment variable, a template directive, YAML, or cloud-init commands.

Set the Azure administrator username to exactly firecall-provision and disable password authentication.

The helper writes one typed session_compute_input object with only the signed envelope and provisioning public key.

Terraform passes the public key only through azurerm_linux_virtual_machine.admin_ssh_key.

It sets VM custom_data to the base64-encoded exact envelope bytes.

Do not put the provisioning private key in Terraform, custom data, a file, environment, argument, output, state, log, or artifact.

Keep it only in locked runner memory until the compute apply ends, then overwrite and release that memory.

The session NSG never permits the runner address to use this key.

Its saved plan must add only the expected NSG, public address, NIC, disk, and VM after the already proved identity objects.

Terraform state and the Azure VM model retain the public envelope.

The status record binds the envelope, payload, request, and plan digests before the compute apply.

The runner securely removes its temporary variable file after apply or failure.

Cloud-init stores the exact decoded bytes at /var/lib/cloud/instance/user-data.txt.

The root finalization unit passes only that fixed file to the typed Firecall bootstrap parser.

The parser rejects duplicate keys, unknown fields, invalid UTF-8, noncanonical JSON, bad base64url, size excess, or extra bytes.

It verifies the Sigstore bundle offline against image-pinned Fulcio and Rekor trust.

It requires the exact control repository, numeric owner, workflow, protected main ref, workflow commit, source commit, and payload digest.

It compares every static contract, CA, image, profile, session, role-assignment, proxy SPKI, client-root lock, and expiry value before it writes configuration.

It finds the one Azure-created provisioning authorized-key file for firecall-provision.

It requires exactly one key with the signed provisioning fingerprint and no options or second key.

It deletes that file, removes every other default authorized-key file, locks the provisioning account, and sets its shell to nologin.

If SSH is enabled, it reparses the signed requester key and source /32.

It writes only from="<validated-ip>",restrict,pty <canonical-key> to /etc/ssh/firecall_authorized_keys.

If SSH is disabled, it writes no requester key and keeps sshd disabled.

It verifies that AuthorizedKeysFile names only /etc/ssh/firecall_authorized_keys before SSH can start.

The finalizer accepts the signed session envelope only from /var/lib/cloud/instance/user-data.txt.

It accepts the Azure provisioning key only from /home/firecall-provision/.ssh/authorized_keys.

It reads immutable verification input only from /usr/local/share/firecall/bootstrap-material.json.

It installs the validated session, trust, connection, scope, evidence, and session-supervisor documents under fixed paths.

It generates the agent and loopback-proxy private keys, certificates, and local bearer token on the box.

It writes separate fixed namespace kubeconfigs under /etc/firecall-agent/kubeconfigs and /etc/firecall-user/kubeconfigs.

When enabled, it installs the one restricted requester SSH authorized key.

It writes /run/firecall-bootstrap/finalized only after it applies the final network guard.

Every installed file is root-owned. Its fixed mode grants only the required root, service-group, or user-group access.

It removes the cloud-init source file after successful validation and records only safe digests in the local readiness result.

Any signature, identity, schema, digest, expiry, scope, image, assignment, or file-mode failure keeps agent, kube proxy, SSH, and login stopped.

The watchdog destroys the identity and every partial session resource after a bootstrap failure.

The kube proxy requests IMDS tokens with the exact configured client_id and exact audience.

It never uses IMDS default-identity selection.

It loads only the reviewed CA file for AKS validation.

It verifies the exact DNS name and does not use system roots, insecure mode, or trust on first use.

Rotate an AKS host, CA, audience, tenant, or cluster identity in this order:

  1. Disable session creation.
  2. Close all active Firecall sessions.
  3. Retrieve and independently verify the new public connection facts.
  4. Merge the owner-reviewed contract and CA update.
  5. Rebuild, scan, seal, and pin the immutable image.
  6. Recheck the live cluster against the merged contract.
  7. Run one disposable managed-identity canary through the exact loopback proxy.
  8. Run complete dark preflight.
  9. Enable creation only through the normal Group F gate.

Never permit an old allowed image after the contract generation changes.

Never accept a runtime CA download, certificate skip, system-root fallback, or caller-supplied API host.

10.6 Immutable debug image

Build one immutable image with Packer.

Publish exact versions through Azure Compute Gallery.

Place the standing gallery in rg-firecall-control-dev-eastus.

Name the gallery gal_firecall_dev_eastus.

Name its image definition firecall-debug-ubuntu-2404.

Use publisher OpenMined, offer Firecall, and SKU ubuntu-24_04-debug.

Every profile and evidence statement uses the resulting complete Azure resource ID.

Do not use latest image references.

Do not run remote install scripts during session boot.

The image contains:

  • firecall-agent.
  • firecall-sessiond.
  • fc helper.
  • tmux.
  • kubectl.
  • kubelogin.
  • Azure CLI.
  • k9s.
  • stern.
  • helm.
  • jq.
  • yq.
  • fzf.
  • psql.
  • redis-cli.
  • NATS CLI.
  • grpcurl.
  • curl.
  • dig.
  • openssl.
  • socat.
  • less.
  • vim.

The firecall user has no password and no general sudo access.

The image contains no credential.

Disable swap and ship no swap file, swap partition, or swap fstab entry.

Configure chrony to use only the Azure Hyper-V PTP device /dev/ptp_hyperv.

PTP means Precision Time Protocol.

Keep the exact chrony.conf and its digest in the image lock and signed image evidence.

Configure no network server, pool, or peer source.

Disable every network NTP fallback and keep UDP port 123 denied.

Fail image qualification when the PTP device is absent or chrony selects another source.

Qualify the source first on the normal Standard_D2ds_v5 image builder.

The first Group F owner canary supplies the first Standard_B2s proof through the normal session path.

Require five consecutive authenticated samples with absolute offset at most 100 milliseconds.

Session readiness requires the same selected source, configuration digest, and five-sample offset proof.

A missing device, unsynchronized clock, source change, or excessive offset blocks readiness and starts cleanup.

Set interactive shell history defaults to disabled with zero history sizes and /dev/null as HISTFILE.

Set core size to zero and configure systemd-coredump with Storage=none and ProcessSizeMax=0.

Disable process accounting, TTY auditing, and audit rules that capture command arguments or terminal bytes.

The agent connects PTY streams directly and never sends their bytes to journald or application logs.

Set the default tmux history limit to 2,000 lines per window.

Treat tmux scrollback, user files, process memory, and the complete OS disk as session-local sensitive data.

Exclude session disks from Azure Backup, restore points, snapshots, and image capture through Policy and role denial.

Use exactly one Standard_D2ds_v5 Packer builder in East US.

That SKU has 2 vCPUs, 8 GiB memory, and a 75-GiB temporary disk.

Use one 32-GiB Standard SSD OS disk.

This keeps the captured Gallery image deployable to every 32-GiB session profile.

The live standardDDSv5Family quota was 0 of 350 vCPUs on 2026-08-24.

Activation requires at least two free DDSv5-family vCPUs.

The official retail rate was USD 0.113 per builder hour on 2026-08-24.

Budget USD 0.20 for one one-hour build, temporary disk, address, and transactions.

Create one standing build VNet 10.241.0.0/24 and subnet 10.241.0.0/28.

Attach one fixed subnet NSG named nsg-firecall-image-build.

Do not peer this VNet with AKS, Firecall sessions, or another VNet.

Use official HashiCorp azure-arm plugin v2.6.3 at commit 7f7e7b4aabf41dc853d6d65c494d95c258150bc4.

Pin its released linux_amd64 archive and checksum-manifest digests in the global tool lock.

The privileged image job logs in to Azure with its exact GitHub OIDC credential before Packer starts.

Set Packer use_azure_cli_auth = true and verify the expected tenant, subscription, application, and principal IDs.

Permit no client secret, client certificate, managed identity, device login, service-principal fallback, or environment credential.

Unset and reject every Azure secret, certificate, password, and metadata-authentication variable before packer build.

Set temp_compute_name, temp_nic_name, and temp_os_disk_name to fixed reviewed names.

Set custom_resource_build_prefix = "fcimg".

Set the existing VNet fields and private_virtual_network_with_public_ip = true.

Do not set allowed_inbound_ip_addresses or public_ip_sku with this plugin version.

Version 2.6.3 rejects those fields when virtual_network_name is set.

It also does not create or attach a temporary NSG for an existing VNet.

The fixed subnet NSG intentionally supplies the missing SSH and outbound controls.

The official plugin does not expose fixed names for its deployment or public address.

It creates fcimgdp<random5> and fcimgip<random5> names with the same suffix.

It computes fcimgsg<random5> internally, but that NSG must not exist in this configuration.

The random alphabet is the pinned plugin's lowercase alphanumeric alphabet.

The workflow captures the generated deployment and public-address names through Packer generated build data.

It asserts that the generated NSG name has no live resource.

The generated ARM template changes for each build and is not an exported Packer artifact.

Do not claim or authorize an unavailable template digest.

The separate fixed-NSG parent document contains the runner IPv4 /32.

Record the fixed-NSG document digest, callback proof, runner address, plugin checksum, and Packer configuration digest in signed evidence.

Azure Policy does not evaluate Microsoft.Resources/deployments wrapper resources.

Azure RBAC at resource-group scope cannot restrict a deployment operation to one deployment name.

A compromised builder can create another deployment wrapper and multiple policy-compliant prefixed public addresses.

Every nested resource still faces the fixed builder Policy and the builder's narrow resource actions.

Workflow concurrency permits one image build and never cancels an active build.

Pre-create firecall-image-build-operations/operation.lock.

The owner data-bootstrap script creates it once with If-None-Match: *.

Its content is one schema-valid immutable empty-lock document with generation zero.

Apply the indefinite container legal-hold tag firecalloperationlock before a runtime identity receives authority.

The script rereads its ETag, digest, and legal-hold state, and no Terraform root owns its content.

Azure authorizes Blob lease operations through the Blob write DataAction.

Give each workflow exact-object read and write DataActions, but no list, create, delete, tag, or sibling authority.

The legal hold rejects content overwrite and deletion while the write DataAction permits lease acquire, renew, and release.

Workflows use that write action only for leases and verify the fixed content digest before and after each operation.

Removing the legal hold needs the explicit deletion-approval path.

The image and standing workflows share its Blob lease.

The image workflow holds that lease from the active NSG change through idle reset and final inventory.

The standing workflow holds it through every plan, apply, and NSG verification.

Terraform owns the fixed NSG but ignores only its inline runtime rule set after bootstrap.

It never ignores the NSG identity, subnet attachment, tags, or delete protection.

A separate firecall-image-idle.yml schedule reconciles idle state every 30 minutes.

It acquires the lease, proves no image build runs, inventories temporary resources, and restores the idle rule set.

It appends one immutable image-idle/ audit event after each check.

An active NSG without a live lease or build run alerts immediately.

Azure Policy fixes the VM, NIC, and disk names.

It permits only a 12-character fcimgip name prefix for the random public address.

Policy cannot enforce one-build cardinality for those prefixes.

Packer creates the public address only for the build.

Before Packer starts, the workflow proves its current public IPv4 through an outbound callback.

The callback is a no-secret Cloudflare Worker at firecall-ip.openmined.app/v1/runner-ip.

It accepts one canonical lowercase 64-character nonce and returns Cloudflare's caller IPv4.

It rejects other hosts, paths, methods, query fields, address families, and non-public ranges.

The callback cannot require interactive Access login because the GitHub runner cannot complete that login.

The existing *.openmined.app application also matches this host.

One more-specific exact-host application uses an everyone bypass policy before the custom domain becomes public.

The Worker still rejects every other path and request shape, and signed build evidence records each accepted callback.

The Worker and custom domain remain absent while local.runner_ip_callback_open is false.

It then replaces the complete fixed subnet NSG with one Policy-compliant parent PUT.

The active rule permits TCP 22 from only that IPv4 /32.

The idle rule uses the non-routed documentation address 192.0.2.1/32.

Both shapes deny all other inbound traffic.

The fixed outbound rules permit Azure DNS and general Internet TCP 443.

They deny VNet, private, link-local, AKS, service, Pod, session, metadata, and other outbound traffic.

The stock plugin has no outbound-rule setting in version 2.6.3.

The pre-attached subnet NSG therefore enforces outbound restrictions before first boot.

The host firewall repeats those restrictions after provisioning starts.

General Internet TCP 443, normally used for HTTPS, is an image-build exfiltration residual. The network controls do not enforce TLS or HTTP.

The build VM receives no production credential, managed identity, or Firecall secret.

The host firewall permits 168.63.129.16/32 on TCP 80 and 32526 for the Azure guest agent.

Azure NSGs do not control traffic to that platform virtual address.

The marketplace source starts Azure Guest Agent before the first Packer provisioner can install this firewall.

The temporary builder therefore has a bounded pre-provisioning WireServer window.

It has no managed identity, production credential, or Firecall secret during that window.

Record this as a build-platform residual.

Image qualification proves the captured session image starts with the final firewall already enabled.

During build, it denies private, link-local, AKS, service, Pod, session, and metadata address ranges.

The final image installs the UID-scoped IMDS exception for only the root kube-proxy process.

Azure DataSourceAzure also needs IMDS during first-boot cloud-init discovery.

Install one temporary host-firewall exception for only the cloud-init systemd cgroup.

Permit that cgroup to reach IMDS TCP 80 only until cloud-final.service completes.

A root finalization unit removes the temporary rule before the agent, kube proxy, SSH, or interactive login can start.

The finalization unit then enables the permanent UID-scoped rule for only the root kube-proxy process.

The agent, SSH service, and interactive login units require successful finalization through explicit systemd ordering.

A failed cloud-init or finalization unit leaves those services stopped and makes readiness fail.

The temporary cloud-init process can request the session identity token during this bounded boot window.

Treat the protected image and session configuration as trust holders for that residual.

The builder VM has no managed identity, so that final exception grants no build-time Azure authority.

The build verifies every downloaded digest before use.

Cleanup resets the fixed NSG to its idle /32 before it removes temporary resources.

Successful cleanup uses Packer generated data and a live inventory.

Failure cleanup never depends on the missing successful artifact.

It lists the dedicated group's allowed temporary resource types.

It deletes exact deterministic children and every tagged fcimgip child.

Packer does not tag its ARM deployment wrapper.

While the current workflow still holds the lease, cleanup proves no other build run exists.

It then deletes every deployment wrapper in the dedicated image-build group.

It completes final inventory before it releases the lease.

That group contains no standing ARM deployment resource.

It never deletes the standing VNet, subnet, fixed NSG, assignment, or lock object.

Cleanup removes every tagged build resource after success or failure.

It then requires the build group to contain only the standing VNet, subnet, fixed NSG, and exact Policy assignment.

The standing build VNet and subnet remain empty between builds.

10.7 Front-end stack

Use one TypeScript workspace for the portal and workspace interfaces.

Treat every requester, Kubernetes, preview, status, audit, and upstream field as untrusted text.

Render external values only through React text nodes.

Forbid dangerouslySetInnerHTML, raw HTML, untrusted Markdown, template HTML, and string-built DOM insertion.

Construct links through the URL API from validated structured fields and fixed schemes and hosts.

Never use an external value as a URL, style, class name, element name, or React key without its specific validator.

Reject control characters at input boundaries where the contract permits rejection.

Display an escaped marker for unexpected control or bidirectional-override characters from Kubernetes or upstream data.

Apply this rule to reasons, emails, preview names, events, conditions, labels, annotations, workload fields, and safe errors.

Use:

  • React.
  • Vite.
  • xterm.js.
  • Playwright.
  • axe accessibility checks.

Pin every dependency through the lock file.

Serve all assets from the Go binaries.

Use no runtime CDN.

Use no service worker.

10.8 Isolated control repository

OpenMined/firecall-control owns the Azure session authority.

It will contain:

.github/
  CODEOWNERS
  workflows/
    validate.yml
    open.yml
    close.yml
    sweep.yml
    breakglass.yml
    capacity-bootstrap.yml
    capacity-reconcile.yml
    audit-verify.yml
    breakglass-audit-reconcile.yml
    assignment-audit.yml
    privileged-open.yml
    privileged-close.yml
    privileged-sweep-schedule.yml
    privileged-sweep-manual.yml
    privileged-breakglass.yml
    firecall-capacity-bootstrap-privileged.yml
    privileged-capacity-reconcile-push.yml
    privileged-capacity-reconcile-manual.yml
    privileged-audit-verify-schedule.yml
    privileged-audit-verify-manual.yml
    privileged-breakglass-audit-reconcile-schedule.yml
    privileged-breakglass-audit-reconcile-manual.yml
    privileged-assignment-audit-schedule.yml
    privileged-assignment-audit-manual.yml
contracts/
  breakglass-consumption.schema.json
  request-envelope.schema.json
  request-receipt.schema.json
  status.schema.json
  profile.schema.json
  capacity-ledger.schema.json
  readiness-attestation.schema.json
  revocation.schema.json
  audit-event.schema.json
  protocol-fixtures/
keys/
  broker-request-current.pem
  broker-request-previous.pem
profiles/
  dev-namespace-debug.yaml
  dev-fusion-debug.yaml
config/
  breakglass.yaml
  capacity.yaml
release-evidence/
  vm/
trust/
  sigstore/
scripts/
  verify-request
  reserve-capacity
  session
  verify-cleanup
terraform/
  session/
README.md

This repository contains no infrastructure repository credential.

Its runner can only manage Firecall session resources.

Protect control main with:

  • Pull requests only.
  • One owner review.
  • CODEOWNERS review for workflows, profiles, Terraform, and keys.
  • Required validation checks.
  • Dismissed approvals after protected-file changes.
  • No force pushes.
  • No branch deletion.
  • SHA-pinned allowed actions only.
  • No repository Actions secrets for Azure authentication.

Configure at least two platform maintainers before enabling required reviews.

One approved CODEOWNER review also satisfies the one-review requirement.

Do not require a second duplicate approval from the same reviewer class.

Keep one reviewed emergency bypass team for repository recovery.

Normal close and sweep workflows never wait for an approval environment.

The private control repository cannot rely on paid environment-review protection.

The manual break-glass path uses two distinct allowlisted humans.

The first human requests cleanup.

One different platform owner approves that exact request.

Store the exact GitHub repository ID, approval-issue number, and actor IDs in protected config/breakglass.yaml.

Use one locked issue named [Firecall] Break-glass approvals in OpenMined/firecall-control.

Do not accept approval from another issue, repository, pull request, workflow summary, or chat message.

The public breakglass.yml entry workflow accepts:

  • One exact session identifier.
  • The exact deletion confirmation text.
  • A reason of 20 through 500 characters.

It reads the requester login and numeric actor ID from the GitHub workflow-run API.

It generates 256 random bits through the operating-system cryptographic random source.

The random value is an approval correlation nonce, not an access credential.

The workflow posts one canonical request comment through its repository GITHUB_TOKEN:

/firecall-breakglass request session=<session_id> nonce=<64-lowercase-hex> commit=<40-lowercase-hex> run=<run_id> requester_id=<numeric_id> reason_sha256=<64-lowercase-hex> expires=<RFC3339>

The expiry is exactly 15 minutes after the request comment creation time.

The entry workflow masks the reason and never places it in a command line.

The approver posts this exact canonical line on the same issue:

/firecall-breakglass approve session=<session_id> nonce=<64-lowercase-hex> commit=<40-lowercase-hex> run=<run_id> reason_sha256=<64-lowercase-hex>

The privileged workflow obtains issue and workflow facts directly from the GitHub API.

It requires the request comment author to be the repository Actions bot for that exact run.

It requires one approval by a different allowlisted numeric actor ID.

It requires both comments after the dispatch time and before the 15-minute expiry.

It requires created_at to equal updated_at for both comments.

It rejects deleted, edited, duplicated, ambiguous, stale, or malformed comments.

It binds session, nonce, commit, run, requester, approver, reason digest, issue, repository, and workflow path.

It rereads the protected main commit and exact reusable workflow path before Azure login.

Immediately before cleanup, it creates this immutable Blob with If-None-Match: *:

firecall-breakglass/consumed/<sha256(nonce)>.json

The object follows breakglass-consumption.schema.json.

It stores all bindings, comment identifiers, canonical comment digests, and consumption time.

It never stores the raw nonce or reason.

A container immutability policy prevents overwrite and deletion for 90 days.

An existing object makes every later use fail.

The workflow then attempts the deterministic break-glass audit append before it starts reduction.

An audit outage does not delay cleanup.

The immutable consumption object records the exact audit path, canonical payload, and digest.

It excludes the raw nonce and reason.

Run breakglass-audit-reconcile.yml every 15 minutes and by manual dispatch.

Its public workflow calls the event-specific privileged-breakglass-audit-reconcile-schedule.yml or privileged-breakglass-audit-reconcile-manual.yml workflow on protected control main.

It uses a separate read-and-append OIDC identity.

It lists and reads only firecall-breakglass/consumed/.

It validates every marker schema, binding, canonical payload, path, and digest.

It reads only the exact expected object under firecall-audit/breakglass-consumption/.

An existing exact object means the append is complete.

An absent object permits one conditional create with the marker's exact bytes.

Every other read or create result fails and alerts.

The reconciler cannot change a consumption marker or another audit prefix.

The locked audit policy prevents overwrite and deletion.

The original workflow and reconciler create the same deterministic object.

This makes the append idempotent after the original workflow exits.

If cleanup fails after consumption, that workflow can retry its fixed reduction steps.

A different workflow run needs a new nonce and approval.

This control prevents accidental or replayed cleanup.

It does not protect against a malicious reviewed main workflow.

10.9 Portal API

Expose these broker routes:

Method Route Result
GET /api/v1/me Verified identity and owner flag
GET /api/v1/csrf Mint one short-lived portal CSRF token for the current subject key and host
GET /api/v1/profiles Reviewed profile catalog
GET /api/v1/capacity Safe current capacity summary
GET /api/v1/sessions Authorized session list
GET /api/v1/active-identities Owner-only bounded offboarding selection view
POST /api/v1/sessions Create one idempotent request
GET /api/v1/sessions/<id> Authorized session status
POST /api/v1/sessions/<id>/retry Retry one confirmed pre-reservation failure
POST /api/v1/sessions/<id>/close Request authorized close
POST /api/v1/offboarding Owner-only durable human offboarding operation
POST /api/v1/access-policy-canaries Owner-only three-audience Access identity canary batch
POST /.firecall/v1/access-policy-canary Consume one portal-audience identity canary handoff
GET /health/ready Internal readiness only

State-changing calls require:

  • Content-Type: application/json.
  • Exact portal Origin.
  • Signed CSRF token.
  • Idempotency-Key.
  • A fresh Cloudflare JWT.

GET /api/v1/active-identities requires a current owner and returns only the section 22.2 bounded fields.

It accepts no body, cursor, filter, target, or caller-supplied subject.

Permit ten active-identity reads per owner each minute and 100 service-wide each minute.

POST /api/v1/offboarding accepts exactly selection_token, typed_fingerprint, and reason.

Limit its decoded body to 4 KiB and its normalized reason to 500 Unicode scalar values.

Permit one outstanding operation per target and four new operations per owner each UTC day.

POST /api/v1/access-policy-canaries accepts only the empty JSON object {}.

Limit its decoded body to 256 bytes.

Use the section 24.1.5 per-owner and service-wide canary limits.

Both POST routes require the common CSRF and idempotency controls above.

They bind their idempotency records to the authenticated owner subject key, canonical body digest, route, and 24-hour expiry.

The same key and body return the same operation.

The same key with different bytes fails.

The portal requests its CSRF token through same-origin GET /api/v1/csrf.

That route requires a fresh portal-audience JWT, exact X-Firecall-Origin, and Sec-Fetch-Site: same-origin.

The browser sets X-Firecall-Origin to its current origin.

The broker rejects a missing, duplicate, or different value.

A cross-origin request with that header requires a preflight, which Firecall rejects.

When the browser also sends Origin, the broker requires the same exact portal origin.

The browser keeps the token only in memory and sends it as X-Firecall-CSRF.

The token is a compact JWS with exactly three unpadded base64url segments.

Its protected header contains only alg=ES256, a known kid, and typ=firecall-portal-csrf+jwt;v=1.

Its RFC 8785 payload contains only iss, aud, purpose, v, subject_key, host, origin, iat, exp, and a random 128-bit jti.

It binds the derived subject key, exact portal host, and exact portal Origin.

It expires after 15 minutes and permits two minutes of clock skew.

The complete token cannot exceed 4 KiB.

The parser rejects padding, duplicate keys, extra fields, JSON JWS serialization, unprotected fields, none, crit, b64=false, jku, jwk, x5u, and x5c.

It rejects a wrong key, type, issuer, audience, purpose, version, subject key, host, origin, time, or signature before mutation.

Use one error envelope:

{
  "error": {
    "code": "capacity_full",
    "message": "All 3 dev sessions are in use. Try again after one closes.",
    "request_id": "uuid-v4"
  }
}

Do not return internal exception text.

Retry preserves the request and session identifiers.

Allow it only after the recorded workflow is terminal and no capacity-ledger entry or runner status exists.

Reject retry when a close revocation exists.

Permit at most two manual retry generations for one receipt.

10.10 Workspace API

Expose these terminal-proxy routes:

Method Route Result
GET /api/v1/csrf Mint one short-lived workspace CSRF token for the current subject key and host
POST /api/v1/invitations Mint a random join capability for requester or owner
DELETE /api/v1/invitations/<id> Revoke one invitation for requester or owner
DELETE /api/v1/invitations Revoke all invitations for requester or owner
POST /api/v1/join-exchanges Exchange one fragment capability for pending collaborator authorization
GET /api/v1/join-context Return scope for the requester, an owner, or a pending collaborator
POST /api/v1/join-acknowledgements Record explicit scope confirmation and set the acknowledgement cookie
GET /api/v1/session Acknowledged session facts and expiry
WS /api/v1/presence One workspace-page presence connection
GET /api/v1/namespaces Profile namespace list
GET /api/v1/workloads Selected namespace workload summary
GET /api/v1/workloads/<kind>/<name> Fixed structured projection
GET /api/v1/events Selected namespace events
GET /api/v1/logs Bounded current or previous logs
WS /api/v1/logs/follow Bounded live logs
GET /api/v1/terminals Current tmux windows
POST /api/v1/terminals Create one allowed terminal window
DELETE /api/v1/terminals/<id> End one terminal window
WS /api/v1/terminals/<id>/stream PTY bytes and resize control
GET /api/v1/previews Current bounded previews
POST /api/v1/previews Create one validated preview
DELETE /api/v1/previews/<id> Stop one preview
POST /api/v1/previews/<id>/handoff Mint a subject-key-bound preview acknowledgement handoff
POST /.firecall/v1/access-policy-canary Consume one workspace-audience identity canary handoff

Apply these decoded request-body limits before JSON parsing:

Route class Maximum body
Invitation mint 2 KiB
Join exchange 1 KiB
Join acknowledgement 1 KiB
Terminal create 2 KiB
Preview create 4 KiB
Preview handoff 2 KiB
Every workspace DELETE 0 bytes

Reject an oversized declared Content-Length before reading the body.

Wrap every chunked or undeclared body in an enforcing maximum-byte reader.

Accept one JSON object with the exact media type.

Reject unknown fields, duplicate fields, trailing values, invalid UTF-8, and trailing non-whitespace bytes.

Return a fixed 413 or 400 response without echoing input.

Open pod shell uses the existing terminal routes.

It creates one normal shell window in the selected namespace.

The client then preloads this fixed command without a newline:

kubectl --namespace='sf-aigw' exec --stdin --tty 'pod/aigw-68cb67856f-7jhk5' --container='aigw' -- /bin/sh

The user must press Enter to execute it.

The UI shows Review this command, then press Enter. above the terminal.

The proxy first validates the exact Pod and container through the structured workload API.

The renderer accepts only Kubernetes-valid namespace, Pod, and container names.

It rejects control bytes, whitespace, quotes, option prefixes, separators, and noncanonical names.

It uses one fixed argument order and one fixed /bin/sh target.

It never accepts a command, shell path, flag, or argument from the browser.

The terminal API has no Pod-exec operation.

firecall-sessiond starts only the normal local shell and does not execute the Kubernetes command.

If /bin/sh is absent, the terminal shows the normal kubectl error.

The join-exchange body contains the raw capability and no other credential.

The proxy hashes it before lookup and never stores or logs the raw value.

The pending cookie is Secure, HttpOnly, SameSite Strict, host-only, scoped to /, and limited to five minutes.

The confirmation body contains accepted: true and the displayed profile digest.

The acknowledgement cookie is Secure, HttpOnly, SameSite Strict, host-only, and scoped to /.

Reject a request with duplicate pending, acknowledgement, or preview-acknowledgement cookie names.

Its expiry cannot exceed the session expiry.

The join-context route does not authorize a terminal or Kubernetes data route.

The trusted workspace shell remains available for known non-ready states.

It shows build, failure, close, closed, and expiry information without opening the data plane.

Every Kubernetes route requires one explicit namespace.

The proxy validates the namespace before it contacts the agent.

The agent validates it again before Kubernetes access.

The terminal stream uses binary frames for PTY bytes.

Use small JSON control frames for resize, ping, and close.

A resize frame contains only type="resize", cols, and rows.

cols and rows are base-10 JSON integers without a sign, fraction, exponent, duplicate key, or unknown field.

Require 1 <= cols <= 500, 1 <= rows <= 200, and checked cols * rows <= 100000.

Reject an overflow before allocation or system call.

Limit a control frame to 4 KiB.

Coalesce resize input to the newest valid frame once every 50 milliseconds.

Close the stream after more than 100 resize frames arrive in one second.

The box agent repeats the schema, range, product, and rate checks before it resizes the PTY.

Version the stream protocol independently from the HTTP API.

Use these structured-API bounds:

  • Default list page: 50 items.
  • Maximum list page: 200 items.
  • Maximum list response: 1 MiB.
  • Maximum detail response: 256 KiB.
  • Maximum event page: 200 events.
  • Maximum initial log result: 2,000 lines or 1 MiB.
  • Maximum log line: 64 KiB before a visible truncation marker.
  • Maximum follow buffer: 256 KiB per stream.

Use opaque signed continuation cursors with a ten-minute expiry.

Bind each cursor to session, namespace, resource query, and subject key.

Responses include truncated and next_cursor when applicable.

Close a slow follow stream with slow_consumer before its buffer exceeds the limit.

The logs page keeps the received tail in memory after slow_consumer.

It shows Live logs paused because this browser fell behind.

A Resume live logs action opens a new bounded stream for the same Pod and container.

The resumed stream starts with a fresh tail and does not claim that it recovered missing lines.

10.11 Agent API

The agent accepts only mutual-TLS proxy traffic.

It exposes one fixed idempotent session-revoke route only to the pinned proxy client identity.

That route accepts the configured session identifier and no other field.

It sends the fixed revoke operation to sessiond and returns no process, key, token, or command data.

It exposes no browser HTML.

It uses the narrow typed Kubernetes REST adapter from ADR-0015 for structured resource calls.

The adapter connects only to the box-local Kubernetes proxy.

It uses a fixed resource table and private typed response structures.

It never forwards a raw Kubernetes response.

It returns fixed fields for workload details.

It never returns arbitrary Kubernetes object YAML through the graphical API.

It does not build shell commands from API strings.

It uses fixed command templates only for terminal launchers.

It rejects:

  • An unknown namespace.
  • An unknown resource kind.
  • A Secret resource request.
  • An arbitrary command launcher.
  • An arbitrary preview host.
  • An arbitrary preview IP.
  • A preview target without an explicit http or https scheme.
  • An ExternalName Service.
  • A selectorless Service.
  • A Service with any stale, mixed, external, ambiguous, or unowned endpoint.
  • A target Pod that uses hostNetwork, hostPort, hostPath, or privileged mode.
  • A terminal count above the profile limit.
  • A preview count above the profile limit.

10.12 Mutual-TLS lifecycle

CA means certificate authority.

Use four fixed non-exportable Azure Key Vault key objects:

  • firecall-client-root-a.
  • firecall-client-root-b.
  • firecall-origin-root-a.
  • firecall-origin-root-b.

Each object uses an ECDSA P-256 key with only sign and verify operations.

Set exportable=false and create a new key version for each slot replacement.

Use distinct fixed subjects for client and origin roots.

No Terraform root owns a key object whose versions rotate at runtime.

Use the owner key-bootstrap path to create one disabled placeholder version for each fixed key name.

The placeholder is non-exportable, cannot sign while disabled, and never enters an active selector or public-root lock.

The bootstrap records only public-key metadata and version identifiers.

No private root key can enter Terraform state or bootstrap evidence.

Purge protection, the fixed inventory, and the absence of delete authority protect these objects.

Create a dedicated firecall-ca-roots-privileged.yml identity for later root versions.

Use firecall-ca-roots.yml as its public caller.

Bind its flexible OIDC credential to that exact infrastructure main reusable workflow.

Assign key create, get, sign, and verify only at the four exact key-object scopes.

Give it no Firecall application-secret read except the exact root-publication App key.

Give it no decrypt, unwrap, export, delete, purge, role-management, or session authority.

The only AKS exception is Azure Kubernetes Service Cluster User Role on the exact dev cluster.

Bind that identity to one firecall-edge Role through one exact RoleBinding.

Bind it to a second Role in firecall for only the named certificate-rotator Lease.

That Role permits get, update, and patch on only firecall-certificate-rotator.

The Role can get public root ConfigMaps and the named cloudflared Deployment.

It can get and list Pod metadata and status in firecall-edge for rollout proof.

It can read ReplicaSet metadata and Pod templates in firecall-edge.

It filters ReplicaSets by the exact cloudflared Deployment owner UID.

It cannot read Pod logs, Secrets, exec, attach, port-forward, create, delete, or access another namespace.

The workflow can create a new version in a fixed slot but cannot create a fifth key name.

Reviewed Go code constructs the root X.509 TBSCertificate bytes.

It sets a random positive 128-bit serial number and a 24-month validity.

It sets critical Basic Constraints to CA:TRUE with path length zero.

It sets critical Key Usage to keyCertSign, cRLSign, and digitalSignature only.

It copies the exact P-256 public key from the selected Key Vault key version.

It hashes the canonical DER TBSCertificate with SHA-256.

It sends only that digest to the exact Key Vault ES256 Sign API.

It converts the returned ECDSA signature to the ASN.1 form required by X.509.

It assembles and self-verifies the final certificate through Go crypto/x509.

It validates subject, issuer, curve, key version, Basic Constraints, usage, validity, and self-signature.

It writes only public root material to source and public trust storage.

The workflow never writes directly to infrastructure main.

It uses the Firecall root-publication GitHub App to create one candidate branch and pull request.

That App is installed only on OpenMined/infrastructure.

Its only permissions are Metadata read, Contents read and write, and Pull requests read and write.

Branch protection blocks the App from approving or merging its pull request.

The pull request contains only public roots, their lock, and a public Sigstore bundle.

After merge, the protected caller verifies the exact merged files before immutable Blob publication.

The CA-root Azure identity can read only this App's exact Key Vault private-key secret.

Store reviewed public files under:

platform/firecall/trust/client-root-a.pem
platform/firecall/trust/client-root-b.pem
platform/firecall/trust/origin-root-a.pem
platform/firecall/trust/origin-root-b.pem
platform/firecall/trust/roots.lock.json

The lock records one strictly increasing generation, exact Key Vault key-version IDs, DER SHA-256 digests, validity, serials, slot state, and generation workflow run.

Publish the same DER files and a signed lock to immutable firecall-root-trust Blob paths.

Give the CA-root identity read and create-only data authority on that exact container.

Give it no list, overwrite, delete, lease, lifecycle, policy, or account authority.

Use these versioned create-once path shapes:

roots/<client|origin>/<a|b>/<key-version>/<der-sha256>.der
locks/<source-commit>/<lock-sha256>.json
bundles/<source-commit>/<sigstore-bundle-sha256>.json

Every write uses If-None-Match: *.

The container immutability policy prevents later overwrite or deletion.

The workflow rereads each exact object and verifies content length, ETag, digest, and immutability before success.

The root workflow signs the lock through keyless Sigstore identity on the exact protected workflow and commit.

The control runner verifies that signature, workflow identity, commit, version IDs, and every DER digest offline.

The control profile pins the accepted client-root bundle digest.

The control repository does not copy or independently own the root certificates.

The VM image build copies both reviewed client roots and verifies roots.lock.json.

Create one immutable ConfigMap named firecall-client-roots-<bundle_sha256_12> for each reviewed client bundle.

That ConfigMap contains only the active and standby client roots, full bundle digest, and root-lock digest.

Mount the exact digest-named ConfigMap read-only into broker, proxy, readiness worker, and certificate-rotator Pods.

The broker uses it to verify proxy-signed immediate-revocation receipts.

The proxy uses it to verify proxy-client, readiness-worker, and broker-revoker identities on their fixed routes.

The certificate rotator uses it for candidate validation.

Each Deployment Pod template carries firecall.openmined.org/client-root-bundle-sha256.

Every consumer validates the mounted digest and exact two-root inventory before readiness.

Give the ConfigMap an earlier Argo wave than every consumer.

Roll and prove every consumer on a new client bundle before any leaf uses its new root.

Keep the old client ConfigMap until no current Pod, Job, CronJob template, or rollback ReplicaSet references it.

Remove it only in a later pull request with zero-reference evidence.

Create one immutable ConfigMap named firecall-origin-roots-<bundle_sha256_12> for each reviewed bundle.

That ConfigMap contains only the active and standby origin roots, full digest, and lock digest.

cloudflared mounts the exact digest-named ConfigMap as its caPool file.

The same reviewed pull request adds the new ConfigMap and changes the Deployment volume reference and expected digest.

Give the ConfigMap an earlier Argo sync wave than the Deployment.

Argo must make that ConfigMap Healthy before it starts the Deployment wave.

The old Pods keep their immutable old ConfigMap mount during the rolling update.

Keep the old ConfigMap until no Pod or ReplicaSet references it.

Remove it in a later reviewed pull request after zero-reference proof.

That proof checks every owned ReplicaSet Pod template, including scaled-to-zero rollback revisions.

Add one digest-pinned firecall-cloudflared-health sidecar to each connector Pod.

Implement it under platform/firecall/cloudflared-health/.

The sidecar mounts the same caPool file read-only.

GitOps sets firecall.openmined.org/origin-root-bundle-sha256 on the Deployment Pod template.

The sidecar reads that expected digest through the downward API.

Its readiness check requires:

  • The mounted file digest equals the expected digest.
  • The bundle contains exactly the reviewed active and standby roots.
  • TLS succeeds to firecall-broker.firecall.svc.cluster.local:8443 with the exact broker name.
  • TLS succeeds to firecall-proxy.firecall.svc.cluster.local:8443 with the exact proxy name.
  • The local stock cloudflared readiness endpoint reports one healthy tunnel connection.

Disable ServiceAccount-token automount for the complete connector Pod.

The sidecar reads connection health from local /diag/tunnel.

Expose only one local readiness endpoint to kubelet.

Configure the sidecar container readiness probe to call that endpoint.

The Pod becomes ready only when both cloudflared and sidecar container probes pass.

The sidecar has no ServiceAccount token, tunnel token, Key Vault access, Secret read, or Kubernetes API access.

It returns no certificate bytes, token, account ID, or routed request data.

Kubelet marks the connector ready only when every local check succeeds.

The CA-root verifier can read only the named Deployment, root ConfigMap, and Pod metadata and status in firecall-edge.

It cannot read logs, Secrets, exec, attach, port-forward, patch, or delete.

Qualification requires the expected digest on the current Pod template.

It also requires exactly two updated and ready Pods from that Deployment revision.

This Pod readiness is the origin-bundle proof.

The first bootstrap order is:

  1. Create all four root key objects with disabled placeholders through the owner key-bootstrap path.
  2. Construct, sign, validate, and publish only the four public certificates.
  3. Merge the reviewed public roots and exact lock.
  4. Build and seal the VM image with both client roots.
  5. Deploy the immutable origin bundle, consumers, rotator, and candidate health Services.
  6. Set publishNotReadyAddresses: true only on those candidate health Services.
  7. Keep the normal broker and proxy Services limited to ready endpoints.
  8. Issue every inactive broker, proxy, proxy-client, broker-revoker, and readiness leaf under its assigned root A.
  9. Reach each unready consumer through its candidate health Service.
  10. Verify every inactive leaf and select the active epoch.
  11. Prove normal broker and proxy origin TLS now works.
  12. Require two ready connectors and the exact origin-bundle proof.

Rotate an active root after 12 months.

Alert 180, 90, and 30 days before any root expires.

The first A to B cycle uses the complete [A,B] bundle from initial activation.

Later bundle-replacement steps require the Pod-owned handoff contract defined below.

Client-root rotation uses this order:

  1. Publish the standby public root and signed lock before a leaf uses it.
  2. Create the immutable client-root ConfigMap and update every cluster verifier reference.
  3. Roll broker, proxy, and readiness consumers and prove the exact new bundle digest.
  4. Prove a certificate-rotator candidate Pod with the same bundle.
  5. Build and seal an image that contains active and standby client roots.
  6. Pin that image in the control profile.
  7. Issue proxy-client, readiness-worker, and broker-revoker leaves under the standby root.
  8. Prove every current and previous leaf slot through its real route.
  9. Select the new leaves and root for new sessions.
  10. Wait until every old-root session and old leaf ends.
  11. Replace the retired root slot only after cluster and image zero-use proof.

Origin-root rotation uses this order:

  1. Add a new immutable digest-named ConfigMap with the standby public root.
  2. Change the volume reference and expected digest in the same reviewed pull request.
  3. Apply the ConfigMap wave before the Deployment wave.
  4. Roll both cloudflared replicas and prove the new bundle digest.
  5. Issue inactive broker and proxy origin leaves under the standby root.
  6. Hot-reload both origin consumers and verify each through cloudflared.
  7. Select the new leaves only after those tests pass.
  8. Keep the old root through the complete old-leaf validity window.
  9. Remove it in a new digest-named bundle only after no serving leaf chains to it.

Never switch an origin leaf before both cloudflared replicas trust its root.

Never remove a client root while a sealed allowed image or live session needs it.

The initial activation publishes the complete [A,B] client and origin bundles.

It supports the first A to B signer rotation without a bundle replacement.

Do not replace [A,B] with [B,C] through the initial activation contract.

That later replacement requires a reviewed Pod-owned bundle-digest handoff.

Each new Pod must bind its immutable mounted bundle through an exact template annotation and downward-API file.

The Pod must prove that every selected or accepted leaf chains to one of its two mounted roots.

The certificate rotator must prove the mounted digest on every Pod in the complete current ReplicaSet.

Only then can it move the selector bundle digest and rotate leaves to the new signer.

Keep the old bundle until zero Pods, Jobs, CronJobs, rollback ReplicaSets, images, and sessions reference it.

Root retirement uses this barrier:

  1. Publish a signed slot state of retiring_no_new_leaves in the root lock.
  2. Keep that state as a persistent gate through the complete retirement.
  3. Require every rotator replica to report the exact new lock digest.
  4. Acquire the exact firecall-certificate-rotator Lease after its current holder exits.
  5. Repeat serving-leaf, public-leaf, allowed-image, and live-session zero-use proof.
  6. Publish a signed retirement-proof object and release the Lease.
  7. Open a later removal pull request that pins that proof and lock digest.
  8. Merge and roll that removal only while the persistent gate still matches.
  9. Verify every consumer, connector, Pod, and rollback ReplicaSet after the rollout.

A root cannot move directly from active to removed.

The rotator verifies the signed slot state before every Key Vault signing request.

No human review or Argo rollout occurs while a short Kubernetes Lease is held.

Implement one dedicated Go rotator under platform/firecall/certificate-rotator/.

Build it as a separate digest-pinned container.

Run it as a Kubernetes CronJob every six hours with concurrencyPolicy: Forbid.

The CronJob uses one Kubernetes Lease, a bounded deadline, and idempotent rotation epochs.

Its fixed manifest defines only these leaf identities:

  • Proxy client.
  • Readiness worker client.
  • Broker revoker client.
  • Broker origin server.
  • Proxy origin server.

The rotator accepts no caller-supplied subject, SAN, key usage, Service name, Secret name, or lifetime.

GitOps pre-creates fixed primary and secondary Secret slots and one fixed active-epoch ConfigMap.

Narrow RBAC permits only get and update on those named Secret and selector objects.

It cannot patch a Deployment, execute in a Pod, or create another Secret.

Each Go consumer watches both mounted slots and the active epoch.

Each consumer validates a matching key, certificate, identity, chain, and validity before atomic in-process reload.

The rotator verifies the new leaf through a fixed internal health endpoint before it changes the active epoch.

A failed verification leaves the prior slot active and creates an alert.

Create fixed certificate-health Services for this verification:

Consumer Service port Fixed test
Broker 9444 Serve the inactive origin leaf for full TLS validation
Proxy 9444 Serve the inactive origin leaf for full TLS validation
Proxy 9445 Use the inactive proxy client leaf against the fixed verifier
Readiness worker 9445 Use the inactive worker leaf against the internal proxy probe Service
Broker 9445 Use the inactive broker-revoker leaf against the internal proxy revoke Service without a mutation
Proxy verifier listener 9446 Validate client root, URI identity, SPKI, time, and chain

These Services are headless, cluster-internal Services and have no Cloudflare route.

Set clusterIP: None and publishNotReadyAddresses: true on only these candidate Services.

Use separate candidate probe and revocation peer Services while normal proxy Services exclude unready Pods.

Port 9446 is a fixed listener in the existing proxy binary.

It adds no Deployment, ServiceAccount, credential, or new trust holder.

The health APIs accept only fixed primary or secondary slot actions.

They accept no certificate, key, subject, SAN, destination, or command from the request.

They return only Pod UID, Pod name, slot, kid, public digest, validation state, and time.

The rotator gets the exact broker, proxy, and readiness Deployments.

It lists EndpointSlices for the five exact rotator-facing candidate Services.

It validates each EndpointSlice Service label, Service owner UID, endpoint address, and Pod target UID.

It dials each endpoint address with the fixed Service TLS name.

It compares the returned Pod UID with the EndpointSlice target UID.

It requires every desired, non-terminating replica UID exactly once.

It rejects a missing, duplicate, stale, foreign, or extra replica.

The rotator waits until every broker, proxy, or readiness replica reports the same candidate digest.

It also requires successful proxy-client and readiness-worker network handshakes.

It also requires a successful broker-revoker handshake against the internal proxy revoke Service.

Only then can it update the active epoch.

NetworkPolicy permits rotator egress only to broker and proxy TCP 9444 and consumer TCP 9445 Services.

It permits proxy egress to the verifier on TCP 9446.

It permits readiness-worker egress to the existing internal proxy probe Service.

The matching ingress policies accept only protected Firecall component labels.

Admission binds each protected label to its exact namespace, ServiceAccount, and image digest.

No certificate-health Service accepts traffic from a tenant, preview, or interactive box.

The image-pin files include the rotator container digest.

The deployment overlay owns its CronJob, workload identity, Secret slots, ConfigMap, RBAC, and expiry alerts.

The exact cluster-owner bootstrap owns the certificate-rotator Lease existence.

Its read-only Kubernetes RBAC can get the three named Deployments and five named candidate Services.

It can list EndpointSlice and Pod metadata only in firecall.

Kubernetes RBAC cannot restrict a list operation by Service label.

The rotator therefore rejects every EndpointSlice outside the five exact Service UIDs.

The certificate rotator generates only fixed Firecall certificate identities.

Key Vault signing permission cannot itself constrain certificate SAN values.

The image contains active and standby client CA public certificates only.

The proxy client leaf uses the URI identity spiffe://firecall/proxy.

The broker revoker leaf uses the URI identity spiffe://firecall/broker-revoker.

The proxy accepts that identity only on the internal revocation listener.

It never accepts the broker revoker on a workspace, preview, probe, certificate-health, or box-agent route.

Each leaf lasts 30 days and rotates after 15 days.

The proxy mounts the current and previous client leaf keys through separate Kubernetes Secrets.

The certificate rotator publishes current and previous public proxy leaf certificates to two fixed slots in firecall-trust.

It also creates one immutable evidence object per rotation in firecall-trust-evidence.

It publishes one ETag-protected active-leaf record with each slot, evidence path, digest, and rotation epoch.

The runner reads the active record and both exact fixed public-leaf slots before capacity reservation.

It reads only the root objects, lock, and bundle pinned by the protected profile.

It validates the CA chain, URI identity, validity, rotation epoch, and certificate digest.

It requires validity through the requested session expiry plus one hour.

It then derives the exact SPKI fingerprint.

It binds that public fingerprint, the client-root bundle digest, root-lock digest, and root-lock generation into the signed session-bootstrap payload.

The proxy selects the mounted private leaf whose SPKI matches the session status.

It fails closed when neither mounted leaf matches.

New sessions record the selected leaf SPKI fingerprint as proxy_spki_sha256.

SPKI means the public-key part of a certificate.

The agent requires the CA, URI identity, and recorded SPKI fingerprint.

At boot, it verifies the image-pinned client roots, bundle digest, root-lock digest, and generation against the signed bootstrap payload.

The eight-hour session ceiling permits safe current-and-previous overlap.

Each box generates its own server key at first boot.

The box creates a self-signed ECDSA P-256 server certificate.

Its DNS SAN is <session_id>.firecall.internal.

Its validity ends no later than one hour after the session expiry.

Azure VM Run Command retrieves only the public certificate.

The runner validates the PEM, self-signature, SAN, validity, and identifier.

It publishes the lowercase SHA-256 hash of the exact DER certificate in status.

The proxy uses a dedicated TLS verification callback for this self-signed leaf.

It verifies time, self-signature, DNS SAN, session identifier, and exact DER fingerprint.

It never falls back to a system CA or a generic certificate-skip option.

No server private key enters cloud-init, Terraform state, status, or GitHub logs.

A root boot helper creates the server key and agent configuration.

Store them under /etc/firecall-agent with root ownership and group access for firecall-agent only.

Run the network agent as a dedicated firecall-agent service user.

The interactive firecall shell user cannot read or replace keys, CA roots, proxy pins, expiry, or agent configuration.

The root firecall-sessiond boundary starts only reviewed fixed processes as the firecall user.

Rotation tests must prove current, previous, wrong-identity, expired, and compromised-leaf incident behavior.

The readiness worker uses a separate client-root-signed ECDSA P-256 client certificate.

Its URI identity is spiffe://firecall/readiness-worker.

The internal probe Service accepts only the current or previous exact worker SPKI.

It also verifies the reviewed client CA chain, time, key usage, URI identity, and selected root-lock digest.

The worker certificate lasts 30 days and rotates after 15 days.

The rotator publishes the current and previous worker chains and SPKI fingerprints into the proxy probe configuration atomically.

The readiness worker mounts only its two fixed client-root-signed leaf Secret slots and the immutable client-root bundle.

The proxy probe verifier mounts the same immutable client-root bundle.

Client-root rotation rolls and proves both verifier and worker before a worker leaf uses the new root.

The proxy client certificate does not authenticate to the internal probe Service.

The worker certificate does not authenticate to a box agent.

Client CA rotation follows the exact root order above.

If a proxy leaf private key leaks, immediately set creation_enabled=false and block every new reservation.

List every nonterminal session whose proxy_spki_sha256 equals the compromised SPKI.

Close each affected session and verify its agent listener stops before rotation completes.

If normal close fails, use the reduction-only path to remove its allow-agent rule and stop or delete its VM.

Rotate the proxy leaf, remove the compromised leaf from both mounted slots, and prove the replacement on every proxy replica.

Resume creation only after no live session pins that SPKI and direct use of the stolen leaf fails everywhere.

There is no CRL, OCSP, or agent-side deny list for a proxy leaf.

An active box accepts its pinned stolen leaf until its agent stops, its network rule closes, or its session expires.

The incident record measures this residual window from detection through the last verified denial.

If a broker or proxy origin leaf private key leaks, immediately disable creation and freeze ordinary certificate rotation.

Issue replacement broker and proxy leaves under the uncompromised standby origin root.

Roll every broker and proxy replica, then prove each origin through the fixed candidate Services.

Remove the compromised root from the cloudflared CA bundle and roll both tunnel connectors.

Prove the stolen leaf fails through each connector before creation resumes.

Publish a new standby root only after the incident completes.

Ordinary origin-leaf rotation does not revoke a stolen valid leaf because cloudflared trusts its signing root.

Treat compromise of the CA-root workflow, certificate rotator, or one authorized CA key version as a root incident.

Immediately disable creation and revoke both signing identities, federated credentials, workload bindings, and Key Vault assignments.

Revoke both root-publication GitHub App keys and disable both selected Key Vault App-key versions.

Create replacement App keys through the exact manual credential JIT path in section 24.10.

Prove an old root-publication App key cannot mint an installation token before source publication resumes.

Disable every Key Vault version that the compromised identity could use.

Wait for all issued Azure and Kubernetes tokens to expire before recovery signing starts.

Disabling a Key Vault version does not revoke a rogue leaf that an accepted root already signed.

Restore only a new commit-pinned CA-root identity from reviewed trust state after that token-expiry proof.

Prove every revoked identity and old workflow pin still fails to sign.

For affected client roots, close every session whose image trusts any affected root.

Verify each agent stops or loses its 8443 path.

Create replacement root versions through the owner recovery path and publish their reviewed public certificates and lock.

Build, scan, seal, and pin a replacement VM image that contains no affected client root.

Issue new proxy leaves only under the replacement client trust.

For affected origin roots, issue new broker and proxy leaves under replacement origin trust.

Roll both origin consumers, remove every affected root from the immutable cloudflared bundle, and roll both connectors.

Restore the workload certificate rotator only after replacement trust reaches every image and connector.

Resume creation only after direct rogue client-leaf and origin-leaf tests fail on every route.

Record the residual window from first possible rogue signature through the final affected-root removal.

10.13 GitHub App permissions

Install one broker GitHub App on OpenMined/firecall-control only.

Use these repository permissions:

  • Actions: read and write.
  • Contents: read.
  • Metadata: read.

Grant no issue, pull-request, administration, secret, or organization permission.

GitHub does not scope Actions write to one workflow.

A stolen installation token can disable, enable, cancel, re-run, or delete any control workflow run.

It can also delete control run logs and artifacts.

It cannot change protected source, but it can deny close, sweep, and break-glass automation.

Keep installation tokens in memory only.

Do not log token-bearing GitHub requests.

Install one separate image-pin GitHub App on these repositories only:

  • OpenMined/infrastructure.
  • OpenMined/firecall-control.

Give the image-pin App these permissions:

  • Contents: read and write.
  • Pull requests: read and write.
  • Metadata: read.

The image workflow uses it only to open reviewed image-pin pull requests.

The infrastructure pull request updates one closed-set manifest for every Firecall Kubernetes image.

It includes broker, proxy, egress, DNS proxy, readiness, both rotators, binding validator, health sidecar, and cloudflared.

It includes every declared container and init-container and permits no unlisted sidecar.

The control pull request pins the exact Compute Gallery VM image version.

Branch protection prevents the App from merging that pull request.

The broker App receives no content-write or pull-request permission.

Install one root-publication GitHub App on OpenMined/infrastructure only.

Give it Metadata read, Contents read and write, and Pull requests read and write.

Use it only for public-root branches and pull requests.

Branch protection prevents its approval, direct main write, or merge.

Install one Classic-observer GitHub App on OpenMined/infrastructure only.

Give it Actions read, Contents read, and Metadata read.

Grant it no Actions write, Contents write, pull-request, administration, secret, or organization permission.

The capacity reconciler uses it only for Classic source and workflow-run proof.

Install one separate control-observer GitHub App on OpenMined/firecall-control only.

Give it Actions read, Contents read, and Metadata read.

Grant it no Actions write, Contents write, pull-request, administration, secret, or organization permission.

Install one settings-observer GitHub App on OpenMined/infrastructure only.

Give it Metadata read, Contents read, Actions read, and Administration read.

Grant it no Actions, Contents, pull-request, deployment, secret, administration, or organization write permission.

Its canary verifies the exact installation and all four declared read permissions.

It proves every repository write, secret endpoint, organization endpoint, and uninstalled-repository request fails.

Run firecall-control-monitor.yml from the infrastructure repository every five minutes.

The monitor verifies each required control workflow file, protected commit, enabled state, recent run record, and expected evidence presence.

It alerts on a disabled workflow, an expected or previously observed deleted run, a premature evidence deletion, or an unexpected workflow identity.

The broker App cannot disable this external workflow or use the observer App.

Its durable monotonic checkpoint records each observed workflow run number, run ID, and evidence-retention deadline.

A request receipt or lifecycle record supplies every expected broker-dispatched run ID.

A run deleted before the next poll is detected from its expected receipt or a later run-number gap.

An unobserved manual run with no receipt and no later run-number gap is not detectable through repository read APIs.

The monitor detects covered GitHub evidence deletion but cannot restore deleted GitHub data.

Immutable Azure lifecycle audit remains the authoritative Firecall evidence.

For suspected broker App compromise, an App owner suspends the installation and revokes both App keys immediately.

The owner restores required workflow enabled states from protected source and verifies them through the observer App.

Close and sweep use reduction-only operator paths until the broker App receives a new proved key.

11. Repository placement

11.1 Infrastructure repository

Add or update:

README.md
CLAUDE.md
docs/ledger.md
docs/adr/0001-platform-monorepo-structure.md
docs/adr/0002-delivery-architecture.md
docs/adr/0012-firecall-access-broker.md
docs/architecture/platform.md
docs/architecture/firecall-v2-implementation-plan.md
docs/architecture/firecall-v2-assurance-model.md
docs/architecture/firecall-v2-threat-model.md
docs/architecture/firecall-v2-migration.md
docs/conventions.md
docs/runbooks/firecall.md
docs/runbooks/firecall-v2-workspace.md
docs/runbooks/onboard-a-repo.md
docs/runbooks/firecall-v2-requester.md
docs/runbooks/firecall-v2-operator.md
docs/runbooks/firecall-v2-rollback.md
docs/runbooks/firecall-v2-image-update.md
docs/runbooks/firecall-v2-credential-rotation.md
.github/workflows/firecall-runtime-validate.yml
.github/workflows/firecall-images.yml
.github/workflows/firecall-images-privileged.yml
.github/workflows/firecall-image-seal.yml
.github/workflows/firecall-image-seal-privileged.yml
.github/workflows/firecall-image-seal-recovery-privileged.yml
.github/workflows/firecall-image-pin.yml
.github/workflows/firecall-image-pin-privileged.yml
.github/workflows/firecall-image-pin-continuation.yml
.github/workflows/firecall-image-pin-continuation-privileged.yml
.github/workflows/firecall-dev-terraform.yml
.github/workflows/firecall-dev-images.yml
.github/workflows/firecall-standing.yml
.github/workflows/firecall-standing-privileged.yml
.github/workflows/firecall-trust.yml
.github/workflows/firecall-trust-privileged.yml
.github/workflows/firecall-trust-drift.yml
.github/workflows/firecall-trust-drift-schedule-privileged.yml
.github/workflows/firecall-trust-drift-manual-privileged.yml
.github/workflows/firecall-ca-roots.yml
.github/workflows/firecall-ca-roots-privileged.yml
.github/workflows/firecall-credentials.yml
.github/workflows/firecall-credentials-privileged.yml
.github/workflows/firecall-policy.yml
.github/workflows/firecall-policy-privileged.yml
.github/workflows/firecall-cloudflare.yml
.github/workflows/firecall-cloudflare-privileged.yml
.github/workflows/firecall-cloudflare-drift.yml
.github/workflows/firecall-cloudflare-drift-privileged.yml
.github/workflows/firecall-zone-policy.yml
.github/workflows/firecall-zone-policy-privileged.yml
.github/workflows/firecall-preflight.yml
.github/workflows/firecall-preflight-privileged.yml
.github/workflows/firecall-activation-verify.yml
.github/workflows/firecall-activation-verify-privileged.yml
.github/workflows/firecall-load-qualification.yml
.github/workflows/firecall-load-qualification-privileged.yml
.github/workflows/firecall-control-monitor.yml
.github/workflows/firecall-control-monitor-schedule-privileged.yml
.github/workflows/firecall-control-monitor-manual-privileged.yml
.github/workflows/firecall-rescan.yml
.github/workflows/firecall-rescan-schedule-privileged.yml
.github/workflows/firecall-rescan-manual-privileged.yml
.github/workflows/firecall-scanner-db-update.yml
.github/workflows/firecall-scanner-db-update-schedule-privileged.yml
.github/workflows/firecall-scanner-db-update-manual-privileged.yml
.github/workflows/firecall-tool-acquire.yml
.github/workflows/firecall-tool-acquire-push-privileged.yml
.github/workflows/firecall-tool-acquire-manual-privileged.yml
.github/workflows/firecall-retention.yml
.github/workflows/firecall-retention-schedule-privileged.yml
.github/workflows/firecall-retention-recovery-privileged.yml
.github/workflows/firecall-image-idle.yml
.github/workflows/firecall-image-idle-privileged.yml
.github/workflows/firecall-image-inventory.yml
.github/workflows/firecall-image-inventory-privileged.yml
.github/workflows/firecall-node-tuple-monitor.yml
.github/workflows/firecall-node-tuple-monitor-schedule-privileged.yml
.github/workflows/firecall-node-tuple-monitor-manual-privileged.yml
.github/workflows/firecall-classic-reduction.yml
.github/workflows/firecall-classic-reduction-schedule-privileged.yml
.github/workflows/firecall-classic-reduction-manual-privileged.yml
.github/workflows/firecall-classic-auth-denial.yml
.github/workflows/images.yml
.github/workflows/talks-site.yml
.github/workflows/talks-site-publish.yml
.github/workflows/validate.yml
.github/workflows/terraform.yml
platform/talks-site/tests/test_delivery.py
scripts/ci/firecall-runtime-validate.sh
scripts/ci/firecall-go-validate.sh
scripts/ci/firecall-changed-paths.py
scripts/ci/test_firecall_changed_paths.py
scripts/ci/firecall-build-vm-components.py
scripts/ci/firecall-dev-vm-inputs.py
scripts/ci/firecall-dev-vm-cloud.py
scripts/ci/firecall-packer-execution-authority.py
scripts/ci/firecall-prepare-vm-scanners.py
scripts/ci/test_firecall_prepare_vm_workflow.py
.github/workflows/firecall-dev-vm.yml
scripts/ci/firecall-prepare-vm-inputs.py
scripts/ci/firecall-prepare-vm-material.py
scripts/ci/firecall-prepare-scanner-runtime.py
scripts/ci/test_firecall_prepare_vm_inputs.py
scripts/ci/test_firecall_prepare_vm_material.py
scripts/ci/test_firecall_prepare_scanner_runtime.py
scripts/ci/firecall-images.sh
scripts/ci/firecall-image-build-operation.sh
scripts/ci/firecall-image-seal.sh
scripts/ci/firecall-image-pin.sh
scripts/ci/firecall-image-trust-verdict.sh
scripts/ci/firecall-gallery-version.sh
scripts/ci/firecall-pin-images.py
scripts/ci/firecall-session-permissions-test.py
scripts/ci/firecall-pin-images-test.sh
scripts/ci/firecall-strict-json.py
scripts/ci/firecall-standing.sh
scripts/ci/firecall-admission-interface-test.sh
scripts/ci/firecall-cloudflare-interface-test.sh
scripts/ci/firecall-approval-receipt-test.sh
scripts/ci/firecall-approval-receipt.sh
scripts/ci/firecall-live-environment-test.sh
scripts/ci/firecall-live-environment.sh
scripts/ci/firecall-protected-source-test.sh
scripts/ci/firecall-protected-source.sh
scripts/ci/firecall-trust.sh
scripts/ci/firecall-trust-drift.sh
scripts/ci/firecall-ca-roots.sh
scripts/ci/firecall-credentials.sh
scripts/ci/firecall-policy.sh
scripts/ci/firecall-cloudflare.sh
scripts/ci/firecall-cloudflare-drift.sh
scripts/ci/firecall-zone-policy.sh
scripts/ci/firecall-preflight.sh
scripts/ci/firecall-activation-verify.sh
scripts/ci/firecall-load-qualification.sh
scripts/ci/firecall-control-monitor.sh
scripts/ci/firecall-rescan.sh
scripts/ci/firecall-scanner-db-update.sh
scripts/ci/firecall-tool-acquire.sh
scripts/ci/firecall-retention.sh
scripts/ci/firecall-image-idle.sh
scripts/ci/firecall-image-inventory.sh
scripts/ci/firecall-node-tuple-monitor.sh
scripts/ci/firecall-classic-reduction.sh
scripts/ci/firecall-classic-auth-denial.sh
scripts/ci/firecall.sh
scripts/ci/firecall-jsonschema-reference.py
scripts/ci/firecall-reference-requirements.txt
scripts/ci/firecall-validate.sh
scripts/ci/firecall-flexible-fic.sh
scripts/ci/tf-stack-check.sh
scripts/ci/control-drift-expected.yaml
scripts/ops/firecall-owner-operation.sh
scripts/ops/firecall-control-bootstrap.sh
scripts/ops/firecall-control-seed/
scripts/ops/firecall-trust-bootstrap.sh
scripts/ops/firecall-standing-bootstrap.sh
scripts/ops/firecall-image-orphan-delete.sh
scripts/ops/firecall-control-monitor-bootstrap.sh
scripts/ops/firecall-key-bootstrap.sh
scripts/ops/firecall-data-bootstrap.sh
scripts/ops/firecall-manual-secret-slot.sh
scripts/ops/firecall-broker-app-keys.sh
scripts/ops/tests/test_firecall_broker_app_keys.py
scripts/ops/firecall-pim-bootstrap.sh
scripts/ops/firecall-pim-ca-bootstrap.sh
scripts/ops/firecall-conditional-access-bootstrap.sh
scripts/ops/firecall-cloudflare-idp-bootstrap.sh
scripts/ops/firecall-cloudflare-identity-canary.sh
scripts/ops/firecall-pre-epoch-token-canary.sh
scripts/ops/firecall-notification-verify.sh
scripts/ops/firecall-trust-observer-bootstrap.sh
scripts/ops/firecall-identity-reinstate.sh
platform/firecall/README.md
platform/firecall/v1-handover.md
platform/docs-site/mkdocs.yml
platform/firecall/activation-evidence/firecall-state-bootstrap/3f497057d7f56506c9def5bd17993f6531b1bb15fe97ad95f861694219b4d3d3.json
platform/firecall/profiles/dev-fusion-debug.yaml
platform/firecall/profiles/dev-namespace-debug.yaml
platform/firecall/contracts/
platform/firecall/owner-bootstrap-evidence/
platform/firecall/owner-bootstrap-readbacks/
platform/firecall/contracts/data-bootstrap.json
platform/firecall/contracts/manual-secret-slots.json
platform/firecall/contracts/pim-jit.json
platform/firecall/contracts/conditional-access.json
platform/firecall/contracts/cloudflare-entra-idp.json
platform/firecall/contracts/cloudflare-identity-canary.json
platform/firecall/contracts/notification-targets.json
platform/firecall/contracts/aks-connection.json
platform/firecall/contracts/assurance/firecall-live-configuration.json
platform/firecall/contracts/assurance/firecall-live-environment.json
platform/firecall/contracts/assurance/binding-admission-proof-selection.json
platform/firecall/contracts/assurance/binding-admission-canary.json
platform/firecall/contracts/cloudflare-cache-phase-imports.json
platform/firecall/contracts/node-tuples.json
platform/firecall/classic-cutover.json
platform/firecall/go.mod
platform/firecall/go.sum
platform/firecall/internal/
platform/firecall/vendor/
platform/firecall/integration/
platform/firecall/licenses/
platform/firecall/evidence/
platform/firecall/broker/
platform/firecall/terminal-proxy/
platform/firecall/egress/
platform/firecall/dns-policy-proxy/
platform/firecall/node-binding-validator/
platform/firecall/node-binding-validator/cmd/firecall-node-binding-validator/lease_activation_gate.go
platform/firecall/node-binding-validator/cmd/firecall-node-binding-validator/lease_activation_gate_test.go
platform/firecall/node-binding-validator/cmd/firecall-node-binding-validator/lease_activation_receipt.schema.json
platform/firecall/node-binding-validator/cmd/firecall-node-binding-validator/validation-manifest.json
platform/firecall/readiness-probe/
platform/firecall/browser/
platform/firecall/browser-token-helper/
platform/firecall/certificate-rotator/
platform/firecall/credential-rotator/
platform/firecall/cloudflared-health/
platform/firecall/cloudflared-mirror/
platform/firecall/agent/
platform/firecall/sessiond/
platform/firecall/containers/
platform/firecall/web/
platform/firecall/image/
platform/firecall/release-evidence/
platform/firecall/security-exceptions/
platform/firecall/toolchain/
platform/firecall/packer/
platform/firecall/scanners/
platform/firecall/web/pnpm-store.lock.json
platform/firecall/trust/
platform/firecall/trust/sigstore/
kubernetes/apps/firecall/base/
kubernetes/apps/firecall/compositions/
kubernetes/apps/firecall/compositions/dev-activation-post-prepared/
kubernetes/apps/firecall/dev-evidence-reference.json
kubernetes/apps/firecall/dev-validator-evidence-reference.json
kubernetes/apps/firecall/dev-rotator-evidence-reference.json
kubernetes/apps/firecall/overlays/dev/
kubernetes/apps/firecall/overlays/dev/activation/post-prepared/
kubernetes/apps/firecall-edge/base/
kubernetes/apps/firecall-edge/compositions/
kubernetes/apps/firecall-edge/overlays/dev/
kubernetes/bootstrap/argocd/kustomization.yaml
kubernetes/bootstrap/README.md
kubernetes/bootstrap/firecall-state/
kubernetes/appsets/firecall-apps.yaml
kubernetes/appsets/kustomization.yaml
kubernetes/clusters/aks-dev/apps/firecall.yaml
kubernetes/clusters/aks-dev/apps/firecall-edge.yaml
kubernetes/infrastructure/signoz/overlays/dev/netpol.yaml
scripts/ci/firecall-state-bootstrap-test.sh
scripts/ci/k8s-validate.sh
terraform/stacks/azure/firecall-control/dev/
terraform/stacks/azure/firecall-aks/dev/
terraform/stacks/azure/firecall-trust/prod/
terraform/stacks/azure/firecall-trust/prod/firecall-federated-credentials/
terraform/stacks/azure/firecall-policy-definition/prod/
terraform/stacks/azure/firecall-image-build-policy/prod/
terraform/stacks/azure/firecall-budget/prod/
terraform/stacks/azure/firecall-policy/dev/
terraform/README.md
terraform/stacks/azure/acr/README.md
terraform/stacks/azure/acr-preview/preview/outputs.tf
terraform/stacks/azure/aks-platform/dev/main.tf
terraform/stacks/azure/ci-identities/README.md
terraform/stacks/azure/ci-identities/
terraform/stacks/azure/dns/README.md
terraform/stacks/azure/firecall/README.md
terraform/stacks/azure/firecall/dev/cloud-init.yaml
terraform/stacks/azure/firecall/dev/main.tf
terraform/stacks/azure/firecall/dev/variables.tf
terraform/stacks/azure/tfstate-bootstrap/README.md
terraform/stacks/azure/tfstate-bootstrap/prod/
terraform/stacks/azure/tfstate-bootstrap/prod/main.tf
terraform/stacks/azure/tfstate-bootstrap/prod/variables.tf
terraform/stacks/cloudflare/firecall/dev/
terraform/stacks/cloudflare/firecall-zone-policy/prod/
observability/alerts/firecall-rollback-failed.md
observability/alerts/firecall-control-failure.md

The inventory assigns one primary implementation group to each path.

The listed PIM, Conditional Access, and Cloudflare Entra identity-provider paths are retained inactive. They are deferred candidates, not development implementation or activation requirements.

The two post-prepared prefixes use the credential-bootstrap-prepared gate.

That gate requires exact prepared status before a composition can reference those ExternalSecrets.

Group A has one truth-reconciliation exception for existing shared source. It can change only prose, comments, Terraform descriptions, user-facing name labels, and unused description labels in these paths:

docs/runbooks/onboard-a-repo.md
scripts/ci/control-drift-expected.yaml
scripts/ci/firecall.sh
platform/firecall/profiles/dev-fusion-debug.yaml
platform/firecall/profiles/dev-namespace-debug.yaml
terraform/stacks/azure/acr-preview/preview/outputs.tf
terraform/stacks/azure/aks-platform/dev/main.tf
terraform/stacks/azure/ci-identities/
terraform/stacks/azure/firecall/dev/cloud-init.yaml
terraform/stacks/azure/firecall/dev/main.tf
terraform/stacks/azure/firecall/dev/variables.tf
terraform/stacks/azure/tfstate-bootstrap/prod/main.tf
terraform/stacks/azure/tfstate-bootstrap/prod/variables.tf

This exception makes the recorded current workflow and legacy identity state truthful. It permits no resource argument, default, permission, role, scope, output value, or workflow behavior change. Groups E and F retain primary implementation ownership after Group A merges.

Update platform/firecall/README.md to make v2 the active implementation and keep the v1 link.

Mark platform/firecall/v1-handover.md as historical v1 evidence and remove its parked implementation status.

Do not delete v1 source or evidence during Group A.

Each privileged workflow is a thin wrapper around its matching scripts/ci/firecall-*.sh entry point.

The script owns validation, planning, evidence creation, and post-apply verification.

The workflow owns only permissions, OIDC login, concurrency, environment approval, and artifact transport.

11.2 Control repository

The complete control-repository source inventory is:

.gitignore
.github/CODEOWNERS
.github/workflows/validate.yml
.github/workflows/open.yml
.github/workflows/close.yml
.github/workflows/sweep.yml
.github/workflows/breakglass.yml
.github/workflows/capacity-bootstrap.yml
.github/workflows/capacity-reconcile.yml
.github/workflows/audit-verify.yml
.github/workflows/breakglass-audit-reconcile.yml
.github/workflows/assignment-audit.yml
.github/workflows/privileged-open.yml
.github/workflows/privileged-close.yml
.github/workflows/privileged-sweep-schedule.yml
.github/workflows/privileged-sweep-manual.yml
.github/workflows/privileged-breakglass.yml
.github/workflows/firecall-capacity-bootstrap-privileged.yml
.github/workflows/privileged-capacity-reconcile-push.yml
.github/workflows/privileged-capacity-reconcile-manual.yml
.github/workflows/privileged-audit-verify-schedule.yml
.github/workflows/privileged-audit-verify-manual.yml
.github/workflows/privileged-breakglass-audit-reconcile-schedule.yml
.github/workflows/privileged-breakglass-audit-reconcile-manual.yml
.github/workflows/privileged-assignment-audit-schedule.yml
.github/workflows/privileged-assignment-audit-manual.yml
contracts/breakglass-consumption.schema.json
contracts/request-envelope.schema.json
contracts/request-receipt.schema.json
contracts/status.schema.json
contracts/profile.schema.json
contracts/capacity-ledger.schema.json
contracts/readiness-attestation.schema.json
contracts/revocation.schema.json
contracts/audit-event.schema.json
contracts/protocol-fixtures/
keys/broker-request-current.pem
keys/broker-request-previous.pem
profiles/dev-namespace-debug.yaml
profiles/dev-fusion-debug.yaml
config/breakglass.yaml
config/capacity.yaml
release-evidence/vm/
trust/sigstore/
scripts/verify-request
scripts/reserve-capacity
scripts/session
scripts/verify-cleanup
terraform/session/
README.md

This repository owns profiles, requests, status, capacity, session Terraform, readiness, cleanup proof, and VM evidence.

The infrastructure repository owns:

  • Broker and proxy source.
  • Agent source.
  • Front-end source.
  • Image build source.
  • Kubernetes deployment.
  • Azure trust declarations.
  • Runbooks and architecture.

12. Profile contract

Profiles are reviewed code.

The browser cannot override their authority.

Use a strict schema with no unknown fields.

Example:

apiVersion: firecall.openmined.org/v1
kind: SessionProfile
metadata:
  name: dev-namespace-debug
spec:
  enabled: false
  environment: dev
  cluster:
    name: aks-dev-eastus
    resourceGroup: rg-aks-platform-dev-eastus
    dnsDomain: cluster.local
    networkPathGeneration: 1
    connectionContractGeneration: 1
    connectionContractSha256: 1111111111111111111111111111111111111111111111111111111111111111
    caBundleSha256: 2222222222222222222222222222222222222222222222222222222222222222
  # This list is the namespace ceiling. A request receives only the selected
  # service-profile union beneath this list.
  namespaces:
    - sf-aigw
    - sf-fusion
    - sf-scoreboard
  vm:
    size: Standard_B2s
    imageVersionId: /subscriptions/.../versions/20260824.1.0
    imageEvidenceSha256: 3333333333333333333333333333333333333333333333333333333333333333
    osDiskType: StandardSSD_LRS
    osDiskSizeGiB: 32
  ttl:
    defaultMinutes: 240
    maxMinutes: 480
  costEstimate:
    currency: USD
    defaultSession: 0.20
    maximumSession: 0.40
    pricedAt: "2026-08-24"
    vmHourly: 0.0416
    publicIpHourly: 0.005
    osDiskMonthly: 2.40
    excludes: [network-egress, vnet-peering-data, shared-platform-cost, failed-cleanup-retention]
  ssh:
    allowed: false
    defaultEnabled: false
  access:
    clusterRoleDefinitionId: 4abbcc35-e782-43d8-92c5-2d3f1bd2253f
    namespaceReaderRoleDefinitionId: 7f6c6a51-bcf8-42ba-9220-52d62157d7db
    namespaceWriterRoleDefinitionId: a7ffa36f-339b-4b5c-8bdf-e2c188b2c0eb
  observability:
    signozBaseUrl: https://signoz.pulse.dev.openmined.org
  terminal:
    maxTabs: 8
    maxPreviews: 3
  protocol:
    agent: 1
    workspace: 1

The broker reads the catalog through the control-repository GitHub App.

Each selectable service uses one declarative adapter file:

apiVersion: firecall.openmined.org/v1
kind: ServiceProfile
metadata:
  name: aigw
spec:
  enabled: false
  baseProfile: dev-namespace-debug
  namespaces:
    - sf-aigw

The service profile is the complete onboarding adapter. Product code contains no hard-coded service list.

The portal permits one through eight service profile choices. Each selected service uses read by default. The requester can select write for any selected service without human approval. The browser sends selected names in service_profiles. It sends the write-selected subset in write_service_profiles. It sends no raw namespace or role. It contains no implicit dependency expansion.

The broker loads all selected files at the base profile's exact repository revision. It rejects an unknown, disabled, duplicate, mismatched, or out-of-ceiling service profile. It rejects a write name outside the selected service names.

The broker sorts selected names. It creates a sorted unique union with at most 16 namespaces. It calculates effective access for each namespace. Write wins when read and write services share a namespace. It then creates this canonical resolved scope:

{
  "namespace_access": [
    {
      "access": "write",
      "namespace": "sf-aigw"
    }
  ],
  "namespaces": ["sf-aigw"],
  "profile": {
    "name": "dev-namespace-debug",
    "revision": "40-lowercase-hex-characters",
    "sha256": "64-lowercase-hex-characters"
  },
  "schema": "firecall.resolved-scope.v2",
  "service_profiles": [
    {
      "access": "write",
      "name": "aigw",
      "sha256": "64-lowercase-hex-characters"
    }
  ]
}

scope_sha256 is the SHA-256 digest of those canonical JSON bytes. The broker stores it in the receipt and signs it in the open request. The control runner recomputes the complete object before Azure login.

The control runner copies the exact profile revision, service bindings, service access modes, namespace union, effective namespace access, and scope_sha256 into the signed bootstrap payload. The box recomputes the digest before it writes agent configuration. Lifecycle status and workspace status preserve the same fields. The broker and terminal proxy reject a changed, unsorted, duplicate, missing, or digest-mismatched scope.

Join invitations, pending tokens, acknowledgements, and accepted join tokens bind scope_sha256. The base profile digest alone is insufficient because one base profile can produce multiple service scopes.

This infrastructure repository does not produce lifecycle status. The control runner must copy the verified signed scope into every status write. Until that producer lands in the protected control repository, the broker and terminal proxy reject new v2 status records.

Before first activation, the verifier lists the bounded v2 status and invitation prefixes. It rejects any record without the complete scope fields. The operator must drain or migrate an incompatible record. The verifier does not enable a compatibility fallback.

Two service profiles can map to the same namespace. Kubernetes namespace RBAC cannot isolate services that share one namespace. For example, separate AIGW and AIGW UI choices both grant the reviewed sf-aigw namespace scope. If either service uses write, effective sf-aigw access is write.

An enabled profile requires a positive approved cost estimate and current image version.

The example remains disabled until the activation pull request supplies both values.

The request includes the profile revision and digest.

The runner checks both against its checked-out main revision.

A revision mismatch stops before Azure login.

Status records the selected agent and workspace protocol versions.

The proxy supports the current and previous protocol versions during rollout.

Roll out the agent image first, then the proxy, then the profile pin.

Both repositories keep shared contract fixtures for both supported versions.

13. Request contract

Use a signed JWS envelope.

JWS means JSON Web Signature.

Use ES256 with an Azure Key Vault P-256 key.

Use compact JWS serialization only.

Require exactly three canonical unpadded base64url segments.

The protected header contains exactly:

{"alg":"ES256","kid":"<known-key-id>","typ":"firecall-request+jws;v=1"}

Reject JWS JSON serialization, an unprotected header, multiple signatures, padding, and a noncanonical encoding.

Reject alg=none, another algorithm, crit, b64=false, jku, jwk, x5u, x5c, and every extra header.

Require the decoded payload bytes to equal their RFC 8785 canonical JSON form.

Reject duplicate keys before canonicalization.

Limit the complete compact envelope to 32 KiB and the decoded payload to 24 KiB.

The open workflow accepts only action=open.

The close workflow accepts only action=close.

No signed action can substitute for another workflow action.

Azure Key Vault does not support Ed25519 keys.

The private request key remains non-exportable.

The runner stores the current and previous public verification keys.

The open payload contains:

{
  "version": 1,
  "action": "open",
  "request_id": "uuid-v4",
  "idempotency_id": "uuid-v4",
  "session_id": "26-lowercase-base32-characters",
  "requester_subject_key": "64-lowercase-hex-characters",
  "requester_email": "developer@openmined.org",
  "profile": "dev-namespace-debug",
  "profile_revision": "git-commit-sha",
  "profile_sha256": "hex-digest",
  "service_profiles": [
    {
      "access": "write",
      "name": "aigw",
      "sha256": "hex-digest"
    }
  ],
  "namespaces": ["sf-aigw"],
  "namespace_access": [
    {
      "namespace": "sf-aigw",
      "access": "write"
    }
  ],
  "scope_sha256": "hex-digest",
  "aks_connection_contract_generation": 1,
  "aks_connection_contract_sha256": "64-lowercase-hex-characters",
  "aks_ca_bundle_sha256": "64-lowercase-hex-characters",
  "ttl_minutes": 240,
  "reason": "Investigate failing AIGW request",
  "ssh": {
    "enabled": false,
    "public_key": "",
    "source_cidr": ""
  },
  "issued_at": "2026-08-24T18:00:00Z",
  "envelope_expires_at": "2026-08-24T18:20:00Z"
}

The close payload contains:

  • Request identifier.
  • Target session identifier.
  • Target requester subject key.
  • Acting subject key.
  • Acting email.
  • Acting role.
  • Close reason.
  • Issue and expiry times.

The runner compares the target facts with durable status.

For an SSH-enabled open, it reparses the signed canonical key and source CIDR with the same strict contract.

It requires reserialization to equal the signed values byte for byte.

Place the canonical requester key, fingerprint, proof digest, and source CIDR only inside the signed typed bootstrap payload.

The AzureRM provider's administrator SSH field receives only the runner-generated provisioning public key.

Never interpolate either key or source CIDR into a shell command, cloud-init text, template directive, or Terraform expression string.

If runner status does not exist, it compares them with immutable target fields in the request receipt.

It records the acting identity separately from the original requester.

The runner rejects:

  • An invalid signature.
  • An unknown key identifier.
  • An expired envelope.
  • A future issue time.
  • An invalid identifier.
  • An unknown field.
  • An unknown action.
  • A raw or custom namespace selection.
  • An empty or duplicate service profile selection.
  • An invalid, duplicate, or unselected write service profile.
  • An unknown or disabled service profile.
  • A service profile from another base profile.
  • A service profile namespace outside the base profile ceiling.
  • A mismatched service access mode, namespace access, or resolved scope digest.
  • A custom role.
  • A custom VM size.
  • A mismatched profile revision.
  • A mismatched AKS connection-contract generation or digest.
  • A mismatched AKS CA-bundle digest.

The runner performs these checks before Azure login.

Require envelope_expires_at to equal issued_at + 20 minutes exactly.

Permit at most 120 seconds of clock skew against authenticated Azure Storage time.

The broker must receive GitHub's dispatch acceptance within 60 seconds after issued_at.

GitHub can queue the accepted run for at most 15 minutes after its authoritative created_at.

The runner must start verification before both that queue deadline and envelope_expires_at.

The runner rejects a delayed start, even when the signature remains valid.

14. Idempotency and workflow correlation

The browser sends an Idempotency-Key UUID.

The broker binds it to the derived subject key.

The broker derives the session identifier from the subject key and idempotency key.

Use the first 128 bits of SHA-256 over the version, environment, subject key, and idempotency key.

Encode those bits as 26 lowercase unpadded base32 characters.

Session identifiers are not credentials.

A random join capability is a revocable bearer secret for authenticated collaboration.

Never store or log its raw value.

The same form retry produces the same session identifier.

The broker creates one request receipt with If-None-Match: *.

A broker replica takes a finite lease on that receipt before dispatch.

A duplicate request returns that receipt instead of starting an independent dispatch.

A background dispatcher reconciles stale accepted receipts.

It reconstructs and signs the same logical request from the receipt.

It retries a pending open audit before any dispatch attempt.

It dispatches only when the receipt records the exact committed audit digest.

If the GitHub response was ambiguous, it can dispatch the same request again.

The runner's leased capacity-ledger entry and session lease still create only one session.

The receipt records dispatch attempt count, last attempt time, and safe error code.

Use at most three attempts with five, fifteen, and forty-five-second delays.

After the third failure, the dispatcher records dispatch_failed.

A failed GitHub dispatch records dispatch_failed without creating a capacity-ledger entry.

Transport retries can create duplicate workflow runs, but never duplicate ledger entries or Azure sessions.

Every open runner acquires the session lease before it evaluates durable session facts.

It checks the permanent replay tombstone, status, and capacity ledger before reservation.

A matching tombstone makes the run exit successfully without Azure login or capacity change.

A mismatched tombstone fails closed and alerts.

A matching ready status makes a duplicate run exit successfully.

A matching active status owned by another workflow makes the duplicate exit without resuming work.

The broker watcher and sweep reconcile an abandoned active run.

A duplicate never treats a freed capacity slot as permission to recreate a known session.

Terminal status can expire under retention, but its replay tombstone never expires.

The control runner creates one session entry through the leased capacity-ledger update.

Use GitHub API version 2026-03-10.

Send only the documented workflow-dispatch request fields.

Set return_run_details: true in the dispatch request.

Validate the HTTP 200 response schema and store its workflow_run_id directly.

A 204 response or absent run details is a dispatch failure.

Do not scrape run names or list recent runs for correlation.

The session time limit starts when capacity is reserved.

Provisioning time counts against the limit.

15. Durable status

Use a dedicated Firecall storage account.

Name it stfirecalldev30185530.

Place it in the standing rg-firecall-control-dev-eastus resource group.

The Firecall control stack owns this resource group and storage account.

Create it as StorageV2 with Standard tier, locally redundant storage, and hot access tier.

Locally redundant storage means account_replication_type=LRS.

Set hierarchical namespace, NFS, SFTP, cross-tenant replication, and infrastructure encryption to disabled.

Set allow_nested_items_to_be_public=false and shared_access_key_enabled=false.

The initial dev release accepts that LRS does not copy data to another Azure region or availability zone.

An East US regional loss can remove active and recoverable Firecall records despite versioning and soft delete.

Record this owner-accepted durability residual, the exact account kind, tier, replication, and feature flags in live evidence.

Before the first saved plan, query Azure name availability for stfirecalldev30185530 and kv-firecall-dev-eastus.

An unavailable name blocks bootstrap and requires a reviewed contract change.

Do not add a random suffix during apply.

Set Terraform prevent_destroy on the storage account and every listed container.

The first owner-run standing-control apply creates lock-firecall-storage-delete on the account.

That CanNotDelete lock starts in the protected standing-control state and needs no import.

The standing identity can read the lock but cannot create, update, or delete any resource lock.

It also lacks storage-account and container delete actions.

An Azure resource lock protects ARM resources but does not protect Blob data-plane content.

Use the exact Blob roles, immutability, versioning, soft delete, and legal holds for content protection.

Disable shared-key authorization.

Use Entra ID data-plane authorization only.

No Firecall human, workload, or OIDC custom role permits these actions:

Microsoft.Storage/storageAccounts/listKeys/action
Microsoft.Storage/storageAccounts/listAccountSas/action
Microsoft.Storage/storageAccounts/listServiceSas/action
Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action

Do not grant account-key read, account SAS, service SAS, or user-delegation-key creation through another built-in role.

An SAS means shared access signature.

Activation proves every Firecall principal receives denial for all four operations.

It also proves a copied SAS cannot bypass exact-path role conditions because no Firecall principal can mint one.

Enable public network access because GitHub-hosted runners use changing source addresses.

Require TLS 1.2 or later.

Disable anonymous Blob access.

This design accepts a public storage endpoint with identity-only authorization.

Private endpoints require a private runner and belong in a later hardening change.

Create these containers:

firecall-sessions
firecall-requests
firecall-status
firecall-tombstones
firecall-breakglass
firecall-capacity
firecall-capacity-bootstrap
firecall-probe-requests
firecall-probe-results
firecall-revocations
firecall-invitations
firecall-ratelimits
firecall-trust
firecall-trust-evidence
firecall-root-trust
firecall-audit
firecall-image-build-operations
firecall-control-monitor
firecall-security-status
firecall-security-evidence
firecall-scanner-data
firecall-scanner-status
firecall-node-tuple-status
firecall-node-tuple-evidence
firecall-access-policy-status
firecall-access-policy-evidence
firecall-tool-cache

Apply immutable-storage controls before any runtime identity receives Blob write authority.

Enable ordinary Blob versioning at the storage-account level.

Do not enable account-level or container-level version-level immutability support.

Set immutableStorageWithVersioning=false on every Firecall container.

Use container-level legal holds with protected append writes disabled on:

  • firecall-capacity-bootstrap, tag firecallbootstrap.
  • firecall-tombstones, tag firecallreplay.
  • firecall-trust-evidence, tag firecalltrust.
  • firecall-root-trust, tag firecallroot.
  • firecall-image-build-operations, tag firecalloperationlock.

The standing root creates the image-build-operations container and declares that exact required hold in its contract.

The owner data bootstrap applies the hold because routine standing authority cannot set or clear it.

Every standing plan and image workflow reads the live hold and fixed lock digest and fails on drift.

Use a locked 90-day policy on firecall-breakglass, firecall-audit, firecall-security-evidence, firecall-scanner-data, firecall-node-tuple-evidence, firecall-access-policy-evidence, and firecall-tool-cache.

These container-level WORM controls permit a new Blob but reject an overwrite or delete of its existing path.

For every protected container, prove that Put Blob, Put Block List, and Copy Blob cannot replace an existing path.

For firecall-image-build-operations, also prove lease acquire, renew, release, and break follow the declared role contract.

The legal hold protects the fixed operation-lock content while its narrow identities use only Blob lease actions.

WORM means write once, read many.

Keep mutable indexes, status, leaves, and capacity data outside these containers.

No routine Firecall identity can set, clear, shorten, unlock, or delete an immutability policy or legal hold.

The standing identity has read-only ARM access to these policies after the first owner-run apply.

Use these object paths:

firecall-sessions/azure/firecall/dev/sessions/<session_id>.tfstate
firecall-requests/sessions/<session_id>.json
firecall-status/sessions/<session_id>.json
firecall-status/locks/sessions/<session_id>
firecall-tombstones/sessions/<session_id>.json
firecall-breakglass/consumed/<nonce_sha256>.json
firecall-capacity/ledger.json
firecall-capacity-bootstrap/bootstrap.json
firecall-probe-requests/readiness/<session_id>.json
firecall-probe-results/readiness/<session_id>.json
firecall-revocations/sessions/<session_id>.json
firecall-revocations/identities/<subject_key>.json
firecall-revocations/deliveries/<session_id>/<operation_id>.json
firecall-invitations/sessions/<session_id>/index.json
firecall-ratelimits/<action>/<subject_key>/<window>.json
firecall-ratelimits/workspace/<action>/<session_id>/<subject_key>/<window>.json
firecall-trust/proxy-leaves/slots/current.pem
firecall-trust/proxy-leaves/slots/previous.pem
firecall-trust/proxy-leaves/active.json
firecall-trust/request-key-index.json
firecall-trust-evidence/proxy-leaves/<rotation_epoch>.pem
firecall-trust-evidence/request-key-candidates/<kid>.json
firecall-trust-evidence/request-key-status/<kid>/<state>.json
firecall-trust-evidence/credential-bootstrap/complete.json
firecall-trust-evidence/network-path/<generation>/<statement_sha256>.json
firecall-root-trust/roots/<client|origin>/<a|b>/<key-version>/<der_sha256>.der
firecall-root-trust/locks/<source_commit>/<lock_sha256>.json
firecall-audit/<component>/<yyyy>/<mm>/<dd>/<logical_event_id>/<payload_sha256>.json
firecall-audit/identity-reinstatement/intents/<subject_key>/<operation_id>.json
firecall-audit/identity-reinstatement/completions/<subject_key>/<operation_id>.json
firecall-image-build-operations/operation.lock
firecall-control-monitor/checkpoint.json
firecall-security-status/current.json
firecall-security-evidence/rescans/<scan_epoch>/<statement_sha256>.json
firecall-security-evidence/classic-cutover-trust/<epoch>/<statement_sha256>.json
firecall-scanner-data/trivy-db/v2/<oci_digest>/<blob_sha256>.tar
firecall-scanner-data/trivy-db/v2/<oci_digest>/<statement_sha256>.json
firecall-scanner-status/trivy-db-v2-current.json
firecall-scanner-data/clamav/<main|daily|bytecode>/<cvd_version>/<blob_sha256>.cvd
firecall-scanner-data/clamav/bundles/<bundle_generation>/<statement_sha256>.json
firecall-scanner-status/clamav-current.json
firecall-node-tuple-status/current.json
firecall-node-tuple-evidence/checks/<epoch>/<statement_sha256>.json
firecall-access-policy-status/current.json
firecall-access-policy-evidence/challenges/<yyyy>/<mm>/<dd>/<challenge_sha256>.json
firecall-access-policy-evidence/consumed/<challenge_sha256>.json
firecall-access-policy-evidence/checks/<epoch>/<statement_sha256>.json
firecall-tool-cache/artifacts/sha256/<artifact_sha256>/<canonical_filename>
firecall-tool-cache/evidence/sha256/<artifact_sha256>/<statement_sha256>.json
firecall-tool-cache/bundles/sha256/<statement_sha256>/<bundle_sha256>.json

The owner data-bootstrap script pre-creates all five mutable current-object sets with generation zero and state uninitialized.

Those sets are the security, Trivy, ClamAV, node-tuple, and access-policy current paths listed above.

It uses If-None-Match: * and records the canonical placeholder digest in activation evidence.

No Terraform root owns its content, and no routine identity can delete it.

15.1 Storage data-path authority

Azure custom roles limit operations, but they do not limit Blob names.

Every path-limited Blob role assignment uses Azure attribute-based access control.

Set condition_version = "2.0" on every such assignment.

Use this resource attribute for direct read, create, write, delete, and lease operations:

@Resource[Microsoft.Storage/storageAccounts/blobServices/containers/blobs:path]

Use StringEquals for one object and StringLike '<prefix>/*' for one bounded prefix.

Values contain no container name, leading slash, or URL encoding.

Blob listing uses the read DataAction with the Blob.List suboperation.

Every permitted list operation separately requires this request attribute:

@Request[Microsoft.Storage/storageAccounts/blobServices/containers/blobs:prefix]

The request prefix must start with the exact allowed prefix, including its final slash.

An empty prefix, container-root list, sibling prefix, and adjacent prefix fail.

For direct reads, the condition targets read with NOT SubOperationMatches{'Blob.List'}.

For permitted lists, it targets read with SubOperationMatches{'Blob.List'}.

For a role without list authority, add this complete deny clause:

!(ActionMatches{'Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read'}
  AND SubOperationMatches{'Blob.List'})

For each granted direct DataAction, add a separate conjunct with the exact path test.

This includes blobs/read without list, blobs/add/action, blobs/write, and blobs/delete when granted.

Do not grant tag, move, ownership, permission, permanent-delete, or super-user DataActions.

Use one role assignment per table row and operation set.

Create below means blobs/add/action for a new object.

Write means blobs/write and never implies delete.

Lease means the minimum Blob lease DataActions for the named object.

Identity Container Maximum path Operations List
Normal control firecall-sessions azure/firecall/dev/sessions/* Read, create, write, delete, and lease Exact prefix only
Normal control firecall-requests sessions/* Read and delete Exact prefix only
Normal control firecall-status sessions/* and locks/sessions/* in separate assignments Read, create, write, delete, and lease Exact prefixes only
Normal control firecall-tombstones sessions/* Read and create No
Normal control firecall-capacity ledger.json Read, write, and lease No
Normal control firecall-capacity-bootstrap bootstrap.json Read No
Normal control firecall-trust proxy-leaves/active.json and the two fixed selected-leaf slot paths Read No
Normal control firecall-trust-evidence network-path/* Read and create No
Normal control firecall-root-trust Exact roots/, locks/, and bundles/ objects pinned by the protected profile Read No
Normal control firecall-security-status current.json Read No
Normal control firecall-security-evidence rescans/* Read No
Normal control firecall-node-tuple-status current.json Read No
Normal control firecall-node-tuple-evidence checks/* Read No
Normal control firecall-access-policy-status current.json Read No
Normal control firecall-access-policy-evidence checks/* Read No
Normal control firecall-probe-requests readiness/* Read, create, and delete Exact prefix only
Normal control firecall-probe-results readiness/* Read and delete Exact prefix only
Normal control firecall-revocations sessions/* Read and delete Exact prefix only
Normal control firecall-invitations sessions/* Read and delete Exact prefix only
Broker firecall-requests sessions/* Read, create, write, and lease Exact prefix only
Broker firecall-revocations sessions/* Read, create, write, and lease No
Broker firecall-revocations deliveries/* Read No
Broker firecall-ratelimits Its exact component prefixes Read, create, write, and lease No
Broker firecall-revocations identities/* Read and create No
Broker firecall-audit broker/* Create No
Broker firecall-status sessions/* Read Exact sessions/ prefix only
Broker firecall-invitations sessions/*/index.json Read Exact sessions/ prefix only
Broker firecall-capacity ledger.json Read No
Broker firecall-security-status current.json Read No
Broker firecall-security-evidence rescans/* Read No
Broker firecall-node-tuple-status current.json Read No
Broker firecall-node-tuple-evidence checks/* Read No
Broker firecall-access-policy-status current.json Read No
Broker firecall-access-policy-evidence checks/* Read No
Broker firecall-access-policy-evidence challenges/* Create No
Proxy Status and session-revocation containers Their exact section 15 prefixes Read No
Proxy firecall-revocations identities/* Read No
Proxy firecall-revocations deliveries/* Read and create No
Proxy Invitation and rate-limit containers Their exact section 15 prefixes Read, create, write, and lease No
Proxy firecall-audit proxy/* Create No
Readiness worker firecall-probe-requests readiness/* Read Exact prefix only
Readiness worker firecall-probe-results readiness/* Create No
Capacity bootstrap firecall-capacity ledger.json Read and write No
Capacity bootstrap firecall-capacity-bootstrap bootstrap.json Read and create No
Capacity reconciler firecall-capacity ledger.json Read, write, and lease No
Capacity reconciler firecall-capacity-bootstrap bootstrap.json Read No
Capacity reconciler firecall-status sessions/* Read Exact prefix only
Capacity reconciler Classic tfstate container Exact keys azure/firecall/dev.tfstate, azure/firecall/dev-slot2.tfstate, and azure/firecall/dev-slot3.tfstate Read and lease-status inspection No
Capacity reconciler firecall-security-evidence classic-cutover-trust/* Read No
Capacity reconciler firecall-audit capacity/* Create No
Assignment auditor firecall-capacity ledger.json Read No
Assignment auditor firecall-status sessions/* Read Exact prefix only
Break glass firecall-breakglass consumed/* Read and create No
Break glass firecall-sessions azure/firecall/dev/sessions/* Read Exact prefix only
Break glass firecall-status sessions/* Read Exact prefix only
Break glass firecall-audit breakglass-consumption/* Create No
Break-glass audit reconciler firecall-breakglass consumed/* Read Exact prefix only
Break-glass audit reconciler firecall-audit breakglass-consumption/* Read and create No
Control monitor firecall-audit control-workflow-monitor/* Read and create Exact prefix only
Control monitor firecall-control-monitor checkpoint.json Read, write, and lease No
Control monitor firecall-requests sessions/* Read Exact prefix only
Control monitor firecall-status sessions/* Read Exact prefix only
Security rescan firecall-security-status current.json Read, write, and lease No
Security rescan firecall-security-evidence rescans/* Read and create No
Security rescan firecall-audit security-rescan/* Create No
Security rescan firecall-scanner-data trivy-db/v2/* Read No
Security rescan firecall-scanner-data clamav/* Read No
Security rescan firecall-scanner-status trivy-db-v2-current.json and clamav-current.json in separate assignments Read No
Scanner-database publisher firecall-scanner-data trivy-db/v2/* and clamav/* in separate assignments Read and create No
Scanner-database publisher firecall-scanner-status trivy-db-v2-current.json and clamav-current.json in separate assignments Read, write, and lease No
Tool-cache publisher firecall-tool-cache artifacts/sha256/*, evidence/sha256/*, and bundles/sha256/* in separate assignments Read and create No
Image builder firecall-tool-cache The same three prefixes in separate assignments Read No
Scanner-database publisher firecall-tool-cache The same three prefixes in separate assignments Read No
Retention reconciler firecall-security-status current.json Read No
Retention reconciler firecall-scanner-status Both fixed current objects in separate assignments Read No
Retention reconciler firecall-node-tuple-status current.json Read No
Retention reconciler firecall-access-policy-status current.json Read No
Retention reconciler firecall-status sessions/* Read Exact prefix only
Retention reconciler firecall-audit Each closed producer prefix in a separate assignment Read Exact prefixes only
Retention reconciler firecall-scanner-data trivy-db/v2/* and clamav/* in separate assignments Read and delete Exact prefixes only
Retention reconciler firecall-security-evidence Closed immutable evidence prefixes Read and delete Exact prefixes only
Retention reconciler firecall-node-tuple-evidence checks/* Read and delete Exact prefix only
Retention reconciler firecall-access-policy-evidence challenges/*, consumed/*, and checks/* in separate assignments Read and delete Exact prefixes only
Retention reconciler firecall-revocations deliveries/* Read and delete Exact prefix only
Retention reconciler firecall-tool-cache The three cache prefixes in separate assignments Read and delete Exact prefixes only
Retention reconciler firecall-audit retention/* Read and create Exact prefix only
Trust observer firecall-audit trust-drift/* Create No
Trust observer firecall-security-evidence classic-cutover-trust/* Create No
Trust observer firecall-access-policy-status current.json Read, write, and lease No
Trust observer firecall-access-policy-evidence checks/* Read and create No
Trust observer firecall-access-policy-evidence challenges/* Read Exact challenges/ prefix only
Trust observer firecall-access-policy-evidence consumed/* Read and create No
Image-idle reconciler firecall-audit image-idle/* Create No
Image-inventory observer firecall-security-evidence image-inventory/* Read and create No
Image-inventory observer firecall-audit image-inventory/* Create No
Cloudflare drift observer firecall-audit cloudflare-drift/* Create No
Node-tuple observer firecall-node-tuple-status current.json Read, write, and lease No
Node-tuple observer firecall-node-tuple-evidence checks/* Read and create No
Node-tuple observer firecall-audit node-tuples/* Create No
Standing apply firecall-standing-state Its two exact backend keys Read, create, write, delete, and lease Exact keyenv: prefixes only
Standing apply firecall-image-build-operations operation.lock Read and write for lease only No
Image builder and idle reconciler firecall-image-build-operations operation.lock Read and write for lease only No
Trust apply firecall-trust-state azure/firecall-trust/prod.tfstate Read, create, write, delete, and lease Exact keyenv: prefix only
Owner bootstrap firecall-policy-definition-state azure/firecall-policy-definition/prod.tfstate and azure/firecall-image-build-policy/prod.tfstate in separate assignments Read, create, write, delete, and lease Exact keyenv: prefixes only
Budget owner firecall-policy-definition-state azure/firecall-budget/prod.tfstate Read, create, write, delete, and lease Exact keyenv: prefix only
Policy apply firecall-policy-state azure/firecall-policy/dev.tfstate Read, create, write, delete, and lease Exact keyenv: prefix only
Edge apply firecall-edge-state cloudflare/firecall/dev.tfstate Read, create, write, delete, and lease Exact keyenv: prefix only
Edge apply firecall-security-evidence cloudflare-cache/* Read No
Bounded owner recovery firecall-edge-state cloudflare/firecall/dev.tfstate Read, write, and lease Exact keyenv: prefix only
Zone-policy apply firecall-zone-policy-state cloudflare/firecall-zone-policy/prod.tfstate Read, create, write, delete, and lease Exact keyenv: prefix only
Zone-policy apply firecall-security-evidence cloudflare-cache/* Read and create No
Credential rotator firecall-trust request-key-index.json Read, create, write, and lease No
Credential rotator firecall-trust-evidence request-key-candidates/* and request-key-status/* in separate assignments Read and create No
Credential rotator firecall-trust-evidence credential-bootstrap/complete.json Read and create No
Certificate rotator firecall-trust proxy-leaves/slots/current.pem and proxy-leaves/slots/previous.pem in separate assignments Read and write No
Certificate rotator firecall-trust proxy-leaves/active.json Read, create, write, and lease No
Certificate rotator firecall-trust-evidence proxy-leaves/<rotation_epoch>.pem Read and create No
CA-root workflow firecall-root-trust roots/*, locks/*, and bundles/* in separate assignments Read and create No
Temporary reinstatement operator firecall-revocations One exact identities/<subject_key>.json object Read and conditional delete No
Temporary reinstatement operator firecall-audit One exact intent object and one exact completion object in separate assignments Create No
Each other audit writer firecall-audit One fixed prefix from the closed producer set below Create No
Audit verifier firecall-audit One separate assignment for each closed producer prefix Read Exact producer prefix only
Audit verifier firecall-requests sessions/* Read Exact prefix only
Audit verifier firecall-status sessions/* Read Exact prefix only
Audit verifier firecall-tombstones sessions/* Read Exact prefix only
Audit reader group firecall-audit One separate assignment for each closed producer prefix Read Exact producer prefix only

The normal control identity receives no operation on firecall-revocations/identities/*.

It cannot read, list, create, write, lease, delete, tag, move, or permanently delete any global identity marker.

It also receives no operation on firecall-revocations/deliveries/*.

The closed audit producer-prefix set is:

broker/
proxy/
runner/
capacity/
breakglass-consumption/
image-seal/
ca-root/
credential-rotation/
edge/
control-workflow-monitor/
security-rescan/
trust-drift/
image-idle/
image-inventory/
cloudflare-drift/
node-tuples/
identity-reinstatement/

No audit reader or verifier can list the container root.

Each list request must use one exact slash-terminated producer prefix.

The normal-control leaf slots contain the current and previous complete public certificates.

active.json binds each fixed slot to its immutable evidence path, digest, epoch, and validity.

The runner validates that binding before it uses a fixed slot.

The protected profile names every required root, lock, and bundle object before its exact role assignments apply.

The role-condition source records the custom-role digest, condition text, assignment ID, and expected path.

Activation rereads and hashes every live condition.

The credential rotator can read, lease, and conditionally write only request-key-index.json.

It cannot delete that index or create another index.

When the index is absent, correct code creates the schema-valid empty index with If-None-Match: *.

Azure RBAC cannot require that conditional header.

The index is intentionally mutable through a Blob lease and ETag checks.

It never replaces an unreadable or invalid index.

Correct code creates candidate and status objects with If-None-Match: *.

The separate legal-held WORM container rejects their overwrite or deletion.

The rotator cannot list or lease candidate and status prefixes.

Positive tests use the exact object or prefix.

Negative tests use ledger.json.bak, a sibling prefix, an empty list prefix, root listing, and encoded path variants.

No document can describe an object as exact-scoped unless this live condition passes.

The runner writes requests only to firecall-probe-requests.

The readiness worker reads requests and writes only to firecall-probe-results.

The proxy cannot write lifecycle status, probes, the capacity ledger, or Terraform state.

It can create, read, lease, and conditionally update one bounded invitation index per session.

Each index entry contains only identifier, capability hash, session, profile digest, expiry, bound subject key, and revocation time.

The initial bound subject key is null.

The first exchange sets it through an ETag comparison.

Two different subjects racing one capability produce exactly one winner.

The broker writes request receipts only in firecall-requests.

It cannot write runner lifecycle state in firecall-status.

It can read and list only the exact firecall-status/sessions/ prefix for authorized portal views.

It can read only firecall-capacity/ledger.json and cannot lease or write that Blob.

The receipt preserves pending-page authorization, idempotency, and the workflow run identifier.

The runner creates one permanent replay tombstone after zero-resource proof and before terminal status.

It uses If-None-Match: * and never overwrites or deletes that tombstone.

The tombstone contains only session, request, terminal reason, profile digest, zero-resource-proof digest, and verification time.

It contains no human email, reason text, credential, address, or terminal data.

Its terminal status is only rejected, closed, expired, or failed.

rollback_failed never receives a tombstone because cleanup is not proved.

It also records the deterministic open-audit object path, payload digest, and pending or committed audit state.

The broker creates a close-revocation marker after authorization and before dispatch.

The proxy reads this marker and denies new or existing data-plane access.

The marker also stores close dispatch attempts, workflow run identifier, deterministic audit path, payload digest, and audit retry state.

A lease-backed close dispatcher retries the idempotent close workflow at most three times.

Only the normal control sweep clears it after a terminal lifecycle state and committed close audit.

The broker uses per-subject Blob leases for exact cross-replica rate limits.

It stores only a subject key, action, count, and window expiry.

Audit writers create one immutable object per event with If-None-Match: *.

They cannot overwrite, list, read, or delete audit objects.

The runner writes verifying and a minimal probe request with a random readiness nonce.

The readiness worker detects that probe request.

It asks the proxy to test the real mutual-TLS path.

The proxy tests the agent health endpoint and opens one disposable PTY.

The worker writes the proxy-signed result to firecall-probe-results.

The runner validates the proxy certificate chain, URI identity, SPKI, and result signature.

It also verifies the nonce, route facts, time, protocol versions, PTY challenge, and test results.

It then deletes both probe objects and publishes ready.

The attestation proves that the approved proxy key signed the deployed route result.

Independent runner checks still verify image, identity, namespace denial, and local expiry.

The latest status record conforms to the canonical example at platform/firecall/contracts/examples/lifecycle-status.example.json.

Do not duplicate that contract in this document.

Do not store:

  • Private keys.
  • JWTs.
  • Cookies.
  • GitHub tokens.
  • Kubernetes credentials.
  • Terminal data.
  • Secret values.
  • Requester email.
  • Free-text request reason.

Lifecycle status is the proxy-readable projection.

It includes request, bootstrap, and plan digests. It also includes managed identity, role-assignment, cleanup, and bounded audit metadata.

The broker joins display email and reason from its request receipt only when an authorized portal view needs them.

The proxy cannot read request receipts.

Use ETag conditional writes.

Use a monotonic sequence number.

Runner status contains a bounded audit_items map for lifecycle and cleanup events.

Each item stores logical event ID, payload digest, state, attempts, and next retry time.

The safe event payload reconstructs only from immutable request and status fields.

The map has one fixed key per permitted lifecycle transition and cannot accept an arbitrary event type.

Reject an invalid state transition.

Store each completed build stage and its UTC timestamp durably.

The portal renders the build timeline from these stage timestamps.

15.2 Portal status projection

The portal combines request receipts, close revocations, and runner status with fixed precedence.

Use this order:

  1. Show closing, terminal states, or rollback_failed from runner status.
  2. Show Close requested when a revocation exists.
  3. Show Access expired · cleanup pending after exact expiry for other active states.
  4. Show the remaining runner status.
  5. Show the request receipt state before runner status exists.

The first reserved status record is the exact handoff from broker state to runner state.

After that handoff, the portal ignores accepted or dispatched receipt state.

The broker watches the recorded GitHub run until runner status appears.

A failed pre-reservation run becomes control_failed with a safe error code.

A canceled pre-reservation run becomes canceled.

Portal lists and details use the same projection.

16. State machine

stateDiagram-v2
    [*] --> accepted
    accepted --> dispatched
    accepted --> dispatch_failed
    dispatch_failed --> accepted
    dispatched --> control_failed
    dispatched --> canceled
    control_failed --> accepted
    canceled --> accepted
    dispatched --> accepted_request
    accepted_request --> reserved
    accepted_request --> rejected
    reserved --> building
    reserved --> closing
    building --> verifying
    verifying --> ready
    building --> closing
    verifying --> closing
    ready --> closing
    closing --> closed
    closing --> expired
    closing --> failed
    closing --> rollback_failed
    rollback_failed --> closing

platform/firecall/contracts/lifecycle-transitions.json is the normative runner lifecycle graph. Groups B and D must consume this file directly.

The Mermaid diagram also shows broker receipt states before the runner starts. Those receipt transitions are separate from the runner lifecycle contract.

Use these meanings:

State Meaning Counts against capacity
accepted The broker stored an idempotent request receipt No
dispatched GitHub accepted the request No
dispatch_failed Signing or dispatch failed before reservation No
control_failed The control run failed before runner status existed No
canceled The control run ended before reservation No
accepted_request The runner accepted the signed request contract No
rejected The runner safely rejected the request before resource creation No
reserved An atomic capacity slot exists Yes
building Azure resources or readiness are in progress Yes
verifying The proxy is testing mutual TLS, agent health, and one PTY Yes
ready Every readiness probe succeeded Yes
closing New access is denied and destruction is active Yes
closed Early-close cleanup has zero-resource proof No
expired Expiry cleanup has zero-resource proof No
failed Build failed and cleanup has zero-resource proof No
rollback_failed Cleanup is incomplete or unproved Yes

Never publish closed, expired, or failed without cleanup_verified=true.

One ETag-guarded write publishes terminal state, stage, reason, proof digest, proof time, and cleanup_verified=true together.

No intermediate terminal record is valid.

An older ready update can never overwrite closing.

17. Capacity and concurrency

The durable capacity control is one pre-created firecall-capacity/ledger.json Blob.

Terraform owns firecall-capacity and the separate firecall-capacity-bootstrap marker container.

Enable ordinary Blob versioning on the account.

Keep version-level immutability support disabled.

Apply a permanent container legal hold to only firecall-capacity-bootstrap.

Use the valid alphanumeric legal-hold tag firecallbootstrap.

Read the policy back and require that exact tag before bootstrap.

Do not apply that hold to the mutable ledger container.

Terraform never owns the ledger Blob, marker Blob, or their content.

After both containers exist, firecall-capacity-bootstrap-privileged.yml performs one bootstrap operation.

Use a separate one-time OIDC identity for that workflow.

It can read and write only ledger.json and bootstrap.json.

Exact path conditions enforce both object names.

Azure RBAC cannot distinguish a create from an overwrite for these block Blobs.

Correct code uses If-None-Match: * for both creates.

The marker legal hold rejects an overwrite of bootstrap.json.

Before revocation, a compromised bootstrap identity can overwrite ledger.json and deny service.

It cannot list, delete, lease, change a policy, or manage another resource.

It reads the exact initial protected configuration from control main.

That configuration must set creation_enabled=false and capacity_limit: 2.

It creates the schema-valid three-slot ledger with If-None-Match: *.

It rereads the new ledger and creates immutable firecall-capacity-bootstrap/bootstrap.json with If-None-Match: *.

The marker stores the first ledger ETag, content digest, configuration digest, commit, and creation time.

The separate container legal hold makes that marker permanent.

An existing ledger with no marker permits only marker completion when the ledger still equals the exact initial document.

Any occupied slot or changed epoch before marker completion blocks activation and alerts.

An existing marker makes bootstrap read and validate both objects without a write.

If the marker exists and the ledger is absent, fail closed.

Recover a prior Blob version through the incident runbook.

Never create a new empty ledger after the marker exists.

Disable the bootstrap service principal immediately after marker verification.

Remove its two data-role assignments and wait for its existing token to expire before enabling session creation.

Bootstrap never resets slots, occupancy, epoch, gate, limit, digest, or ETag.

After bootstrap proof, remove the one-time federated credential and both data-role assignments.

Wait for the issued Azure token lifetime, then prove the bootstrap workflow can no longer authenticate.

The normal capacity identity can read the marker but cannot modify its container.

Later standing-resource applies verify only container existence and data protections.

They cannot create, replace, import, or destroy ledger.json.

Lease that same Blob before every capacity change.

Do not use GitHub concurrency for per-session serialization.

Use 60-second leases and renew them every 20 seconds.

Treat one failed renewal as lease loss.

Every operation records a monotonic operation_epoch under the lease.

Every capacity-ledger write supplies the current lease identifier and If-Match ETag.

Azure then rejects a ledger update without the active lease identifier.

Status writes use the recorded operation epoch and their own ETag.

Close revocations do not wait for a session lease.

The close runner writes closing through an ETag retry loop with terminal-state precedence.

An open status write that races once can succeed only before that closing ETag wins.

The revocation marker still denies access during that race.

17.1 Open reservation

The ledger always has three numbered slots, capacity_limit, per_subject_limit, creation_enabled, and protected configuration digest fields.

It also has an immutable bootstrap_marker_sha256 field from the first legal-held marker.

Set capacity_limit: 2 during coexistence.

Set capacity_limit: 3 only after Classic open stops and every Classic session ends.

Set per_subject_limit: 1 for the initial release.

Count a subject by its exact domain-separated subject_key.

Do not store the raw Cloudflare subject in the capacity ledger.

The requester, collaborator, and platform-owner roles receive no bypass.

Changing this limit requires the same protected configuration merge, owner review, and capacity reconcile as the global limit.

Store the expected limit in protected control configuration with its digest.

Store creation_enabled in that same protected configuration.

The portal, broker, open runner, and capacity reconciler require matching configuration and ledger values.

They fail closed when either value is unreadable, mismatched, or invalid.

No open can reserve capacity when creation_enabled=false.

Changing either field requires a protected configuration merge and the dedicated capacity reconcile workflow.

Each occupied slot contains session, request, subject key, epoch, reservation time, expiry, and cleanup state.

Use this transaction:

  1. Read the exact bootstrap marker without a list operation.
  2. Validate its schema, initial ledger digest, ETag, configuration digest, and permanent marker digest.
  3. Acquire a finite lease on firecall-capacity/ledger.json itself.
  4. Read that leased Blob and its ETag.
  5. Require its immutable marker digest to match the validated marker.
  6. Stop on any read, schema, gate, limit, configuration-digest, slot-count, duplicate-session, or consistency error.
  7. Return the existing slot when the same request already owns one.
  8. Count occupied slots for the exact subject key while the same lease remains held.
  9. Refuse creation when that count reaches the reviewed per-subject limit.
  10. Refuse creation when creation is disabled or occupied slots equal or exceed the reviewed global limit.
  11. Add one session to one free slot and increment its operation epoch.
  12. Replace the same Blob with the lease identifier and If-Match ETag.
  13. Reread the same Blob and verify the exact committed entry.
  14. Write the per-session reserved status.
  15. Release the ledger lease.
  16. Provision the session in parallel.

If the reserved-status write fails after the committed ledger replacement, retain the occupied ledger slot.

The sweep reconciles that safe orphan before capacity can return.

If lease renewal fails before the ledger replacement, create no ledger entry.

If the ledger-replacement result is ambiguous, reacquire the lease and reread the ledger.

Never allocate a second slot until that reread proves the first update absent.

The single leased ledger is the only capacity counter.

Do not create separate reservation Blobs.

Status validates ledger ownership but never adds another capacity count.

17.1.1 Protected capacity reconciliation

Store the reviewed values in config/capacity.yaml in the control repository.

Add capacity-reconcile.yml and event-specific privileged-capacity-reconcile-push.yml and privileged-capacity-reconcile-manual.yml workflows.

A merge to that exact config path starts the public workflow on control main.

Only the reusable job receives Azure OIDC permission.

Use a dedicated capacity-reconcile Entra application and service principal.

Each exact flexible credential binds the immutable control repository subject and one event-specific capacity workflow on main.

Give it read access to only the Classic-observer App primary and secondary Key Vault slots.

The protected config pins the App ID, installation ID, infrastructure repository ID, and workflow numeric ID.

The workflow mints the App JWT and installation token only in memory.

It rejects a token for another installation, account, or repository.

Give it conditional read access to only these Classic state objects:

  • tfstate/azure/firecall/dev.tfstate.
  • tfstate/azure/firecall/dev-slot2.tfstate.
  • tfstate/azure/firecall/dev-slot3.tfstate.

Give it the minimum Azure read authority needed for the exact active Classic resource-group IDs derived from those states. Every accepted group name must equal rg-firecall-dev-eastus or match rg-firecall-dev-<session_id>-eastus.

Give it roleAssignments/read on only the exact dev cluster and reviewed Classic namespace scopes.

It receives no control-plane or data-plane write outside the capacity ledger and audit append.

The Classic rejection change adds platform/firecall/classic-cutover.json.

That manifest fixes the combined workflow ID, all three state locations, the resource-group naming template, namespace scopes, and expected source files.

It records SHA-256 digests for the manifest, workflow, and scripts/ci/firecall.sh.

The combined Classic workflow reads this manifest before Azure authentication.

It rejects open, but it keeps close and sweep enabled.

The combined Classic workflow adds a protected zero-proof operation.

Add one new firecall-classic-reduction.yml caller and commit-pinned privileged reusable workflow.

Its new flexible credentials authorize only close, sweep, and zero-proof on current protected source.

Before capacity becomes three, remove the historical combined-workflow federated credential.

A rerun of one fixed pre-rejection Classic open must then fail Azure authentication.

Use firecall-classic-auth-denial.yml for that one owner-approved cutover operation.

Its dispatch input must equal the historical run ID in platform/firecall/classic-cutover.json.

The workflow verifies the exact Classic workflow ID, original source SHA, original attempt one, completed state, and pre-rejection open operation.

It then calls the GitHub rerun API once and waits for the new attempt to finish.

It requires the exact Azure-login step to fail before any Terraform or Azure mutation step starts.

It publishes a bounded artifact with run, attempt, job, step, source, time, and conclusion metadata only.

In the same owner-approved run, it lists only the exact Classic open workflow ID.

It cancels at most 100 returned runs whose operation is open and state is queued, waiting, pending, requested, or in progress.

It verifies each repository, workflow ID, run ID, operation, source, and state again immediately before cancellation.

It never cancels Classic close, sweep, zero-proof, v2, or another repository workflow.

It repeats the exact open-run query and includes the bounded cancellation results in its artifact.

The workflow has repository Actions write but no OIDC, Azure, App, secret, or source-write authority.

The trust observer separately publishes signed create-once cutover evidence.

That evidence proves the historical FIC is absent and both current Classic reduction FICs match protected source.

Store it under firecall-security-evidence/classic-cutover-trust/<epoch>/<digest>.json with ten-minute validity.

The new reduction identity receives no VM, role-assignment, or state action that can create or expand a Classic session.

The final Classic close publishes a keyless-signed firecall-classic-zero-proof artifact when it removes the last session.

If Classic is already empty, dispatch zero-proof after the open-rejection commit merges.

That operation creates no Azure resource, role, or state change.

It reads all three retained states, every exact state-derived group, exact role assignments, the rejection manifest, and current protected source.

It refuses any source commit other than current infrastructure main with the merged rejection.

It uses id-token: write only in the fixed proof-signing job.

Either path uses the same artifact schema and exact Sigstore workflow identity.

The artifact binds its rejection commit, source digests, all three state ETags and digests, exact prior assignment IDs, and zero-resource checks.

The capacity reconciler downloads that exact artifact through the read-only App.

It verifies the Sigstore workflow identity, source commit, artifact digest, and every bound identifier.

It then performs independent live checks.

It requires all three current Classic state Blobs to exist without active leases.

Each decoded state must contain zero managed resources and only null or disabled session outputs.

All three state Blobs remain retained for audit and rollback.

It requires zero resources in rg-firecall-dev-eastus and every state-derived group matching rg-firecall-dev-<session_id>-eastus.

It also requires every bound Classic Cluster User and namespace Writer assignment to be absent.

It queries the current protected infrastructure main ref through the App.

It rereads the three exact source files at that commit and compares their digests.

It requires the combined workflow to remain enabled.

It also requires the historical Classic federated credential to be absent.

It requires the exact current reduction-workflow credentials and source SHA to match protected trust evidence.

It does not rerun a workflow.

It reads the exact owner-approved authentication-denial run and artifact through the Classic-observer App.

Protected config/capacity.yaml pins that run ID, artifact digest, historical run ID, rerun attempt, and trust-evidence path.

It requires the cutover workflow, protected head SHA, firecall-live approval, successful conclusion, and exact artifact metadata.

It reads the signed trust-observer evidence without a list operation and requires current validity.

It then requires Azure-login failure in the fixed historical attempt before any mutation step.

It lists every non-terminal run for that workflow.

No queued, waiting, pending, requested, or in-progress Classic open run can remain.

Repeat the source, run, state, resource, and assignment reads twice, five minutes apart.

Repeat them again after acquiring the capacity lease and immediately before the ledger write.

It can:

  • Read the protected config at its exact workflow commit.
  • Read, lease, and conditionally update only firecall-capacity/ledger.json.
  • Read lifecycle status and Classic zero-session evidence.
  • Append a capacity-configuration audit event.

It cannot create, update, execute on, or delete a session resource.

It cannot change a role, network, Key Vault, Gallery, policy, request, status, or tombstone.

The workflow uses this transaction:

  1. Prove the triggering commit is the current protected control main commit.
  2. Parse the strict config and compute its digest.
  3. Read and validate the exact bootstrap marker.
  4. Acquire the lease on firecall-capacity/ledger.json.
  5. Read and validate the complete ledger, marker digest, and ETag.
  6. Require the signed proof and repeated live Classic checks before a two-to-three increase.
  7. Require creation_enabled=false before a reduction below current occupancy.
  8. Append the deterministic capacity-change-intent audit event.
  9. Update gate, limit, digest, source commit, and operation epoch in one leased conditional write.
  10. Reread and compare the complete ledger.
  11. Append the deterministic capacity-change-applied audit event.
  12. Release the lease and publish safe evidence.

An intent-audit failure makes no ledger change.

An applied-audit failure leaves the reviewed new gate and limit in place and alerts.

The next protected reconcile retries its deterministic audit event.

A scheduled read-only check compares the current protected config, ledger, and active occupancy.

Any drift blocks open in the portal, broker, and runner.

The ledger schema permits occupied slots above capacity_limit only when creation_enabled=false.

That state represents rollback drain and can only decrease occupancy.

The initial live config uses creation_enabled=false and capacity_limit: 2.

The canary activation change enables creation at limit two.

The final cutover change keeps creation enabled and raises the limit to three.

Rollback first disables creation and lowers the limit to two in one protected merge.

17.2 Per-session serialization

Use firecall-status/locks/sessions/<session_id> as the operation lock.

Before any lease, create its canonical binding with If-None-Match: *.

The binding contains only schema version, session ID, request ID, requester subject key, and profile digest.

An existing object must match every immutable field before a runner can lease it.

An absent state Blob is valid and does not affect operation-lock creation.

Terraform separately uses its normal backend lease after it creates or reads session state.

Open, close, sweep, and recovery use the same operation-lock object.

Do not use an unverified workflow input as a GitHub concurrency key.

Open holds the operation lease during Terraform and readiness work.

Open starts a five-second revocation watchdog before capacity reservation.

The watchdog interrupts Terraform and every later stage after a close marker appears.

Close does not wait for that lease before it publishes closing.

The broker requests cancellation of the exact recorded open workflow after close audit commits.

It verifies repository ID, workflow ID, run ID, session receipt, and open action before that GitHub call.

Cancellation is best effort; the five-second revocation watchdog remains the authoritative interruption control.

Close deallocates any deterministic session VM as a reduction-only action.

Close removes discovered session role assignments as a reduction-only action.

Close then waits for the operation lease and runs the normal Terraform destroy.

Multiple close requests remain idempotent.

Renew the session lease for the complete Terraform and readiness operation.

On lease loss, stop every stage after the current Azure or Terraform call.

Do not force-kill an active Terraform provider process after Azure can accept its request.

Let that process return or reach its fixed provider timeout, then start reconciliation.

Reread revocation, operation epoch, and status before each stage resumes.

Blob fencing cannot cancel an Azure request that the runner already sent.

Before Terraform apply, record the operation epoch, start time, exact state ETag, and deterministic resource IDs.

After Terraform returns, query the session-group Activity Log before any readiness stage.

Persist each discovered Azure correlation ID, request ID, resource ID, operation, status, and event time as immutable audit evidence.

Direct Azure SDK actions record their response request and operation IDs before the next action.

Resource-group operations require a terminal Activity Log record.

AKS-scoped role assignments use a separate direct-operation journal.

The runner records each assignment PUT or DELETE response and deterministic assignment ID immediately.

It polls the exact assignment through direct GET or absence checks until the expected state is stable.

It also lists assignments at the exact target scope and rejects a conflicting GUID or principal.

Require one-minute reads for 15 continuous minutes before readiness or zero-resource proof.

A late assignment, ambiguous response, unexpected principal, or unreadable scope restarts that interval.

The workflow refuses readiness when a resource operation lacks terminal Activity Log state.

It also refuses readiness when an assignment operation lacks stable direct proof.

Close and sweep wait for the recorded open workflow to finish or reach a terminal GitHub conclusion.

They require every resource operation to reach Succeeded, Failed, or Canceled.

They require every role-assignment journal entry to reach its expected stable presence or absence.

They also query for unjournaled session-tagged Activity Log events from the recorded start time.

After destroy, run deterministic inventory, role-assignment, deployment, and Activity Log checks for 15 continuous minutes.

Use one-minute reads and require all 15 reads to remain empty and terminal.

Any late resource, new operation, ingestion gap, unknown status, or unreadable log restarts the interval.

Cap total reconciliation at two hours.

After that cap, publish only rollback_failed and keep the capacity slot.

Close, sweep, and zero-resource proof must reconcile every late result before a tombstone or terminal status.

17.3 Close capacity release

  1. Set closing with the ETag retry loop before the operation lease.
  2. Let the proxy close active streams.
  3. Cancel the recorded open workflow when it still runs.
  4. Deallocate the session VM and remove discovered role assignments.
  5. Acquire the session operation lease.
  6. Destroy only the selected state.
  7. Reconcile every journaled and newly discovered Azure operation.
  8. Prove the 15-minute empty quiescence interval.
  9. Create or verify the exact permanent replay tombstone with If-None-Match: *.
  10. Write terminal state, proof fields, and cleanup_verified=true atomically under the status ETag.
  11. Reread and validate the complete terminal record.
  12. Acquire the capacity-ledger lease.
  13. Verify the exact session, request, epoch, tombstone, terminal record, journal, and quiescence proof.
  14. Remove that entry through a leased If-Match update.
  15. Reread the ledger and verify the slot is free.

A failed cleanup retains its occupied ledger slot.

17.4 Orphan recovery

The sweep checks occupied ledger slots without current status.

It inspects state and Azure resources.

It never frees an orphan slot without zero-resource proof.

It also reconciles duplicate or mismatched status by session_id.

18. Open flow

  1. Cloudflare authenticates the browser.
  2. The broker validates the JWT again.
  3. The broker validates sub, email, issuer, audience, token type, and time claims, then derives the subject key.
  4. The broker validates Origin and CSRF tokens.
  5. The broker reads the selected service profiles and their base profile at one exact control-repository revision.
  6. The broker defaults each selected service to read and validates the write-selected subset.
  7. The broker resolves effective namespace access and validates the scope, time limit, and SSH fields.
  8. The broker verifies certificate validity exceeds the requested time plus one hour.
  9. The broker performs a non-authoritative capacity precheck.
  10. The broker creates an idempotent accepted receipt with its open-audit fields set to pending.
  11. The broker appends the deterministic open_accepted audit event.
  12. The broker changes the receipt audit state to committed through its ETag.
  13. The broker signs the request through Key Vault.
  14. The broker dispatches open.yml on control main.
  15. GitHub returns the workflow run identifier.
  16. The broker updates the request receipt to dispatched.
  17. The browser opens the durable status page.
  18. The runner verifies the envelope before Azure login.
  19. The runner acquires the session lease and checks tombstone, status, and ledger replay facts.
  20. A known matching session exits idempotently without resource creation.
  21. The runner checks that no close revocation exists.
  22. The runner requires the exact request receipt and committed open-audit digest.
  23. The runner verifies the signed VM evidence, seal, live policy allowlist, exact Gallery version, client-root lock, and active proxy leaf SPKI.
  24. The runner verifies the standing VNet, subnet, peering, CNI, and route facts.
  25. The runner reserves capacity atomically.
  26. The runner appends the deterministic capacity_reserved audit event.
  27. An audit failure starts failed-open cleanup before any VM creation.
  28. The runner writes building and rechecks revocation before Terraform.
  29. Terraform applies the identity-only session phase under the revocation watchdog.
  30. The runner copies and recomputes the complete resolved scope, then creates and keyless-signs the typed session-bootstrap envelope.
  31. The runner saves and verifies the compute-phase plan with the exact envelope digest.
  32. Terraform applies the isolated session compute phase.
  33. The runner rechecks revocation at every provisioning stage boundary.
  34. The box verifies the envelope and resolved scope before any interactive service starts.
  35. The box generates its server private key.
  36. The runner retrieves only the public certificate.
  37. The runner publishes the certificate fingerprint.
  38. The runner completes local and Kubernetes readiness checks.
  39. The runner writes verifying with a one-time probe nonce.
  40. The proxy proves the real mutual-TLS and PTY path.
  41. The runner rereads status and requires the same verifying ETag.
  42. The runner rechecks revocation and appends the successful readiness audit event.
  43. The runner publishes ready with a conditional write.
  44. The browser enables the workspace action.

A signing or dispatch failure changes only the request receipt to dispatch_failed.

It creates no reservation or Azure resource.

An open-audit failure changes the receipt to audit_failed after three bounded retries.

It never dispatches that request.

Session creation returns a durable session after dispatch. The browser does not hold the provisioning workflow open. A developer can leave and resume the status page. Email readiness alerts and other durable notifications remain a later delivery after the priority create, use, and close path passes.

19. Readiness

Terraform success is not readiness.

The runner must prove:

  • Cloud-init completed.
  • The exact image version booted.
  • The proxy reaches the agent through the Firecall private address.
  • The session public address provides explicit outbound connectivity.
  • Public port 8443 remains unreachable.
  • The managed identity can authenticate.
  • The loopback kube proxy authenticates upstream with the managed identity.
  • The interactive user cannot reach Instance Metadata Service.
  • The kubeconfig contains only the box-local bearer token.
  • Every allowed namespace can list pods.
  • Every allowed namespace can read logs.
  • An unlisted namespace is denied.
  • Cluster-wide pod listing is denied.
  • The agent is healthy.
  • The server private key never left the box.
  • Mutual TLS succeeds from the proxy path.
  • The server certificate fingerprint matches status.
  • A terminal PTY opens.
  • tmux preserves a test window.
  • The proxy readiness attestation matches the one-time nonce.
  • The local expiry timer is active.

The runner publishes ready only after all probes succeed.

20. Close and expiry

20.1 Early close

  1. The requester or platform owner submits close.
  2. The broker verifies authorization.
  3. The broker creates a durable close-revocation marker with a pending close-audit item.
  4. The portal shows Close requested.
  5. The proxy closes active connections and sends the fixed mutual-TLS session-revoke call within five seconds.
  6. The broker appends close_requested and records success through the marker ETag.
  7. An audit failure stays pending and never delays reduction.
  8. The broker signs a close envelope.
  9. The broker dispatches the close workflow.
  10. The control runner writes closing before it waits for the session lease.
  11. The runner removes optional SSH through one complete parent-NSG update.
  12. It proves that Internet port 22 is closed when SSH existed.
  13. Fixed VM Run Command invokes /usr/local/libexec/firecall-revoke <session_id>.
  14. That helper stops sshd, the kube proxy, the agent, sessiond, and interactive processes.
  15. The broker requests cancellation of the verified recorded open run without waiting for success.
  16. The runner deallocates the VM and removes discovered role assignments.
  17. Terraform destroys the selected session.
  18. The runner verifies zero resources.
  19. The runner creates or verifies the permanent replay tombstone.
  20. The runner atomically writes closed, proof fields, cleanup_verified=true, and a pending cleanup-audit item.
  21. The runner releases capacity.
  22. The runner attempts the cleanup audit and records success through the status ETag.
  23. The normal control sweep clears the revocation only after terminal status and committed close-request audit.

The close confirmation shows the exact namespaces and expiry.

Browser workspace and preview access stop when the revocation marker appears.

The proxy's box-local revoke call is best effort and starts before workflow dispatch.

Optional SSH stops after the parent NSG update succeeds.

Box-local Kubernetes access stops after the revoke helper or VM deallocation succeeds.

If Run Command fails, the runner still deallocates the VM.

The parent NSG update and VM deallocation remain the authoritative SSH denial controls.

If both the box-local revoke call and control workflow fail, optional SSH can remain until recovery or local expiry.

In that state, Firecall reports denial as unproved and alerts immediately.

The system does not claim that all access stopped until these checks pass.

If marker creation fails, the broker does not claim immediate denial.

It still dispatches close, and access stops when the runner publishes closing.

If signing or dispatch fails after marker creation, browser and preview access remain denied.

SSH denial remains unproved when both box revocation and the control workflow fail.

The broker retries and alerts after the third failed dispatch.

The broker owns human close-audit retries in the revocation marker.

The sweep owns expiry, lifecycle, and cleanup-audit retries in runner status.

Both retry after one, five, and thirty minutes, then on every sweep.

They never discard a pending audit item because an attempt limit passed.

20.2 Expiry

The broker expiry dispatcher scans active status every 30 seconds.

At exact expiry, it creates the deterministic revocation and dispatches the idempotent close workflow.

A per-session Blob lease prevents two broker replicas from dispatching different close requests.

The dispatcher has no Azure session-resource authority.

The scheduled sweep runs at minutes 5 and 35.

It creates or reuses a revocation marker with a deterministic pending expiry_started audit item.

It uses the same close function with terminal target expired.

After zero-resource proof, one conditional write publishes expired, proof fields, and cleanup_verified=true.

It continues after one session fails.

The proxy denies access from the exact expiry time.

The box agent stops terminals at the exact expiry time.

Cloud-init installs a root-owned systemd expiry timer.

The firecall user cannot change that timer.

The timer stops the agent and SSH, then powers off the box.

Readiness verifies time synchronization and the exact timer deadline.

The sweep later removes Azure resources and roles.

The workspace derives Access expired · cleanup pending after expires_at.

Do not claim immediate Azure token revocation.

The interactive user cannot obtain the managed-identity token through the supported design.

The root kube proxy remains a token holder.

Microsoft documents a 24-hour lifetime for managed-identity tokens.

A root compromise can export a token immediately before identity deletion.

That stolen token can remain useful until its original exp time, even after identity deletion.

After successful identity deletion, this residual window is at most 24 hours from the last token issue.

For effective write scope, deleting the Firecall identity does not revoke a copied namespace Secret.

It also does not revoke a ServiceAccount token copied from a tenant workload.

The one-hour TokenRequest cap bounds new covered tokens but cannot retract an existing value.

The close proof reports Firecall path revocation and never reports complete information revocation.

A developer-created Pod or controller can refresh tokens after Firecall closes.

Its authority can persist indefinitely until the namespace owner removes that workload.

After a write session, the operator reviews effective write namespace audit events and new or changed Pods, controllers, Secrets, and ServiceAccount references.

Firecall cannot provide command attribution or an automatic safe rollback for those mutations.

The broker targets cleanup dispatch within 30 seconds of expiry.

The scheduled sweep provides a second dispatch within 30 minutes, plus GitHub and Azure processing time.

There is no fixed cleanup bound during a GitHub, Azure control-plane, or storage outage.

Therefore, a root compromise has no fixed residual bound until cleanup succeeds.

This is an explicit owner-accepted residual risk required before activation.

The portal labels exact terminal expiry separately from pending Azure cleanup.

20.3 Zero-resource proof

Verify absence of:

  • VM.
  • NIC.
  • Public IP.
  • OS disk.
  • Disk snapshot.
  • Restore point or backup item.
  • Managed identity.
  • Cluster User assignment.
  • Every namespace Reader or Writer assignment.
  • NSG.
  • Any optional SSH rule.
  • Any session preview process.
  • Any Terraform-managed session resource.

20.4 Reduction-only break glass

The breakglass.yml workflow accepts one exact session identifier.

It can operate when lifecycle status is unavailable.

It reads Terraform state directly when available.

It also discovers resources by deterministic name and session tags.

It first deallocates the VM and removes discovered AKS role assignments.

It then destroys only resources for the selected session.

If storage remains unavailable, it cannot release capacity or publish a terminal state.

It records that proof after storage recovers.

Break glass cannot open a session, change a profile, or add authority.

21. Network design

21.1 Public browser path

Use one dedicated Firecall tunnel.

Do not reuse the ScreamingFace tunnel.

Use ordered routes:

portal.FIRECALL_APEX         -> broker HTTPS Service
*.FIRECALL_APEX              -> terminal-proxy HTTPS Service
*.FIRECALL_PREVIEW_APEX      -> terminal-proxy HTTPS Service
fallback                     -> HTTP 404

Use two core Cloudflare Access applications:

  • Exact portal hostname.
  • Wildcard workspace hostname.

Use a third wildcard Access application for previews.

The broker and proxy validate Access JWTs at the origin.

Tunnel placement alone is not authentication.

21.2 Cluster origin TLS

Use HTTPS from cloudflared to both Services.

Use fixed Service port 8443 for both origins.

Both Services use ClusterIP only.

Create no Firecall LoadBalancer or NodePort Service.

Mount the reviewed immutable digest-named origin-root ConfigMap into cloudflared.

The mounted caPool contains only the active and standby origin roots.

Validate the origin server name.

Never set noTLSVerify.

Keep both fixed non-exportable origin CA signing keys in Azure Key Vault.

A protected certificate rotator generates each leaf private key in memory.

It asks Key Vault to sign broker and proxy certificate requests.

It writes only the leaf key and certificate to their separate Kubernetes Secrets.

Broker and proxy certificates use their exact Kubernetes Service DNS names.

Each certificate lasts 30 days and rotates after 15 days.

cloudflared trusts only the origin CA public certificate.

Rotation tests prove bundle rollout and renewal without noTLSVerify or an origin outage.

Remove an old origin root only after all serving leaves use the new root.

21.3 Proxy-to-box path

The Firecall control stack creates one standing 10.240.0.0/24 VNet.

It creates one 10.240.0.0/27 session subnet.

That stack creates only the Firecall-side peering link.

The dedicated firecall-aks/dev stack owns the AKS-side peering link.

Set allow_virtual_network_access=true on both links.

Set allow_forwarded_traffic=false, allow_gateway_transit=false, and use_remote_gateways=false on both links.

Require both provisioning states to be Succeeded and both peering states to be Connected.

Require both sides to report synchronized local and remote address spaces before session creation.

Peering has no public Internet route between the proxy and box.

Per-session Terraform references these standing resources as data.

Per-session Terraform never owns or destroys them.

Each session creates its own NIC and attaches it to that subnet.

Each session attaches its own NSG to its NIC.

Each session also creates one Standard static IPv4 public address.

That address gives the development box explicit public TCP 443 egress.

Normal tools use HTTPS. The network controls do not enforce TLS or HTTP.

It is also the optional SSH destination.

The proxy never uses that public address.

Azure CNI Overlay translates proxy Pod traffic to the AKS node private address.

Use these exact custom NSG rules:

Name Priority Direction Access Protocol Source Destination Port
allow-agent 100 Inbound Allow TCP 10.224.0.0/16 Any 8443
allow-ssh 110 Inbound Allow TCP Approved IPv4 /32 Any 22
deny-vnet-in 200 Inbound Deny Any VirtualNetwork Any Any
deny-other-in 300 Inbound Deny Any Any Any Any
deny-vnet-out 100 Outbound Deny Any Any VirtualNetwork Any
allow-dns-udp 110 Outbound Allow UDP Any AzurePlatformDNS 53
allow-dns-tcp 111 Outbound Allow TCP Any AzurePlatformDNS 53
allow-imds 120 Outbound Allow TCP Any AzurePlatformIMDS 80
allow-wireserver-http 121 Outbound Allow TCP Any 168.63.129.16/32 80
allow-wireserver-agent 122 Outbound Allow TCP Any 168.63.129.16/32 32526
allow-internet-https 130 Outbound Allow TCP Any Internet 443
deny-other-out 200 Outbound Deny Any Any Any Any

Create allow-ssh only when the reviewed request enables SSH.

The other eleven custom rules are mandatory.

deny-other-in overrides Azure's default priority-65001 AllowAzureLoadBalancerInBound rule.

Do not permit another custom rule.

Azure treats 168.63.129.16 as a platform virtual address.

Azure does not subject traffic to that address to user-defined routes or NSGs.

The two WireServer NSG entries document the intended contract but do not enforce it.

The sealed VM image uses host nftables to permit only TCP 80 and 32526 to that address.

Both permits require the measured Azure Guest Agent service UID and its exact protected systemd cgroup.

The rules deny the interactive, Firecall agent, session supervisor, kube proxy, and every other service cgroup.

It denies every other guest connection to 168.63.129.16 except Azure DNS through the reviewed resolver path.

The first Packer provisioner applies the same host-firewall rule after the marketplace Azure guest agent has already started.

Section 10.6 records that bounded builder bootstrap residual.

The captured image orders its firewall unit before Azure Guest Agent on every later session boot.

Qualification records the Azure Guest Agent UID and cgroup path from the selected marketplace image.

It fails when either value changes.

Enable the Azure Metadata Security Protocol when the selected image and Azure Guest Agent support it.

Treat that protocol as defense in depth, not as a replacement for the host firewall.

Create every NSG rule inline before Terraform creates the NIC.

Then attach that complete NSG to the one exact session NIC.

Destination Any is safe because the NSG attaches only to that NIC.

This order avoids a rule dependency on a dynamic private address.

This is the exact live AKS node subnet.

Mutual TLS is the second control.

The source subnet belongs to all current AKS nodes.

It is not a proxy identity boundary.

Mutual TLS, the proxy certificate fingerprint, and proxy NetworkPolicy provide the identity controls.

The proxy also pins the per-box server certificate.

The proxy validates the status private address against 10.240.0.0/27.

It ignores the public address for agent routing.

The control runner stops when VNet, subnet, CIDR, peering, CNI, or route facts drift.

Standing-resource qualification proves the complete static VNet, subnet, peering, CNI, route, and NetworkPolicy contract without an endpoint.

The first Group F owner canary supplies the first real 10.240.0.0/27:8443 endpoint through the normal session Terraform path.

Before that canary, permit one network-proof bootstrap only when the requester is a current platform owner.

Also require zero v2 sessions, limit two, an enabled qualified profile, and no marker for the protected network-path generation.

The runner still verifies every static network fact before reservation.

It publishes ready only after the proxy reaches that box through mutual TLS from the measured AKS node-subnet source.

On success, create one legal-held network-path statement bound to peering IDs, subnet IDs, CNI mode, source CIDR, endpoint, image, and time.

Its path contains the protected generation and statement digest, and If-None-Match: * prevents replacement.

On failure, keep the marker absent, deny all later nonbootstrap opens, and complete failed-open cleanup.

Every later reservation requires that marker and current matching network facts.

Do not permit Internet traffic to port 8443.

Add an explicit NSG deny for other VirtualNetwork inbound traffic.

This rule overrides Azure's default same-VNet allow rule.

The current AKS API is public and has no source-address allowlist.

The HTTPS rule reaches its public endpoint.

The kube proxy validates the exact hostname and certificate.

The managed identity and Kubernetes RBAC authorize each request.

The session public address gives each box a distinct source for logs and future API allowlisting.

The explicit rules deny same-VNet traffic and other public ports.

Public TCP 443 permits Secret exfiltration or another tunneled protocol from a compromised box.

Read scope permits exfiltration of readable data. Write scope also exposes Secrets and ServiceAccount authority. Each resolved session scope accepts this development risk.

21.4 Optional SSH

Do not create port 22 rules by default.

Without SSH, the session NSG has no Internet inbound allow rule.

When enabled, require:

  • One valid public key.
  • Proof of possession for that private key.
  • The exact Cloudflare-validated requester IPv4 /32.
  • A profile that permits SSH.

The initial profiles keep optional SSH disabled.

An owner can enable it in one reviewed profile after accepting the remaining key-sharing residual.

The broker creates a 256-bit challenge bound to requester subject key, request, key fingerprint, source /32, and ten-minute expiry.

It verifies one OpenSSH sshsig signature with namespace firecall-v2-request before dispatch.

Reject a replay, another namespace, another key, another subject key, another request, another address, or an expired challenge.

A requester can still share the proved private key with a person behind the same public /32.

The profile owner must accept that identity-bypass residual before SSH becomes available.

The box has no password authentication.

The root boot helper generates one Ed25519 SSH host key on the box.

No SSH host private key enters cloud-init, Terraform state, status, or workflow output.

The runner retrieves only the canonical public host key through the fixed Run Command path.

It validates the algorithm and key length, then records the public key and SHA-256 fingerprint in ETag-protected runner status.

Stop and disable sshd when SSH is not requested.

Set these fixed server controls when SSH is enabled:

PermitRootLogin no
AllowUsers firecall
PubkeyAuthentication yes
AuthenticationMethods publickey
PasswordAuthentication no
KbdInteractiveAuthentication no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no
X11Forwarding no
GatewayPorts no
PermitOpen none

Remove every image-builder, default-user, and cloud-init authorized key before image sealing.

At session boot, remove Azure's one ephemeral firecall-provision key and lock that account before readiness.

Store the one session key in /etc/ssh/firecall_authorized_keys as a root-owned mode-0600 file.

The root finalization helper writes fixed from="<validated-ip>",restrict,pty options with the proved canonical key.

The pty override restores only interactive terminal allocation after restrict denies forwarding and user RC files.

Configure AuthorizedKeysFile /etc/ssh/firecall_authorized_keys and no second key source.

The interactive user cannot read, write, replace, or add an authorized key.

Do not permit local, remote, dynamic, agent, X11, or network-device forwarding.

The ready page shows a session-specific known_hosts command and exact SSH command to the requester only.

The first command creates a mode-0600 session file with the exact public address and validated host key.

The SSH command sets UserKnownHostsFile to that file and sets StrictHostKeyChecking=yes.

It also sets UpdateHostKeys=no, GlobalKnownHostsFile=/dev/null, and HostKeyAlgorithms=ssh-ed25519.

The page shows the same SHA-256 fingerprint from runner status beside both commands.

Do not offer trust-on-first-use, accept-new, or disabled host-key checking.

That command uses the session public address, never the private proxy address.

Browser file transfer remains deferred.

Optional SSH permits SCP and SFTP within the session's existing authority.

21.5 Kubernetes network policy

Use separate namespaces:

firecall
firecall-edge

Apply default-deny ingress and egress.

Allow only:

  • cloudflared to broker and proxy Services.
  • Broker directly to the internal proxy revocation Service.
  • Broker, proxy, readiness worker, certificate rotator, and credential rotator to their controlled-egress listeners.
  • Each credentialed Firecall component to only its DNS policy-proxy listener.
  • cloudflared to only its separate DNS policy-proxy listener.
  • The DNS policy proxy to the cluster resolver.
  • Proxy directly to 10.240.0.0/27 on TCP 8443.
  • Proxy to the exact fixed certificate-verifier Service on TCP 9446.
  • Readiness worker directly to the internal proxy probe Service.
  • Certificate rotator directly to the Kubernetes API Service.
  • Credential rotator directly to the Kubernetes API Service.
  • Binding validator directly to the Kubernetes API Service on TCP 443.
  • The exact kube-system Konnectivity tuple to the binding validator on TCP 9443.
  • Certificate rotator directly to certificate-health ports 9444 and 9445.
  • Credential rotator directly to credential-health ports 9544 and 9545.
  • The egress gateway to public TCP 443, excluding every cluster, private, loopback, link-local, metadata, multicast, and reserved range.
  • cloudflared to the reviewed Cloudflare Tunnel IPv4 and IPv6 ranges on TCP 443 and TCP or UDP 7844.
  • The credentialless HTTPS egress gateway to the cluster resolver.
  • Exact telemetry sources to the exact SigNoz collector on OTLP gRPC.

The telemetry sources are broker, proxy, egress gateway, DNS proxy, readiness, both rotators, binding validator, and connector-health sidecar.

Select each source by its exact namespace, ServiceAccount, protected image label, and workload owner.

The only destination namespace is signoz with kubernetes.io/metadata.name=signoz.

The only destination Pods have app.kubernetes.io/name=signoz and app.kubernetes.io/component=otel-collector.

The only destination Service is signoz-otel-collector.signoz.svc.cluster.local on TCP 4317.

Permit no collector CIDR, TCP 4318, namespace-wide destination, or source selected only by a mutable application label.

Update the SigNoz ingress NetworkPolicy to accept only those exact Firecall namespace and protected-source selectors on TCP 4317.

The rendered test resolves the Service to only matching endpoint Pods and proves every other destination and source fails.

Standard NetworkPolicy cannot filter public hostnames.

The live cluster has no paid FQDN or layer-seven network-policy feature.

Do not enable Advanced Container Networking Services only for Firecall.

Use standard Kubernetes NetworkPolicy for normal Firecall layer-three and layer-four controls.

The live AKS callback uses kube-system Konnectivity Pods.

The observed tuple is ServiceAccount konnectivity-agent with labels app=konnectivity-agent and component=tunnel.

Use one CiliumNetworkPolicy source selector for that exact namespace, ServiceAccount, and two labels.

Do not rely on Cilium's kube-apiserver entity for this ingress path.

Konnectivity can hide the original API-server source from Cilium on managed Kubernetes.

Run a real callback before the Firecall pool exists and bind signed evidence to the observed source identity.

A tuple change blocks only the Firecall-prefix webhook until an owner reviews the replacement.

Do not add a host, remote-node, cluster-wide kube-system, or API-server-entity fallback.

Use the controlled egress gateway for exact public HTTPS destinations.

Give the egress gateway no credential and disable its ServiceAccount token mount.

Application Pods cannot connect directly to arbitrary public TCP 443 destinations.

The egress gateway cannot reach any private or cluster destination.

Store the exact Entra, GitHub, Cloudflare, Key Vault, storage, and internal DNS-name sets in reviewed configuration.

The credential-rotator listener permits only Entra, Key Vault, and Firecall Storage FQDNs.

Certificate-health ingress accepts only the certificate-rotator ServiceAccount and protected image label.

Certificate-verifier ingress on TCP 9446 accepts only the proxy ServiceAccount and protected proxy image label.

Credential-health ingress accepts only the credential-rotator ServiceAccount and protected image label.

Admission binds that label to its exact namespace, ServiceAccount, Job owner, and digest.

Store the official Cloudflare Tunnel IP ranges in separate reviewed configuration.

A scheduled unprivileged check reports drift in those published ranges.

Test DNS resolution, DNS-exfiltration denial, direct-bypass denial, and every required connection against the rendered final policies.

21.6 Kubernetes runtime posture

Run two broker replicas, two proxy replicas, and two egress-gateway replicas.

Run two cloudflared replicas.

Create one firecall node pool with:

  • One Standard_D2ds_v5 node.
  • One 64-GiB managed OS disk.
  • Maximum 30 Pods.
  • Exact vnet_subnet_id set to the live aks-subnet resource ID.
  • Label openmined.org/pool=firecall.
  • Taint workload=firecall:NoSchedule.
  • Fixed size one with no autoscale.
  • One temporary surge node during an AKS node upgrade.

The exact subnet resource ID is:

/subscriptions/f1ade722-e418-4647-8094-3f73664dd6a5/resourceGroups/MC_rg-aks-platform-dev-eastus_aks-dev-eastus_eastus/providers/Microsoft.Network/virtualNetworks/aks-vnet-30719917/subnets/aks-subnet

Terraform reads that existing subnet and does not own it.

Assign the standing-resource identity Microsoft.Network/virtualNetworks/subnets/read and Microsoft.Network/virtualNetworks/subnets/join/action at only that exact subnet resource ID.

Grant no parent managed-resource-group, VNet, subnet-write, route-table, NSG, or peering action at that scope.

Run one positive linked-authorization canary for that exact subnet before the saved pool plan.

Run negative canaries against the sibling subnet, VNet parent, route table, NSG, peering, and every subnet write operation.

The plan fails if its address prefix is not 10.224.0.0/16.

Do not let AKS select a default or new subnet for this pool.

After creation, verify the pool VM scale-set NIC configuration references this exact subnet.

Verify each Firecall node InternalIP is inside 10.224.0.0/16.

The first Group F owner canary verifies that its proxy path uses a source inside this exact subnet.

Activation computes quota separately for coexistence and post-cutover.

It checks the BS-family, DDSv5-family, and Total Regional vCPU quotas.

It also checks the East US IPv4StandardSkuPublicIpAddresses quota.

It fails unless the current phase, live use, fixed node, allowed sessions, and one surge fit both quotas.

Each session open repeats both reads immediately before reservation and immediately before VM creation.

Each session quota read reserves one Standard public address and counts occupied reservations not yet visible in Network usage.

It adds occupied reservations whose VM usage has not appeared in Azure usage yet.

The image workflow checks both DDSv5-family and Total Regional vCPU quota before its builder VM creation.

It also reserves one Standard public address in the live Network usage calculation.

Activation requires room for three session addresses, one concurrent builder address, and two unallocated safety addresses.

The control and builder identities use only Microsoft.Network/locations/usages/read at subscription scope for this check.

Azure remains the final concurrency guard when an unrelated subscription workload races either read.

Every Firecall runtime Pod uses that node selector and toleration.

The binding validator remains on the existing AKS system pool to avoid a scheduling trust cycle.

No tenant or preview Pod receives that toleration.

Treat the taint as scheduling control, not a security boundary.

Add one cluster-scoped ValidatingAdmissionPolicy for Firecall-node isolation.

Its non-system branch rejects every tenant, preview, and interactive Pod with any Firecall targeting mechanism.

That branch covers:

  • openmined.org/pool=firecall in nodeSelector.
  • Required or preferred node affinity for that label.
  • A workload=firecall Equal or Exists toleration.
  • A broad empty-key Exists toleration that covers NoSchedule.
  • A direct spec.nodeName with the verified Firecall-node prefix.

Approve Firecall application Pods only through an exact namespace, ServiceAccount, protected label, controller owner kind, and image-digest tuple.

Use a separate kube-system branch for DaemonSet-owned Pods with direct spec.nodeName.

Apply its tuple check only when that node name has the verified Firecall-node prefix.

A managed DaemonSet Pod for another node does not enter the Firecall tuple branch.

Each approved system tuple fixes namespace, ServiceAccount, DaemonSet owner name, and every declared container image reference.

The live AKS system DaemonSets currently declare mutable tags, not digest references.

For example, they include azure-cns:v1.8.11 and cilium-distroless:v1.18.11-260622.

A ValidatingAdmissionPolicy sees spec.image but cannot see the later resolved status.imageID.

Therefore, admission must compare the exact reviewed live spec.image values.

After scheduling, qualification reads every init-container and container status.imageID on the Firecall node.

Every resolved value must contain an immutable registry digest.

Signed evidence binds the reviewed spec reference, resolved digest, DaemonSet UID, Pod UID, and node UID.

A mutable AKS system tag can change between tuple review and image pull.

This supply-chain residual belongs to the managed AKS system plane and blocks a stronger admission guarantee.

Activation derives that list from the live cluster and requires owner review.

A new or changed Firecall-node system tuple blocks Firecall activation until it receives review.

Before the first agent-pool PUT, deploy the isolation and scheduler-identity policies and their bindings.

Load the reviewed system tuples and run admission-only positive and negative probes.

The positive pre-pool Pod uses a synthetic Firecall-prefix node name.

It must pass admission, remain Pending, and be deleted without a bind.

No pre-pool test claims a successful schedule.

firecall-standing-bootstrap.sh fails before Azure planning unless every precondition passes against the live API.

Treat every AKS upgrade or managed-DaemonSet template change as a Firecall maintenance event.

After pool creation, read the real node names and verify the planned prefix matches every Firecall node only.

Deploy a small firecall-node-binding-validator admission webhook before any Firecall application replica starts.

Run two replicas on the existing AKS system pool during bootstrap and steady state.

Pin their node selector to the verified live system-pool label and exclude preview nodes.

Give only these two validator Pods this exact existing system-pool toleration:

tolerations:
  - key: CriticalAddonsOnly
    operator: Equal
    value: "true"
    effect: NoSchedule

The live system pool has taint CriticalAddonsOnly=true:NoSchedule.

Bind the toleration, system-pool label, ServiceAccount, controller UID, image digest, and resource limits into admission and tuple evidence.

Create the Service, certificate, and two validator Pods before the ValidatingWebhookConfiguration exists.

Require both Pods ready and prove the live Konnectivity callback path.

Use preferred Pod anti-affinity because the current system pool has one node.

Record loss of both replicas during that node outage as an accepted fail-closed availability residual.

Only then create the fail-closed webhook configuration and run positive and negative Binding probes.

If either validator cannot schedule, do not create the webhook configuration or any Firecall runtime replica.

Give them no Azure, storage, Key Vault, GitHub, or Cloudflare credential.

Their Kubernetes ServiceAccount can use name-based get on Pods and Nodes needed for a selected Binding.

Kubernetes returns the complete Pod object, including literal commands and environment values.

Grant no list, watch, Secret, log, exec, attach, port-forward, or status-write operation.

Never return or log a Pod specification, command, environment value, annotation, or Secret reference.

Create a namespace-scoped self-signed cert-manager Issuer and one fixed Certificate for the webhook Service.

Use cert-manager CA injection on only its exact ValidatingWebhookConfiguration.

Activation requires the observed cert-manager v1.17.2 components, ready Certificate, and matching injected CA bundle.

Its webhook match conditions select only Binding targets with that exact Firecall prefix.

Cover both core bindings and the pods/binding subresource.

Normalize both request shapes to namespace, Pod name, live Pod UID, target kind, target API version, and target node.

Require the exact live scheduler username in the webhook and scheduler-identity policy.

Require target.kind=Node, target.apiVersion=v1, and one exact Firecall node.

Fetch the current Pod by name and require its exact UID before any tuple decision.

Reject a supplied metadata UID when it differs from that live UID.

The webhook reads the target Pod and Node through a read-only ServiceAccount.

It validates scheduler-managed Firecall applications and reviewed kube-system tuples at bind time.

Other-node Bindings never call this webhook.

This check covers scheduler-managed system Pods with broad tolerations without blocking their use on other nodes.

After the pool exists, run one real scheduler bind and one real managed-DaemonSet creation test.

Verify the scheduled Pod UID, Node UID, owner tuple, and all resolved image digests.

A scheduled tuple monitor compares namespace, ServiceAccount, owner, Pod template, image references, requests, and DaemonSet UID.

For a planned change:

  1. Merge creation_enabled=false and reconcile the capacity ledger.
  2. Close or expire every Firecall session before the maintenance window.
  3. Scale Firecall application workloads to zero through protected source.
  4. Capture the reviewed old and proposed managed-system tuples.
  5. Add only those two exact tuple generations through a protected pull request.
  6. Run admission denial tests before AKS maintenance starts.
  7. Let AKS replace managed Pods and the Firecall node.
  8. Verify every new Pod UID and resolved imageID digest.
  9. Remove the old tuple generation through another protected pull request.
  10. Restore workloads and creation only after full preflight passes.

AKS can change a managed DaemonSet without advance notice.

In that case, admission rejects only the changed replacement that directly targets a Firecall node.

The binding webhook rejects only a changed scheduler-managed Pod when the scheduler selects a Firecall node.

Other-node Pod creation and binding remain outside these Firecall-prefix checks.

Firecall remains unavailable until an owner reviews and merges the new exact tuple.

Do not use a wildcard, namespace-only, ServiceAccount-only, or temporary admission bypass for recovery.

Count every approved system DaemonSet request in the node capacity proof.

Tenant and preview namespaces receive no exception, including Pods with direct nodeName.

Add one ValidatingAdmissionPolicy for Secret create and update in every namespace that an enabled service profile can resolve as write.

It denies type kubernetes.io/service-account-token and any legacy ServiceAccount-token annotation combination.

Add another policy for CREATE on the serviceaccounts/token subresource in those namespaces.

Require a complete Pod boundObjectRef and expirationSeconds between 600 and 3600.

Reject an unbound token, another bound kind, a missing UID, or a duration above one hour.

Activation requires both policies and bindings to compile and pass live kubelet projected-token tests.

These controls do not revoke an already copied Secret or a token copied from an existing workload.

Add a second cluster ValidatingAdmissionPolicy for CREATE on core bindings and pods/binding.

Permit only the exact live AKS scheduler username to create a Binding.

Do not permit a Firecall box identity, namespace Writer, tenant ServiceAccount, or human to create one.

This policy blocks a pending Pod from direct binding after its Pod admission check.

The Firecall-prefix webhook then validates the bound Pod tuple and exact target node.

Set its failure policy to Fail only for Firecall-prefix targets.

Do not add a wildcard, all-Binding, or all-kube-system webhook rule.

Set matchPolicy: Equivalent, sideEffects: None, admissionReviewVersions: [v1], and timeoutSeconds: 2.

Limit operations to CREATE on core bindings and pods/binding.

Use one CiliumNetworkPolicy with the exact reviewed Konnectivity source tuple to Pod port 9443.

Permit validator egress only to the Kubernetes API Service and the exact section 29 SigNoz collector.

Test the callback and validator API read while default-deny policies are active.

Tenant, preview, Firecall runtime, and interactive Pods cannot reach the webhook Service.

Activation blocks if Cilium cannot identify the exact reviewed Konnectivity namespace, ServiceAccount, and labels.

One node does not provide node-level high availability.

Two replicas still permit process replacement and one-at-a-time rollout.

Use maxSurge: 0 and maxUnavailable: 1 for these two-replica Deployments.

This rule includes the cloudflared Deployment.

The connector-token workflow deletes one Pod at a time under the unchanged ReplicaSet.

Use these initial container resources:

Component Replicas CPU request Memory request CPU limit Memory limit
Broker 2 50m each 128Mi each 500m each 256Mi each
Proxy 2 100m each 192Mi each 1 each 512Mi each
HTTPS egress gateway 2 50m each 64Mi each 500m each 256Mi each
DNS policy proxy 2 20m each 32Mi each 200m each 128Mi each
cloudflared 2 50m each 64Mi each 500m each 128Mi each
Origin-trust health sidecar 2 10m each 32Mi each 100m each 64Mi each
Readiness worker 1 25m 64Mi 250m 128Mi
Certificate CronJob 1 while active 25m 64Mi 250m 128Mi
Certificate-bootstrap Job 1 while active 25m 64Mi 250m 128Mi
Credential-rotation Job 1 while active 25m 64Mi 250m 128Mi
Preflight Pod 1 while active 25m 32Mi 100m 64Mi
Load-qualification Pod 1 while active 100m 32Mi 1800m 64Mi
Binding validator on the system pool 2 25m each 64Mi each 250m each 256Mi each

The steady Firecall-node request is 585m CPU and 1088Mi memory.

The system-pool validator adds 50m CPU and 128Mi memory.

The complete steady request is 635m CPU and 1216Mi memory.

The Firecall node has 11 steady application Pods.

The declared peak adds one connector surge Pod, three one-shot or CronJob Pods, and one load-qualification Pod.

That peak is 16 Firecall-node Pods, 820m CPU, and 1408Mi memory before managed DaemonSets.

The system-pool validator raises the cluster-wide declared peak to 870m CPU and 1536Mi memory.

Qualification mode blocks preflight, rotation, certificate maintenance, and connector rollout while the load Pod exists.

Use these initial namespace ResourceQuota ceilings:

Namespace Pods CPU requests Memory requests CPU limits Memory limits
firecall 20 1250m 2Gi 8 8Gi
firecall-edge 6 500m 1Gi 3 2Gi
Each preflight fixture 2 100m 128Mi 500m 256Mi

The quotas limit namespace growth but do not prove node capacity.

Qualification budgets the connector surge, three maintenance Pods, one preflight Pod, and every managed DaemonSet under maxPods: 30.

Verify actual node allocatable capacity and DaemonSet overhead before activation.

Before the AKS saved plan, query the subscription-scoped AKS VM-SKU endpoint for East US.

Use only Microsoft.ContainerService/locations/eastus/vmSkus?api-version=2026-05-02-preview.

Require exactly one Standard_D2ds_v5 result for East US with no restriction.

Bind the API version, authenticated Azure time, complete SKU result, canonical result digest, subscription, and region into activation evidence.

Treat this endpoint as an AKS acceptance check, not a capacity guarantee.

A missing SKU, restriction, location mismatch, duplicate, schema change, or unreadable response blocks the pool plan.

After pool creation, read the new D2ds v5 node's actual allocatable CPU, memory, and Pod count.

Require the Firecall-node peak plus managed DaemonSets to leave a 20-percent allocatable reserve in all three dimensions.

Do not use the existing D2as_v5 system-node allocatable values for that proof.

Check the validator's 50m CPU and 128Mi memory separately against measured system-pool headroom.

Standard_D2ds_v5 is non-burstable and has no CPU-credit balance.

Collect CPU utilization and readiness for the exact Firecall VMSS instance.

The node-tuple observer receives Microsoft.Compute/virtualMachineScaleSets/read and Microsoft.Insights/metrics/read on only that exact VMSS.

It receives no metric, alert, VMSS, autoscale, or node mutation action.

Bind the VMSS resource ID, metric namespace, query interval, values, Azure timestamps, and thresholds into each signed node-tuple verdict.

Block new session creation when either signal is missing or stale.

The broker and control runner enforce that result through the existing current node-tuple gate.

Close and sweep remain available during that gate.

Do not create disposable node pools or session VMs outside the normal authority paths.

After Group F raises normal capacity to three, enter the protected owner-only qualification mode.

Use the real Firecall node and three normal Standard_B2s sessions for the one-hour load test.

The fixed fc qualification cpu command creates only bounded unprivileged CPU workers and stops at its signed three-hour deadline.

One separate owner-approved load-qualification workflow creates the exact credentialless node-load Pod defined in section 24.1.3.

Create and deplete the three qualification sessions.

Keep the non-burstable node at the reviewed load target.

Run terminal, Kubernetes proxy, close, expiry, and local-revoke tests for one hour.

Record agent latency, terminal latency, API latency, expiry error, and cleanup time against fixed acceptance targets.

If the node misses its latency or readiness targets, disable creation and select a reviewed non-burstable SKU.

Recalculate quota, cost, Policy, image evidence, and profile pins before another owner qualification.

Apply the same replacement rule when one depleted session VM misses an agent, terminal, expiry, or cleanup target.

Do not silently select a larger node SKU when quota or availability fails.

Use rolling updates and readiness probes.

Give each two-replica Deployment a PodDisruptionBudget with minAvailable: 1.

For every pod:

  • Run as non-root.
  • Use a read-only root filesystem.
  • Drop all Linux capabilities.
  • Use RuntimeDefault seccomp.
  • Set CPU and memory requests and limits.
  • Disable ServiceAccount token mounting unless workload identity needs it.

Use separate ServiceAccounts for broker, proxy, egress, DNS proxy, readiness worker, certificate rotator, connector, and binding validator.

Use separate ExternalSecrets and materialized Secret names.

Argo owns only each ExternalSecret custom resource.

External Secrets Operator owns its target Secret with creationPolicy: Owner.

Do not render, pre-create, or import an External Secrets target Secret through Argo.

Do not add those target Secrets to Argo ignoreDifferences.

The ten generated certificate-leaf Secrets remain separate, pre-created, and rotator-owned.

Use primary and secondary Secrets for broker origin, proxy origin, proxy client, readiness client, and broker revoker.

The broker mounts only broker-origin and broker-revoker slots.

The proxy mounts only proxy-origin and proxy-client slots.

The readiness worker mounts only readiness-client slots.

Apply a ResourceQuota and LimitRange to both namespaces.

Argo CD owns the existence, metadata, and static specification of each rendered Firecall object.

Argo excludes and does not own the three cluster-owner-seeded coordination Leases.

The current dark Application keeps the live legacy fields and these exact ignores:

Object Runtime owner Ignored JSON pointers
Ten fixed primary and secondary certificate leaf Secrets Certificate rotator /data
firecall-certificate-active ConfigMap Certificate rotator /data/broker_origin_slot, /data/proxy_origin_slot, /data/proxy_client_slot, /data/readiness_worker_slot, /data/broker_revoker_slot, /data/rotation_epoch, /data/client_root_bundle_sha256, /data/origin_root_bundle_sha256, /data/proxy_spki_sha256, /data/worker_spki_sha256, /data/broker_revoker_spki_sha256
firecall-session-token-keys ConfigMap Credential-rotation workflow /data/active_kid, /data/previous_kid, /data/previous_valid_until, /data/denied_kids
firecall-broker-key-epochs ConfigMap Credential-rotation workflow /data/csrf_active_kid, /data/csrf_previous_kid, /data/csrf_previous_valid_until, /data/request_active_kid, /data/request_previous_kid, /data/request_previous_valid_until, /data/request_control_commit, /data/request_public_key_sha256
firecall-credential-bootstrap-status ConfigMap Credential-rotation workflow /data/state, /data/generation, /data/control_commit, /data/selectors_sha256, /data/completed_at
cloudflared-rollout-epoch ConfigMap Protected edge workflow /data/key_vault_version
cloudflared Deployment Protected edge workflow /spec/template/metadata/annotations/firecall.openmined.org~1tunnel-token-epoch

Migrate the three selector ConfigMaps through this reviewed seed-to-ignore sequence:

  1. Select the activation foundation and remove only the three legacy selector ignore entries in one protected revision.
  2. Let Argo replace each legacy field set with its canonical JSON document at wave -60.
  3. Let Argo create the selector-write admission policies at wave -55.
  4. Read each named ConfigMap twice and require one unchanged resource version and one exact JSON key.
  5. In a separate protected revision, add only the three target JSON-document ignore pointers.
  6. Confirm Argo is Healthy and confirm that all three ConfigMap resource versions and bytes remain unchanged.
  7. Start credential bootstrap and candidate consumers only after that exact readback passes.

Do not add a target JSON-document ignore pointer during the seed sync.

RespectIgnoreDifferences=true would omit that document before Argo applies it.

After step 5, runtime controllers own only these target mutable fields:

Object Runtime owner Ignored JSON pointer
firecall-certificate-active ConfigMap Certificate rotator /data/active.json
firecall-session-token-keys ConfigMap Credential-rotation workflow /data/selector.json
firecall-broker-key-epochs ConfigMap Credential-rotation workflow /data/active.json

For the exact firecall-node-binding-validator ValidatingWebhookConfiguration, ignore only these controller-owned jq paths:

.webhooks[].clientConfig.caBundle
.webhooks[].namespaceSelector.matchExpressions[] | select(.key == "control-plane")
.webhooks[].namespaceSelector.matchExpressions[] | select(.key == "kubernetes.azure.com/managedby")

Scope that Argo entry by exact group, kind, and object name.

Put all three expressions under jqPathExpressions, not jsonPointers.

The first path belongs to cert-manager CA injection.

The other two paths belong to the verified AKS admission enforcer.

Do not give another webhook configuration this ignore entry.

Create one dedicated kubernetes/appsets/firecall-apps.yaml ApplicationSet.

It renders only the exact firecall and firecall-edge Applications.

Exclude both entries from the existing platform-apps.yaml generator before this ApplicationSet becomes active.

Add firecall-apps.yaml to the explicit resources in kubernetes/appsets/kustomization.yaml.

The dedicated template carries every exact ignoreDifferences entry and RespectIgnoreDifferences=true.

Set ApplicationSet spec.syncPolicy.preserveResourcesOnDeletion=true.

Do not add a resources finalizer to either generated Application.

Deleting the ApplicationSet must preserve all live Firecall resources.

Use the retirement runbook to disable creation, close sessions, remove runtime replicas, remove the pool, and then delete resources deliberately.

Do not use ApplicationSet or Application deletion as a rollback or emergency control.

Do not rely on an Application field absent from the current shared ApplicationSet template.

Activation proves exactly one Application owns each Firecall destination and resource tree.

Set ignoreDifferences by exact group, kind, namespace, and object name.

Set RespectIgnoreDifferences=true on the two Firecall Argo applications.

Do not ignore an entire ConfigMap, metadata block, Secret object, or namespace.

GitOps pre-creates every listed runtime-owned object and mutable data key with a schema-valid placeholder, except the three coordination Leases.

The exact cluster-owner bootstrap creates those Leases after both dark namespaces exist and before activation admission.

It runs only from clean source at the exact commit used by both healthy Firecall applications.

It verifies the AKS resource, API endpoint, CA digest, kube-system UID, Argo Lease exclusion, namespaces, and complete closed activation map.

It creates only missing exact inert Leases and never updates an existing Lease.

Its --verify mode emits canonical evidence with every UID, resource version, and spec digest.

Activation source pins and verifies that evidence live before it registers create-denying admission.

The broker watches firecall-broker-key-epochs and reloads a complete resource version atomically.

It keeps the prior complete resource version when parsing or verification fails.

It signs requests only with request_active_kid.

The matching public key and digest must exist in the attested protected control commit.

It verifies the prior request key only through request_previous_valid_until.

This rule excludes External Secrets target Secrets because External Secrets Operator creates them.

Each exact runtime actor can update only its named Lease after cluster-owner creation.

Admission denies Argo Lease updates and every Lease create or delete operation.

Narrow RBAC and admission policy protect every ignored field from other ServiceAccounts.

Argo still self-heals labels, owners, immutable settings, RBAC, workloads, and every unlisted field on rendered objects.

Lease loss after activation requires a new incident-bound protected source change.

Close workloads and edge exposure before protected GitOps removes activation admission.

Invalidate all plans and evidence bound to the old UID before the exact bootstrap creates the missing Lease.

Register admission again only after new live evidence passes.

The applications must remain Synced and Healthy after a runtime rotation.

Deleting one External Secrets target Secret must cause that operator to recreate it.

An Argo self-heal must never delete or restore that target Secret's data.

21.7 GitOps safety order

Do not merge a Kubernetes baseline until every referenced container digest exists and passes offline verification.

The first Firecall GitOps merge keeps every runtime Deployment at zero replicas.

It omits every runtime CronJob, ExternalSecret, one-shot Job, and runtime PodDisruptionBudget.

It includes the two-replica binding validator and its PodDisruptionBudget.

Do not create a suspended CronJob in this baseline.

Argo reports a suspended CronJob as Suspended, which cannot satisfy the exact Healthy gate.

Apply these baseline Argo sync waves:

Wave Objects
-70 Namespaces and Pod Security labels
-65 ResourceQuotas and LimitRanges
-60 Complete annotated runtime ServiceAccounts, validator ServiceAccount, ten fixed certificate-leaf Secrets, and immutable ConfigMaps
-55 Digest, node-isolation, scheduler, preflight, and runtime ValidatingAdmissionPolicies and bindings
-50 Default-deny and exact-allow network policies, validator RBAC, Issuer, and Certificate
-45 Validator Service and other zero-runtime Services
-40 Two-replica binding-validator Deployment
-35 Binding-validator PodDisruptionBudget
-30 CA-injected Firecall-prefix webhook configuration after the validator becomes Ready
-20 Runtime RBAC
-10 Runtime Deployments fixed at zero replicas

After the dark Applications create both namespaces, run only kubernetes/bootstrap/firecall-state/bootstrap.sh.

Complete that retry-safe owner seed before any -55 activation admission object exists.

Argo must wait for each lower wave to become Healthy before it applies the next wave.

The validator has no workload RBAC or network path before its policies exist.

Do not use a PostSync Job to authorize activation.

The protected preflight identity has the exact authority in section 24.1.3.

The separate activation verifier reads the live Kubernetes facts in that section.

Standing, trust, image, policy, and edge workflows verify their own external resources.

Its fixed Pods use reviewed digests, no credential, no ServiceAccount token, and exact names.

It can read complete objects for only those names because Kubernetes cannot return status alone.

Use this activation sequence.

Foundation and publication

  1. Merge Groups A through D and inactive Group E Terraform, workflow, contract, and script source.
  2. Keep GitOps workloads, secret delivery, CronJobs, and creation absent, with creation_enabled=false and capacity_limit: 2.
  3. Create only the dedicated Terraform backend containers and their temporary owner assignments.
  4. Run the authenticated owner's first saved standing-control apply through its final state.
  5. Create all standing groups, storage, operational containers, locks, holds, Key Vault, Gallery, networks, fixed NSG, and registry paths.
  6. Run trust seed, import, and zero-change adoption for the circular objects and all required custom role definitions.
  7. Grant only the temporary exact bootstrap roles that those adopted definitions provide.
  8. Run the key and data bootstraps for fixed keys, secret placeholders, status pointers, and checkpoints.
  9. Populate the initial inactive App and Cloudflare slots through the manual credential JIT path.
  10. Apply both owner-only Policy roots and their final states, including the session Policy's initial deny-all image assignment.
  11. Give the owner temporary edge-state access, import all four edge secret objects, record them, and prove a zero-change refresh.
  12. Apply a trust plan that creates the trust-observer application but omits its FIC.
  13. Let a directory administrator grant and prove its one exact Microsoft Graph consent.
  14. Apply a second trust plan that creates its FIC only after the consent evidence passes.
  15. Run every initial credential permission canary and select only the proved primary slots.
  16. Bootstrap the control-monitor checkpoint, run its first poll, and require a successful immutable result.
  17. Remove every temporary owner assignment and wait for every issued storage, vault, Azure, and Microsoft Graph token to expire.
  18. Let the authenticated owner apply and test the USD 130 Firecall-owned-resource budget before any image build.
  19. Create and publish the four public CA roots, root lock, and signed root bundle.
  20. Publish, scan, sign, and offline-verify every Firecall container digest, including the binding validator and rebuilt cloudflared.
  21. Build, qualify, publish, and seal the first VM Gallery version.
  22. Merge container pins, apply the sealed image allowlist, prove the live assignment, then merge the exact control profile pin.
  23. Keep the production profile catalog disabled through every prior publication step.
  24. Verify BS-family, DDSv5-family, Total Regional vCPU, and Standard public-IP headroom for the current activation phase.
  25. Verify cert-manager v1.17.2, the reviewed Konnectivity tuple, Cilium policy support, and the exact live scheduler username.
  26. Prove every protected workflow trust claim and every positive and negative credential canary.
  27. Run the one-time capacity bootstrap and verify the three-slot ledger and legal-held marker.
  28. Disable its principal, remove its assignment and flexible credential, and wait for its issued token to expire.

The prior development PIM-for-Groups bootstrap is deferred with staging and production. It does not gate development foundation, secret delivery, edge activation, or session use. Protected workflows and the bounded owner operation mechanism control development infrastructure changes.

After step 22, publish the first signed scanner databases and run the manual security rescan for the exact pinned subject-set digest.

Require generation one, a signed pass, complete subject coverage, and current validity before cluster activation continues.

Cluster baseline and node pool

  1. Merge the baseline and require both Firecall Applications to become Synced and Healthy.
  2. Verify the Certificate, injected CA, validator replicas, validator PDB, default-deny policies, admission, RBAC, quotas, and zero runtime replicas.

The activation verifier does not read Secret objects.

After candidate-only consumers start, the certificate-bootstrap Job verifies both exact slots for each leaf.

It writes all five first leaves to inactive slots.

It publishes one complete immutable evidence object before one resource-version comparison selects all five leaves.

It accepts only the reviewed placeholder or a valid prepared leaf from a failed prior attempt.

It rejects unknown existing material.

The first issuance requires a candidate-health response from every expected current consumer Pod.

Those Pods run only candidate-health listeners while the certificate selector remains inert.

Every later rotation still requires proof from every expected consumer Pod before selection.

The Job publishes only public certificate metadata and immutable rotation evidence.

It makes no durable evidence write after the first selector becomes active.

A retry after a selector race never replaces prior immutable evidence. 31. Prove a real API-server callback arrives only from the reviewed Konnectivity tuple. 32. Run denied tenant, preview, nonscheduler-Binding, mutable-image, direct-egress, and cross-namespace probes. 33. Run the synthetic positive admission probe, require Pending state, and delete it without a bind. 34. Apply firecall-aks/dev only after the Firecall VNet and steps 29 through 33 have signed evidence. 35. Create only the node pool and AKS-side peering, then rerun final trust for their exact assignments. 36. Verify the real node prefix, subnet, peering, taint, label, system tuples, Node UID, and resolved image digests. 37. Run the node-tuple monitor and require its first signed current pass result. 38. Run one real scheduler test through the Firecall-prefix binding webhook, then delete its Pod.

Before step 39, the authenticated owner adds and tests the USD 105 exact AKS-pool budget.

Internal runtime

  1. Merge only DNS policy-proxy and HTTPS egress-gateway replicas with their PodDisruptionBudgets.
  2. Require both credentialless services to become Ready.
  3. Run exact positive and negative DNS, SNI, TLS-version, private-range, and direct-bypass tests.
  4. Import both complete cache-phase documents into the zone-policy state and require a zero-change saved plan.
  5. Apply the reviewed Firecall cache rule, prove live readback, and publish signed cache evidence.
  6. Verify that exact cache evidence, then apply edge resources at zero connectors and write audiences and tunnel token.
  7. Run bootstrap-prepare, then merge its exact request-public-key pull request into protected control main.
  8. Run bootstrap-activate after that merge and verify the exact selectors and prepared status.
  9. Merge internal HMAC and application ExternalSecrets, including all audience values, then require newer successful refreshes.
  10. Merge candidate Services, consumer replicas, and the suspended certificate-bootstrap Job while status remains prepared.
  11. Run certificate bootstrap against every candidate Pod, then require all five leaves and one atomic selector update.
  12. Require every consumer to become Ready, then run bootstrap-prove and verify the complete status.
  13. Record signed evidence, then remove the completed bootstrap Job from source.
  14. Run internal origin, credential, storage, request-signing, workspace-CSRF, and readiness tests.
  15. Add each unsuspended certificate or maintenance CronJob only after its dependency proof passes.

The certificate-bootstrap Job uses the fixed certificate-rotator ServiceAccount, digest, command, volumes, resources, and network policy.

Give candidate Services wave -20.

Give consumer Deployments and the certificate-bootstrap Job wave -10 in the same source merge.

The bootstrap Job waits for the exact prepared credential status before it mutates a certificate slot.

It requires generation 1 and the reviewed nonzero control commit.

It also requires the status digest to equal both live credential selector ConfigMaps.

Each later certificate rotation requires the exact immutable complete credential status.

Argo starts consumer Pods before certificate bootstrap but keeps their normal listeners unready.

The candidate certificate-health listeners operate while the active certificate selector remains inert.

Argo creates and removes that one Job.

No OIDC identity receives general Job-create authority for certificate bootstrap.

Edge and dark activation

  1. Prove active Universal SSL coverage, chain, SNI, TLS 1.3, and 30-day validity for all three host classes.
  2. Read and require exactly one existing Cloudflare email one-time PIN provider while every application remains closed.
  3. Prove the edge plan does not create, update, import, or delete any identity provider.
  4. Verify the three Firecall applications select only the existing one-time PIN provider.
  5. With zero connectors and fallback-only ingress, apply the normal domain policy temporarily to the portal application.
  6. Complete one email-PIN login from an @openmined.org mailbox and validate the final application JWT.
  7. Require the exact issuer, portal audience, times, type, identity nonce, subject, and canonical email domain.
  8. Remove the temporary policy and prove all policy lists are empty.
  9. Prove fallback-only ingress, zero connectors, and no new Firecall application-token issuance.
  10. Record the provider, application, policy, and successful canary digests without a PIN, cookie, token, or raw subject.
  11. Merge the reviewed development identity configuration and canary evidence.
  12. Require the Cloudflare drift observer to match the one-time PIN provider and all closed application fields.
  13. Require no Entra, Conditional Access, PIM, MFA-claim, or human-approval dependency.
  14. Apply the saved edge transition with the three steady domain policies and routes.
  15. Set protected activation state dark_ready while creation remains disabled.
  16. Verify exact policy order, ingress order, default denial, audiences, certificates, and creation_enabled=false.
  17. Merge only the edge tunnel-token ExternalSecret and wait for its first newer successful refresh.
  18. Write the initial tunnel-token Key Vault version GUID to cloudflared-rollout-epoch.
  19. Patch the zero-replica Deployment with that epoch through the annotation-only policy.
  20. Merge only two cloudflared replicas and their PodDisruptionBudget.
  21. Require two current ready Pods, distinct connector IDs, matching epoch, live connections, and both origin checks.
  22. Complete one fresh email-PIN application-token canary for each exact audience.
  23. Require distinct audiences, exact claims, the one-time PIN login method, and the canonical email domain.
  24. Publish the signed access-policy pass verdict without storing a PIN, cookie, token, or raw subject.
  25. Verify dark_ready with creation still disabled.
  26. Run the complete negative-only dark preflight defined below.
  27. Keep creation disabled and recheck current rescan, node-tuple, and access-policy pass verdicts.
  28. Recheck live certificate selectors, chains, SPKIs, expiry, replica agreement, connector health, and public edge SNI.

Group E ends at step 81.

The dark_ready state permits authentication and health checks. It keeps session creation disabled. Unknown, closed, and unacknowledged data routes still fail before any box contact.

The browser-token helper keeps each canary cookie and token only in locked memory. It clears them after validation and records only safe evidence.

The live Cloudflare global-session duration remains an exact protected activation input. It does not add a human approval step.

The dark preflight creates no session identity, VM, NIC, disk, public address, NSG, role assignment, preview, or agent route.

It proves only these closed-state facts:

  • Portal authentication works, but every create request fails with creation_disabled before dispatch or capacity mutation.
  • Unknown, nonready, expired, and closed workspace hosts fail before proxy-to-agent contact.
  • Unknown and unacknowledged preview hosts fail before upstream resolution or port forwarding.
  • Status, request, capacity, revocation, invitation, audit, and evidence positive and negative permission canaries match their contracts.
  • Certificate, credential, tunnel-token, request-key, and root rotation dry runs produce no selector or secret change.
  • Close of an unknown or already terminal session is bounded, idempotent, and creates no Azure resource.
  • Every internal runtime health, mutual-TLS, origin, edge, storage, audit, policy, and notification check passes.

Do not claim positive terminal, Kubernetes, log, event, preview, collaborator, close, expiry, or cleanup behavior from dark preflight.

Group F's owner canary provides those positive end-to-end proofs with one real session.

Only Group F can enable creation, reconcile capacity, and open the first owner canary.

Each replica merge changes only exact reviewed counts and required PodDisruptionBudgets.

Each secret-delivery merge adds only the exact reviewed ExternalSecret objects.

Each CronJob enters source unsuspended after its dependency proof.

Maintenance removes a CronJob from source instead of suspending it behind a Healthy gate.

The first edge apply ends after exact API comparison and Key Vault delivery.

It does not require a connector while the connector count is zero.

Routine edge applies require the two-connector proof after activation.

A missing, stale, or failed signed artifact blocks the next source merge.

Sync waves provide object order but never replace these live gates.

22. Development Cloudflare identity validation

Use exactly one existing Cloudflare Access identity provider with type=onetimepin. Terraform reads this provider and does not manage it.

Configure all three development Access applications with only that provider ID. Enable automatic redirect because each application has one login method.

Each active application has one inline allow policy. The policy includes only email domain openmined.org. Its only Require selector is the exact one-time PIN login-method ID.

Do not add an Entra login-method selector, auth_method=mfa, bypass rule, service token, group, country, IP, everyone, or alternate identity provider.

Cloudflare unmatched default denial is the only fallback. The closed state keeps every inline policy absent and every origin route absent.

Validate these final Access JWT claims:

  • RS256 signature and current account key.
  • Exact Cloudflare team issuer.
  • Exactly one expected application audience.
  • type=app.
  • Valid iat, nbf, and exp within the existing one-hour bounds.
  • One non-empty identity_nonce.
  • One non-empty sub.
  • One ASCII email with exact normalized domain openmined.org.

Do not require amr=mfa, custom.tid, or custom.oid for development. Derive the durable development subject_key from the canonical email as section 7 specifies. Keep the current Cloudflare sub binding for short-lived browser artifacts.

One-time PIN proves current mailbox access. It does not prove device state, phishing-resistant MFA, or an immutable directory object. Mailbox reassignment can transfer a development identity. This accepted development risk does not apply to staging or production.

An administrator-created alias or shared mailbox can create a separate durable development identity. Offboarding must revoke every known address and disable all aliases and shared addresses. Add an explicit canonical-address directory before person-wide offboarding becomes a stronger requirement.

Normal development session actions have no approver. This includes create, default read, optional write selection, use, close, and automatic expiry.

Cloudflare records successful PIN authentication. Firecall records each session and access choice. Slack notification for a write selection is best-effort and cannot block a development session.

Offboarding closes active sessions, creates the Firecall revocation marker, revokes the user's Cloudflare Access session, and disables corporate mailbox access. Token expiry alone is not immediate offboarding.

The one-time PIN provider is account-wide. Firecall does not create, update, import, or delete it. Firecall changes only its three Access applications.

Staging and production need separate applications and policies. Their future design requires stronger corporate identity, MFA, and explicit environment reviewers. Development readiness does not depend on those future controls.

22.1 Deferred staging and production identity requirements

Staging and production human access is outside the development release. Neither environment can reuse the development applications, audiences, email-PIN policy, or email-derived subject keys.

A later ADR must define each environment before implementation. That decision must include:

  • Separate Cloudflare Access applications, audiences, and policies.
  • A durable corporate identity source.
  • Phishing-resistant multifactor authentication.
  • Explicit environment reviewers and session approval.
  • Separate offboarding and emergency revocation.
  • Positive and negative identity canaries.
  • Exact audit and retention requirements.

PIM means Microsoft Privileged Identity Management. A later ADR can select PIM as one approval mechanism. This plan does not select or activate PIM.

These existing files remain inactive reference material:

  • platform/firecall/contracts/conditional-access.json.
  • platform/firecall/contracts/cloudflare-entra-idp.json.
  • platform/firecall/contracts/pim-jit.json.
  • scripts/ops/firecall-conditional-access-bootstrap.sh.
  • scripts/ops/firecall-cloudflare-idp-bootstrap.sh.
  • scripts/ops/firecall-pim-bootstrap.sh.
  • scripts/ops/firecall-pim-ca-bootstrap.sh.

Development workflows must not evaluate these contracts or run these scripts. Development infrastructure must not create their groups, policies, roles, evidence stores, or trust objects.

22.2 Human offboarding

Expose POST /api/v1/offboarding only to a current platform owner on the portal.

Require the exact portal Origin, fetch-site header, CSRF token, active-identity selection, fingerprint confirmation, and a bounded reason.

The owner first requests the bounded active-identity view.

The broker derives that view from active requester status and bound invitation entries.

It returns the first 12 subject_key characters, role, active-session count, and one signed opaque selection token per subject.

The selection token binds the complete target subject key, active-session set digest, owner subject key, and five-minute expiry.

The owner types the displayed subject fingerprint to confirm the target.

Store the complete subject key in each requester status and each bound invitation index entry.

Do not store a collaborator's plaintext email in the invitation index or offboarding audit.

The broker scans only current active lifecycle statuses, which the capacity limit bounds to three sessions.

For each active session, it reads that exact invitation index.json without a container-root list.

It verifies the selection token and matches requester and collaborator subject keys.

It builds one canonical sorted session set and rejects an active-set digest change until the owner refreshes the view.

If the selected subject is a current platform owner, the set contains every active Firecall session.

If one required index is missing, malformed, or unreadable, it conservatively selects every active session.

The broker first creates firecall-revocations/identities/<subject_key>.json with If-None-Match: *.

That global identity marker blocks every portal, workspace, preview, invitation, and stream authorization for the subject.

Broker and proxy read its exact path before every authorization decision.

They cache an authenticated 404 absence for no more than five seconds.

A storage error, malformed marker, wrong digest, or non-404 response fails closed.

Both proxy replicas poll active subjects within five seconds and close their streams when a marker appears.

The broker also starts the section 10.2.1 internal revocation delivery for each affected session.

That route works without an existing browser or proxy stream.

It then conditionally creates every selected session's close-revocation marker before it dispatches any close workflow.

The agent never reads Blob storage or accepts a caller-supplied revoke command.

The broker then dispatches the existing signed close workflow once for each selected session.

Dispatch inputs contain only session ID, subject key, operation ID, reason digest, and offboarding actor binding.

The operation is idempotent, permits one active request per target digest, and records partial progress durably.

A failed dispatch or local delivery leaves all created revocations active and exposes retry and break-glass actions to the owner.

The fixed revoke operation stops background tmux, shell, and optional SSH processes before Azure cleanup.

Until that cleanup succeeds, status reports Access revoked · cleanup pending and retains capacity.

The offboarding API never changes the protected owner-list source.

If the target is a platform owner, a different owner opens a separate protected pull request that removes the exact subject key.

That change needs CODEOWNERS review, a different-owner approval, green tests, protected merge, and immutable merge evidence.

Broker and proxy replicas atomically reload the new increasing owner-list revision and matching canonical digest.

The global identity marker keeps access denied if that source change or reload fails.

Do not remove that marker before the owner-list change, mailbox disablement, and Cloudflare revocation all pass.

Keep the identity marker permanently by default and exclude it from session-revocation cleanup.

Reinstatement needs a separate owner-approved operation after mailbox restoration and every prior Access token expires.

Use scripts/ops/firecall-identity-reinstate.sh for that operation.

A different owner approves one time-limited assignment on only the exact identity-marker Blob.

That assignment permits direct read and conditional delete, with no list, create, write, lease, tag, or sibling action.

The same approval creates two separate create-only assignments on these exact audit paths:

identity-reinstatement/intents/<subject_key>/<operation_id>.json
identity-reinstatement/completions/<subject_key>/<operation_id>.json

Those assignments permit no audit read, list, overwrite, delete, lease, tag, sibling, or container-root operation.

The script requires terminal zero-resource proof for every prior matched session.

It also requires mailbox restoration, Cloudflare administrator evidence, applicable owner-list state, and storage time beyond token expiry plus skew.

It creates the exact immutable reinstatement-intent event with If-None-Match: * before deleting the marker through its exact ETag.

It verifies authenticated 404 and creates the exact deterministic completion event with If-None-Match: *.

It then removes all three temporary assignments and waits for its issued token to expire.

On retry with an absent marker, the script cannot read audit storage.

It dispatches the audit verifier for the one exact deterministic completion path.

The verifier returns a keyless-signed statement bound to object bytes, ETag, digest, protected workflow, commit, run, and verification time.

The script accepts success only after it validates that statement and its Sigstore bundle offline.

The audit verifier receives no marker-delete or reinstatement-write authority.

Every mismatch, early attempt, stale ETag, partial audit, or sibling path fails closed.

A corporate email administrator disables the mailbox, and a Cloudflare administrator revokes its organization session.

Offboarding completes only after applicable owner-list proof, Cloudflare denial, terminal status, and zero-resource proof all pass.

Another platform owner verifies the immutable evidence and no target email appears in logs or workflow inputs.

23. Credential matrix

Credential Holder Scope Delivery Lifecycle
Cloudflare JWT Browser, broker, proxy Exact portal, workspace, or preview audience Header and browser cookie One-hour app session; mandatory per-user global-session revocation and Firecall close
HTTPS egress gateway None Exact reviewed public DNS names on TCP 443 No credential Replace Pod to revoke access
External Secrets identity External Secrets controller Read every exportable secret in the dedicated Firecall Key Vault Azure workload identity Revoke Firecall-vault role assignment
Broker GitHub App key Broker only Control Actions write and Contents read Read-only mounted Key Vault secret 90-day rotation; 24-hour overlap
Image-pin App key Protected image-pin workflow Mint tokens for infrastructure and control only Exact Key Vault secret through image-pin OIDC 90-day rotation; 24-hour overlap
Root-publication App key Protected CA-root workflow Propose public-root changes in infrastructure only Exact Key Vault secret through CA-root OIDC 90-day rotation; 24-hour overlap
Classic-observer App key Protected capacity workflow Read Classic source and Actions runs in infrastructure only Exact Key Vault slots through capacity OIDC 90-day rotation; 24-hour overlap
Control-observer App key External infrastructure monitor Read control source, workflow states, runs, logs, and artifacts Exact Key Vault slots through monitor OIDC 90-day rotation; 24-hour overlap
Settings-observer App key Trust observer Read infrastructure settings, Actions, and source Exact Key Vault slots through trust-observer OIDC 90-day rotation; 24-hour overlap
Request signing key Broker through Key Vault Sign request envelopes Workload identity and Key Vault API Non-exportable ES256 key
CSRF signing key Broker through Key Vault Sign portal CSRF tokens only Workload identity and Key Vault API 90-day version; one-hour verifier overlap
Request public key Control runner Verify envelopes Protected repository file Current and previous
Broker storage identity Broker Manage request receipts and rate counters; read status, capacity, invitation indexes, security, node-tuple, and access-policy current records and exact evidence; create revocations and Access identity challenges Azure workload identity Revoke role assignment
Proxy storage identity Proxy Read status, session and identity revocations; create delivery receipts; manage invitations and rate counters; append proxy audit Azure workload identity Revoke role assignment
Probe worker storage identity Readiness worker Read probe requests and write probe results only Separate workload identity Revoke role assignment
Probe worker mTLS key Readiness worker Call the internal proxy probe Service Separate read-only Kubernetes Secret Current and previous client-root-signed leaves
Component audit identities Broker, proxy, runner Create immutable audit events Workload identity or OIDC Revoke role assignment
Audit reader group Reviewed platform operators Read firecall-audit only Entra group role assignment Quarterly membership review
Certificate rotator identity Certificate rotator only Key Vault sign, public trust publish, and narrow Secret update Azure workload identity and Kubernetes RBAC Revoke role assignment
Client CA signing key CA-root workflow and certificate rotator Self-sign fixed client roots and sign proxy leaves Exact non-exportable Key Vault key 12-month active-root rotation
Public client trust metadata Certificate rotator writes; control runner reads Public proxy leaves and active rotation only firecall-trust with separate data roles ETag-protected rotation
Proxy client key Proxy only Authenticate to box agents and sign readiness results Separate read-only Kubernetes Secrets Current and previous 30-day leaves
Broker revoker key Broker only Authenticate and sign the reduction-only internal proxy revoke request Separate read-only Kubernetes Secrets Current and previous 30-day leaves
Session token key Proxy only Sign workspace CSRF, acknowledgement, cursor, and preview tokens Separate read-only Kubernetes Secret 30-day key; 24-hour verifier overlap
Box-local kube token Interactive tools and agent on one box Authenticate only to 127.0.0.1:9443 on that box Read-only kubeconfig Random per session; destroy with box
Box server key One box One agent endpoint Generated on the box Destroy with box
SSH host key sshd on one box Pin optional SSH to one session address Generated on box; public key and fingerprint enter runner status Private key never leaves box; destroy with box
Origin CA signing key CA-root workflow and certificate rotator Self-sign fixed origin roots and sign origin leaves Exact non-exportable Key Vault key 12-month active-root rotation
Origin leaf keys Broker and proxy only Exact Service DNS name Separate Kubernetes Secrets Rotate after 15 days
Control OIDC identity Control main runner Firecall resources only GitHub OIDC Short-lived token
Break-glass OIDC identity Allowlisted manual job on control main Reduction-only session cleanup GitHub OIDC Short-lived token
Classic-reduction OIDC identity Exact infrastructure main reduction workflow Close, sweep, and zero-proof for deterministic Classic resources only GitHub OIDC Short-lived token; replaces historical Classic FIC at cutover
Capacity-bootstrap OIDC identity One exact control main bootstrap workflow Create initial ledger and permanent marker only GitHub OIDC Revoke after one use
Capacity-reconcile OIDC identity Exact control main capacity workflow One ledger, three Classic states and proof read, exact App-key read, and capacity audit GitHub OIDC Short-lived token
Audit-verifier OIDC identity Exact control main audit workflow Read audit and correlated lifecycle facts only GitHub OIDC Short-lived token
Break-glass audit-reconciler OIDC identity Exact control main reconciliation workflow Read immutable consumption markers and append their deterministic audit objects only GitHub OIDC Short-lived token
Assignment-auditor OIDC identity Exact control main assignment-audit workflow Read exact AKS assignments, capacity, and active status only GitHub OIDC Short-lived token
Preflight OIDC identity Exact infrastructure main preflight workflow Fixed credentialless fixture Pods and guarded Binding probes only GitHub OIDC, exact Kubernetes RoleBindings, and fail-closed admission Short-lived token
Activation-verifier OIDC identity Exact infrastructure main activation-verifier workflow Read exact Firecall cluster objects and reviewed system tuples only GitHub OIDC and read-only Kubernetes RBAC Short-lived token
Control-monitor OIDC identity Exact infrastructure main external control monitor Read observer App slots and request/status metadata; update one leased checkpoint; append monitor audit GitHub OIDC Short-lived token
Trust-observer OIDC identity Exact infrastructure main trust-drift schedule Read Graph application metadata, exact Firecall assignments, settings App slots, and protected settings GitHub OIDC Short-lived token
Image-idle OIDC identity Exact infrastructure main idle schedule Lease one operation lock and restore only the fixed idle NSG GitHub OIDC Short-lived token
Image-inventory OIDC identity Exact infrastructure main inventory schedule Read Gallery versions, locks, and shared-registry evidence; publish proposals GitHub OIDC Short-lived token
Cloudflare-drift OIDC identity Exact infrastructure main drift schedule Read one observer-token slot and append drift audit GitHub OIDC Short-lived token
Node-tuple-observer OIDC identity Exact infrastructure main schedule or manual monitor Read exact AKS node-plane facts; advance one signed tuple verdict GitHub OIDC Short-lived token
Security-rescan OIDC identity Exact infrastructure main rescan workflow Pull enabled artifacts, read Gallery seals, update one leased verdict, and append signed evidence GitHub OIDC Short-lived token
Scanner-database-publisher OIDC identity Exact infrastructure main database workflow Create signed Trivy DB objects and advance one leased database pointer GitHub OIDC Short-lived token
Tool-cache-publisher OIDC identity Exact infrastructure main acquisition workflow Create verified public tool bytes and signed acquisition evidence in digest paths GitHub OIDC Short-lived token
Retention-reconciler OIDC identity Exact infrastructure main daily workflow Read reference graph; delete only expired unreferenced evidence and terminal delivery receipts; append retention audit GitHub OIDC Short-lived token
Standing-resource OIDC identity Infrastructure main standing workflow Two standing states and exact standing Firecall children GitHub OIDC Short-lived token
Trust-root OIDC identity Infrastructure main trust workflow Owned Firecall applications, fixed scoped assignments, and trust state GitHub OIDC Short-lived token
Image-builder OIDC identity Infrastructure main image workflow Temporary build group, gallery publish, and registry push GitHub OIDC Short-lived token
Image-sealer OIDC identity Infrastructure main seal workflow Gallery read, deterministic seal-lock write, and shared-registry pull GitHub OIDC Short-lived token
Image-pin OIDC identity Infrastructure main pin workflow App-key read, Gallery and lock read, and shared-registry pull GitHub OIDC Short-lived token
CA-root OIDC identity Infrastructure main CA-root workflow Four root keys, public-root publication, root App key, and bounded rollout proof GitHub OIDC Short-lived token
Credential-rotation OIDC identity Infrastructure main credential workflow Trigger and observe one admission-bound rotation Job GitHub OIDC Short-lived token
Credential-rotator workload identity Exact rotation Job Fixed Key Vault slots and keys, rotation ConfigMaps, and one Lease AKS workload identity Revoke role and ServiceAccount binding
Policy-apply OIDC identity Infrastructure main policy workflow Exact Firecall assignment parameters and isolated state only GitHub OIDC Short-lived token
Cloudflare-apply OIDC identity Infrastructure main edge workflow Edge state, exact Key Vault values, rollout ConfigMap, and one protected epoch annotation GitHub OIDC Short-lived token
Zone-policy-apply OIDC identity Owner-approved infrastructure environment workflow Complete cache-phase state and exact zone-policy token slot GitHub OIDC Short-lived token
Cloudflare API token Protected edge workflow only All account tunnels and Access apps; exact-zone DNS in two zones Selected exact Key Vault primary or secondary slot Ninety-day expiry; owner revokes old token after canary switch
Cloudflare observer token Drift observer only Account tunnel and Access read; exact-zone DNS, SSL, certificate, and cache read; account ruleset and filter-list read Selected exact Key Vault primary or secondary slot Ninety-day expiry; no edit permissions
Cloudflare zone-policy token Zone-policy workflow only Cache Rules Edit on two zones; account Rulesets and Filter Lists Edit Selected exact Key Vault primary or secondary slot Ninety-day expiry; owner revokes old token after a no-change plan
Cloudflare zone-policy state Zone-policy identity only Both complete cache-phase documents Dedicated exact Blob key Versioned, soft-deleted, and no standing human access
Cloudflare Terraform state Protected edge workflow; exact bounded owner operation during recovery One exact state key containing the connector token Dedicated Azure Blob container No standing developer access; remove temporary recovery access after use
GitHub-hosted runner VM One exact privileged job Every short-lived token, selected App key, state value, and secret loaded by that job GitHub Actions Fresh ephemeral VM for each job; reviewed image version gate
Automatic GITHUB_TOKEN One exact job Only the per-job permissions in section 24.1 Ephemeral GitHub Actions token Expires when the job ends; permissions default to none
DNS policy proxy None Exact approved DNS questions only No credential Replace Pod to revoke access
Binding validator Read-only Kubernetes ServiceAccount Name-based GET of complete Pod objects and Nodes for Firecall-prefix Bindings Bound ServiceAccount token Revoke ClusterRoleBinding or webhook configuration
cert-manager Existing cluster controller Issue the validator server certificate and inject its exact CA bundle Existing cluster authority Existing platform lifecycle
Box managed identity Root kube proxy on one box Cluster User plus resolved namespace Readers and Writers Azure IMDS, blocked from interactive users Delete with session; copied write-scope Secrets and ServiceAccount tokens remain outside this revocation
Tunnel token cloudflared only One Firecall tunnel One versioned Key Vault secret Connector-safe rotation
SSH public key One box Optional user login Signed request payload Destroy with box
SigNoz token Nobody in Firecall None None None
Reinstatement operator token One different approved owner One exact identity marker and two exact create-only audit objects Time-limited Entra role assignments Remove all assignments and wait for token expiry after one operation

Use separate Kubernetes ServiceAccounts and Secrets.

Mount private keys as files.

Do not place private keys in environment variables.

No Firecall runtime component receives audit-read authority.

The broker storage identity cannot write a security, node-tuple, or access-policy current record.

It cannot read, create, write, list, or delete an Access identity challenge consumption record.

23.1 Residual authority

No design removes all component authority.

Document these honest compromise limits:

Compromised component Maximum intended result
Broker or broker App Read request and status metadata; dispatch fixed-profile sessions; impersonate requesters; create permanent global markers for known subject keys; close their sessions and local processes; disable or re-run control workflows; delete GitHub run evidence; or deny close and cleanup automation
Terminal proxy Read requester subject keys, profiles, namespaces, expiry, box addresses, certificate, image, protocol, request, plan, managed-identity, role-assignment, cleanup, and bounded audit metadata; enter active boxes; use box identities; forge signed delivery receipts; alter invitation or rate state; append proxy audit; or deny service
HTTPS egress gateway Observe destination names and encrypted traffic sizes, deny service, or bypass its destination policy for callers that can reach it
Readiness worker Suppress readiness or publish invalid results that the runner rejects
One box Use resolved Reader and Writer power plus public TCP 443 egress until expiry or cleanup
Root session supervisor Obtain root on its one ephemeral box and use that box identity until expiry or cleanup
Root kube proxy Export a cached Azure token and use resolved Reader and Writer authority until that token expires
Control runner Run root on session VMs; use box identities; create allowed VMs; assign approved AKS roles to any principal; hide live assignments; retain issued access until token expiry; and read subscription Activity Log metadata
Break-glass runner Delete Firecall resources and approved AKS role assignments at its assigned scopes
Classic reduction runner Delete deterministic Classic resources, assignments, and state content, but not create a session
Capacity bootstrap Create or overwrite the exact ledger until revocation; the legal-held marker remains immutable
Capacity reconciler Read all three Classic states, source, workflow metadata, App key, and roles; disable opens or set capacity to three
Audit verifier Read Firecall identity and lifecycle audit metadata, including developer email and subject key
Break-glass audit reconciler Read consumption bindings and append forged consumption audit events, but not change a marker or retained event
Assignment auditor Read active session identity and exact AKS role-assignment metadata, but not change either
Preflight workflow Create only admission-bound credentialless test Pods and guarded Binding attempts in firecall; read their complete Pod objects; or deny activation
Activation verifier Read complete reviewed Kubernetes objects, system Pod specifications, node addresses, and Firecall status metadata; or deny activation
Control monitor Read control workflow evidence and session PII, including email, subject key, reason, namespaces, SSH address, and lifecycle data; corrupt or roll back its checkpoint until chain review; append forged monitor events; or suppress its alert
Trust observer Read directory-wide application metadata, exact Firecall role metadata, and infrastructure repository settings; forge drift evidence or suppress its alert
Image-idle reconciler Replace the fixed build NSG with another Policy-compliant document, deny builds, and suppress its alert
Image-inventory observer Read every shared-registry artifact and Gallery metadata, forge quarantine proposals, or suppress its alert
Cloudflare drift observer Read account tunnel, Access, ruleset, filter-list, exact-zone DNS, SSL, and certificate metadata; forge drift evidence or suppress its alert
Node-tuple observer Read complete node and system-workload specifications, forge a current tuple verdict, deny new sessions, or suppress its alert
Security rescan Read every shared-registry artifact, publish a forged current verdict or rescan record, deny new sessions, or suppress its own alert
Scanner-database publisher Publish a forged but correctly signed vulnerability database or deny future database updates
Tool-cache publisher Add forged public cache objects or deny acquisition, but not overwrite an existing digest path or change a merged lock
Retention reconciler Read retained lifecycle metadata; delete expired unreferenced evidence or receipts; deny historical analysis; or suppress completion audit
Standing-resource workflow Change standing Firecall resources and its exact AKS pool or peering children, but not trust or policy
Trust-root workflow Add credentials to every owned Firecall app, assume their combined Azure roles, and change fixed assignments at approved scopes
Image builder Push any shared-ACR repository, publish unsealed images, and create arbitrary deployment wrappers whose nested resources still face fixed Policy
Image sealer Read every shared-registry artifact and weaken or add a Gallery lock, but not change an image version
Image-pin workflow Read every shared-registry artifact and propose source changes, but not merge them or change Azure resources
CA-root workflow Replace four root slots, read its App key, propose public source, and publish trust, but never export a root key
Credential rotator Change fixed HMAC and signing-key versions and select their active identifiers
Policy-apply workflow Change only the exact Firecall assignment and its reviewed parameters
Cloudflare-apply workflow or API token Change every account tunnel or Access app, two-zone DNS, and force cloudflared rollouts
Zone-policy workflow or token Replace both complete cache phases and change unrelated account rulesets or filter lists
Active edge-state JIT operator Read the connector token, corrupt the exact state key, and attach a rogue connector until rotation
GitHub Actions hosted runner Read or export every credential and sensitive state value loaded into its one privileged job
DNS policy proxy Deny DNS, return poisoned approved-name answers, or use bounded query timing
Binding validator Read complete selected Pod specs, deny Firecall scheduling, or approve an unreviewed Pod for the Firecall node
cert-manager Replace the binding-validator certificate or CA injection through its existing cluster authority
cloudflared Observe routed traffic or attach a rogue Firecall connector
Cloudflare edge platform Terminate browser TLS and observe identity tokens, terminal input and output, logs, preview bodies, and forwarded application authorization
External Secrets controller Read every exportable Firecall secret and materialize it into either allowed Firecall namespace
Argo CD Alter Firecall workloads or policies and deploy a Pod that reads namespaced exportable Secrets through its existing cluster authority
AKS node plane Node root, kubelet, or a privileged managed system DaemonSet can read Firecall Pod memory, mounted volumes, and exportable Secrets on that node
Control repository main Use the Firecall control OIDC identity
Infrastructure repository main Use each exact Firecall image, policy, and Cloudflare reusable-workflow identity
Cloudflare administrator Change Firecall routing and human access policy
Status writer Publish malicious route data within the fixed Firecall private subnet
Audit appender Add forged audit events, but not alter retained events
Certificate rotator Sign arbitrary Firecall client or origin leaves and replace serving or probe leaf Secrets

A compromised broker can cause permanent service denial for any subject key that it knows.

It cannot remove that marker.

Recovery requires a different owner, verified directory and Cloudflare restoration, expiry of prior tokens, zero-resource proof, and the immutable reinstatement audit.

Application.ReadWrite.OwnedBy is a high-impact trust-root permission.

A compromised trust-root principal can add a password, certificate, or federated credential to every Firecall application it owns.

It can then assume the combined Azure authority of those applications until revocation and token expiry.

The checked-in reconciler limits correct automation but cannot constrain a stolen Microsoft Graph token.

Daily credential drift checks alert on any added credential, owner, password, certificate, or application property.

For suspected trust-root compromise, immediately disable creation and every Firecall application that the trust root owns.

Revoke the trust-root flexible credential, service principal, application credentials, and Graph administrator consent.

An authenticated directory owner inventories every owner, password, certificate, and federated credential on each owned application.

Compare that inventory with the signed closed manifest and remove every unapproved persistent object through the owner recovery path.

Do not restore an application merely because the trust-root principal is disabled.

Rotate every downstream GitHub App key, exportable Firecall secret, HMAC, Cloudflare token, and affected signing-key version.

If the trust root could assume a CA identity, run the complete CA incident contract in section 10.12.

Wait for all issued Graph, Azure, GitHub, Cloudflare, and Kubernetes tokens to expire.

Create a new commit-pinned trust credential, restore only closed-manifest owners and credentials, then run all positive and negative canaries.

Keep only reduction paths available until the restored inventory, secret rotations, and immutable incident evidence pass review.

The main threats are:

  • Forged identity headers.
  • Compromised browser content.
  • Cloudflare token leakage.
  • Status route poisoning.
  • Concurrent capacity races.
  • Stale lifecycle writes.
  • Partial Azure cleanup.
  • Supply-chain substitution.
  • Preview SSRF.
  • Cross-origin cookie poisoning.
  • Unbounded terminal or log streams.
  • Abuse of namespace Reader or Writer power.
  • Misuse of the shared AKS outbound source address.
  • Assignment of an approved AKS role to an unrelated principal.
  • Audit-event forgery by a compromised event producer.
  • Exportable secret theft through a compromised External Secrets controller.

The tests in section 36 cover each threat.

Mutual TLS, Secret RBAC, and digest admission do not constrain Cloudflare, Argo CD, or AKS node-root authority.

Firecall stores no terminal content, but Cloudflare can observe it while traffic crosses its edge.

For Cloudflare compromise, disable Firecall Access and tunnel routing, close sessions, and rotate tunnel, App, and session credentials.

For Argo CD compromise, disable creation, stop Firecall synchronization, revoke its repository or cluster credential, and rotate every namespaced Firecall secret.

For AKS node-plane compromise, disable creation, close sessions, cordon and replace the Firecall node, and rotate all credentials mounted there.

Resume only after signed source, workload digests, policies, node images, and live state pass full qualification.

24. Azure authority

24.1 Workflow-bound OIDC trust

OIDC means OpenID Connect.

Do not trust a repository-and-branch subject alone.

Azure flexible federated credentials require an Entra application and service principal.

Do not use a user-assigned managed identity for these GitHub credentials.

Each flexible federated identity credential uses exact eq comparisons for only these four GitHub claims:

  • The complete declared sub claim for a protected main ref or one exact environment.
  • Numeric repository_id.
  • Numeric repository_owner_id.
  • Exact job_workflow_ref with one approved 40-character commit pin.

Set issuer https://token.actions.githubusercontent.com outside the claims expression.

Set one audience, api://AzureADTokenExchange, outside the claims expression.

Set languageVersion to 1 and omit subject.

Do not add job_workflow_sha, run_attempt, event_name, ref, sha, workflow, workflow_sha, environment, aud, or another claim lookup.

Microsoft flexible credentials do not support those extra GitHub claim expressions.

Use one event-specific reusable privileged workflow for each row below.

The pinned reusable workflow enforces the expected event and first attempt.

It also checks the exact public caller workflow and current protected main commit before Azure login.

Do not use an or expression or share one privileged path across events.

GitHub re-runs increment run_attempt. The reusable workflow rejects every attempt other than 1 before its Azure job starts.

Use one infrastructure environment named firecall-live for an owner-approved live mutation.

Require an authenticated owner reviewer, prevent self-review, and restrict its deployment branch to protected main.

Set can_admins_bypass=false and permit no custom branch or tag deployment rule.

Store the exact environment database ID, reviewer subject IDs, and rule digest in platform/firecall/contracts/assurance/firecall-live-environment.json.

Use the environment only as an approval gate.

Store every approved non-secret live value in platform/firecall/contracts/assurance/firecall-live-configuration.json.

The current activation_blocked document contains no values and no digest.

The future verified document contains one exact configuration for each standing, trust, and Cloudflare stack.

Its configuration_sha256 is the SHA-256 of the canonical complete configurations object.

Resolve organization and repository variables in the public caller only as transport values for its reusable-workflow call.

Pass those transport values and the reviewed digest as typed inputs.

Before any post-checkout third-party action or cloud OIDC request, compare the complete selected input set and digest with the exact reviewed source contract.

Bind the same digest in the approval line and every saved-plan manifest.

A privileged workflow that uses firecall-live must not reference the vars or secrets context.

Credentialless validation rejects either context in those workflows.

An input, environment variable, or secret therefore cannot replace an approved operational value.

The credentialless request job prints one canonical approval line before the protected job waits.

The reviewer copies that exact line into the environment approval comment.

Immediately before OIDC, the privileged workflow reads the live environment, current run, and complete review history through GitHub's API.

It binds the run, first attempt, exact workflow job, action, stack, source commit, saved-plan run, requester, environment, and one different configured reviewer.

It rejects a missing, edited, duplicated, stale, self-approved, or unlisted review.

The workflow run ID, job ID, current ephemeral runner identity, and mandatory first attempt make the approval receipt single-use.

Each public dispatch calls only one matching privileged mutation job.

The trust observer verifies that manifest every six hours and before each protected live mutation.

Its flexible credential requires this exact subject:

repo:OpenMined/infrastructure:environment:firecall-live

Use that subject for standing, trust, image build, CA-root, credential, policy, and Cloudflare mutation workflows.

Use the exact ref:refs/heads/main subject for automatic, read-only, proposal-only, and test workflows.

No workflow receives credentials for both subject forms.

A scheduled workflow never uses the owner-approval environment credential.

The verified immutable identifiers are:

  • OpenMined organization: 30185530.
  • OpenMined/firecall-control: 1345147942.
  • OpenMined/infrastructure: 510317576.

Use these subject aliases only as documentation shorthand:

Alias Exact sub value
control-ref repo:OpenMined/firecall-control:ref:refs/heads/main
infra-ref repo:OpenMined/infrastructure:ref:refs/heads/main
infra-live repo:OpenMined/infrastructure:environment:firecall-live

Create one flexible credential for every row in this closed contract:

Identity or mode Exact reusable workflow Subject Event
Open privileged-open.yml control-ref workflow_dispatch
Close privileged-close.yml control-ref workflow_dispatch
Sweep schedule privileged-sweep-schedule.yml control-ref schedule
Sweep manual privileged-sweep-manual.yml control-ref workflow_dispatch
Break glass privileged-breakglass.yml control-ref workflow_dispatch
Capacity bootstrap firecall-capacity-bootstrap-privileged.yml control-ref workflow_dispatch
Capacity reconcile push privileged-capacity-reconcile-push.yml control-ref push
Capacity reconcile manual privileged-capacity-reconcile-manual.yml control-ref workflow_dispatch
Audit verify schedule privileged-audit-verify-schedule.yml control-ref schedule
Audit verify manual privileged-audit-verify-manual.yml control-ref workflow_dispatch
Break-glass audit schedule privileged-breakglass-audit-reconcile-schedule.yml control-ref schedule
Break-glass audit manual privileged-breakglass-audit-reconcile-manual.yml control-ref workflow_dispatch
Assignment audit schedule privileged-assignment-audit-schedule.yml control-ref schedule
Assignment audit manual privileged-assignment-audit-manual.yml control-ref workflow_dispatch
Classic reduction schedule firecall-classic-reduction-schedule-privileged.yml infra-ref schedule
Classic reduction manual firecall-classic-reduction-manual-privileged.yml infra-ref workflow_dispatch
Standing apply firecall-standing-privileged.yml infra-live workflow_dispatch
Trust apply firecall-trust-privileged.yml infra-live workflow_dispatch
Trust drift schedule firecall-trust-drift-schedule-privileged.yml infra-ref schedule
Trust drift manual firecall-trust-drift-manual-privileged.yml infra-ref workflow_dispatch
Image build or cleanup firecall-images-privileged.yml infra-live workflow_dispatch
Image idle reconcile firecall-image-idle-privileged.yml infra-ref schedule
Image seal firecall-image-seal-privileged.yml infra-ref workflow_run
Image seal recovery firecall-image-seal-recovery-privileged.yml infra-live workflow_dispatch
Image inventory firecall-image-inventory-privileged.yml infra-ref schedule
Image pin firecall-image-pin-privileged.yml infra-ref workflow_run
Image pin continuation firecall-image-pin-continuation-privileged.yml infra-ref workflow_run
CA roots firecall-ca-roots-privileged.yml infra-live workflow_dispatch
Credential rotation firecall-credentials-privileged.yml infra-live workflow_dispatch
Policy apply firecall-policy-privileged.yml infra-live workflow_dispatch
Cloudflare apply firecall-cloudflare-privileged.yml infra-live workflow_dispatch
Zone-policy apply firecall-zone-policy-privileged.yml infra-live workflow_dispatch
Cloudflare drift firecall-cloudflare-drift-privileged.yml infra-ref schedule
Preflight firecall-preflight-privileged.yml infra-ref workflow_dispatch
Activation verify firecall-activation-verify-privileged.yml infra-ref workflow_dispatch
Load qualification firecall-load-qualification-privileged.yml infra-live workflow_dispatch
Control monitor schedule firecall-control-monitor-schedule-privileged.yml infra-ref schedule
Control monitor manual firecall-control-monitor-manual-privileged.yml infra-ref workflow_dispatch
Security rescan schedule firecall-rescan-schedule-privileged.yml infra-ref schedule
Security rescan manual firecall-rescan-manual-privileged.yml infra-ref workflow_dispatch
Scanner DB schedule firecall-scanner-db-update-schedule-privileged.yml infra-ref schedule
Scanner DB manual firecall-scanner-db-update-manual-privileged.yml infra-ref workflow_dispatch
Tool cache push firecall-tool-acquire-push-privileged.yml infra-ref push
Tool cache manual firecall-tool-acquire-manual-privileged.yml infra-ref workflow_dispatch
Retention schedule firecall-retention-schedule-privileged.yml infra-ref schedule
Retention recovery firecall-retention-recovery-privileged.yml infra-ref workflow_dispatch
Node-tuple monitor schedule firecall-node-tuple-monitor-schedule-privileged.yml infra-ref schedule
Node-tuple monitor manual firecall-node-tuple-monitor-manual-privileged.yml infra-ref workflow_dispatch

Render every FIC JSON document from this table and compare it with a checked-in golden file.

CI rejects a missing row, extra credential, wrong subject, wrong workflow, or non-40-character workflow pin.

Each public entry workflow uses the fully qualified repository form with one 40-character commit pin.

Do not use a same-repository ./ call or a branch, tag, or symbolic ref.

Only a reusable workflow job receives the job_workflow_ref claim.

Create one normal-control federated credential for each open, close, and sweep privileged workflow.

Use this exact expression shape for open:

claims['sub'] eq 'repo:OpenMined/firecall-control:ref:refs/heads/main' and claims['repository_id'] eq '1345147942' and claims['repository_owner_id'] eq '30185530' and claims['job_workflow_ref'] eq 'OpenMined/firecall-control/.github/workflows/privileged-open.yml@<approved-40-character-commit>'

The expression contains exactly those four eq clauses.

The FIC generator rejects every other claim lookup and every wildcard.

Store the approved reusable-workflow commit beside each flexible credential in trust state.

Each event-specific reusable workflow has two mutually exclusive jobs.

A credentialless rejection job fails on a wrong event or run attempt.

The privileged job starts only for the exact event and github.run_attempt == 1.

Give id-token: write to the exact caller uses: job and that guarded called privileged job. The rejection job receives no token permission.

Before Azure login, that job verifies its exact github.workflow_ref, caller github.workflow_sha, exact job.workflow_ref, and job.workflow_sha.

It compares the caller SHA with the current protected main SHA through the GitHub API.

Each public caller has fixed event-specific jobs. Each job calls only its matching fully qualified reusable path at a 40-character commit.

The caller cannot supply a mode value to the privileged workflow.

An old queued first attempt fails that comparison even when its reusable-workflow SHA remains approved.

Never use GitHub's full, failed-job, or single-job re-run control for a privileged Firecall job.

A legitimate retry creates a new idempotent dispatch and new workflow run.

An unrelated main commit does not change the fully qualified reusable-workflow pin.

To change a privileged reusable workflow, merge the new version while callers still pin the prior commit.

Pause its dispatch and prove no queued or in-progress first-attempt run remains.

The protected trust apply replaces the credential's approved job_workflow_ref commit.

The old caller then fails closed until a protected pull request changes its pin to the new commit.

Prove the old pin and every old run cannot authenticate before dispatch resumes.

Changing the trust privileged workflow itself requires the authenticated owner trust-bootstrap path and explicit approval.

Remove the prior flexible credential before the new reusable workflow receives authority.

The closed table is the sole workflow, subject, and event authority.

Do not derive one identity from another identity's expression.

Generate and test each service principal's exact FIC independently from its one or two declared rows.

The image-builder service principal accepts only this exact expression:

claims['sub'] eq 'repo:OpenMined/infrastructure:environment:firecall-live' and claims['repository_id'] eq '510317576' and claims['repository_owner_id'] eq '30185530' and claims['job_workflow_ref'] eq 'OpenMined/infrastructure/.github/workflows/firecall-images-privileged.yml@<approved-40-character-commit>'

No FIC uses a branch or tag for job_workflow_ref.

Set the only audience to api://AzureADTokenExchange.

Grant id-token: write on both the exact caller uses: job and its matching reusable privileged job. A called workflow cannot elevate caller permissions.

Run every privileged job on a fresh GitHub-hosted runner VM.

Never execute pull-request code, a pull-request checkout, or a pull-request artifact in that job.

Checkout only the protected main commit that the reusable workflow verifies before Azure login.

The FIC does not bind a source commit SHA.

Set every job and step permission to the minimum listed contract.

Set permissions: {} at the top of every Firecall workflow.

Grant only these per-job automatic GITHUB_TOKEN permissions:

Job class Exact permissions
Credentialless source validation contents: read
Standard privileged Azure or Cloudflare job contents: read, id-token: write
Control close or sweep actions: read, contents: read, id-token: write
Break-glass approval and consumption actions: read, contents: read, issues: write, id-token: write
Image-seal or image-pin artifact consumer actions: read, contents: read, id-token: write
Cloudflare edge cache-evidence consumer actions: read, contents: read, id-token: write
Image-idle observer actions: read, contents: read, id-token: write
Classic authentication-denial cutover actions: write, contents: read
External control monitor contents: read, id-token: write
Keyless evidence signer without Azure contents: read, id-token: write
Scanner-database publisher contents: read, id-token: write

All unlisted permissions remain none.

Apply each table row to the caller uses: job. The called job declares the same or a narrower permission set.

The Classic authentication-denial cutover job is the only Firecall job with actions: write through GITHUB_TOKEN.

It receives no id-token, Azure, App, Key Vault, Cloudflare, package, or source-write credential.

It uses firecall-live owner approval and current protected infrastructure main.

GitHub cannot restrict its Actions write permission to one historical run.

The owner accepts that short-lived repository-wide Actions mutation authority for the cutover run.

Use GitHub App installation tokens, not GITHUB_TOKEN, for cross-repository API calls or pull requests.

No job receives pull-requests: write, packages: write, deployments: write, checks: write, or security-events: write through GITHUB_TOKEN.

Only the exact caller uses: job and matching privileged or keyless-signing called job receive id-token: write.

Disable credential persistence in checkout and remove every temporary credential file before job exit.

Record the non-secret ImageOS, ImageVersion, architecture, workflow commit, and runner class in signed evidence.

Keep one reviewed allowlist of runner image versions for privileged jobs.

An unlisted image blocks credential acquisition and alerts before OIDC or Key Vault access.

GitHub can mutate a hosted image behind its version label.

The owner must accept GitHub Actions and its hosted image as privileged trust holders.

Attach firecall-live only when the flexible credential uses its exact environment subject.

Do not attach another environment or use an environment-subject fallback.

Record the decoded non-secret claims from a diagnostic run before the trust apply.

Prove the live environment subject succeeds only after approval.

Prove the ref subject, wrong environment, unapproved environment job, and self-approval each fail.

Do not print or persist the raw OIDC token.

If flexible credentials or required claims are unavailable, block activation.

Do not configure a repository-and-branch fallback or change the organization OIDC subject template.

The AzureAD Terraform provider does not manage these flexible claim expressions.

Terraform creates the Entra applications, service principals, owners, Azure roles, and assignments.

A reviewed Microsoft Graph reconciler creates the flexible federated credentials after that apply.

Place it at scripts/ci/firecall-flexible-fic.sh.

Store one complete JSON request per credential under terraform/stacks/azure/firecall-trust/prod/firecall-federated-credentials/.

Each document includes:

  • Issuer https://token.actions.githubusercontent.com.
  • Audience api://AzureADTokenExchange.
  • One deterministic credential name.
  • claimsMatchingExpression.languageVersion equal to 1.
  • The complete exact expression.
  • No subject value.

The reconciler uses az rest against this Microsoft Graph endpoint:

https://graph.microsoft.com/beta/applications/{applicationObjectId}/federatedIdentityCredentials

It accepts only check and create-missing modes during the normal trust apply.

It parses checked-in JSON files and never builds JSON or expressions from shell interpolation.

It lists every credential, canonicalizes the response, and compares every security field exactly.

It creates a credential only when its deterministic name is absent.

It rereads the new credential and requires an exact match before success.

It fails on a changed credential, unexpected credential, duplicate name, Graph error, or unreadable response.

It never replaces or deletes a credential automatically.

A replacement needs a separate owner-approved removal command with one exact application and credential ID.

Run the reconciler only in the protected trust-root workflow on infrastructure main.

Pull-request jobs validate only the local JSON schema and exact expected expressions.

They receive no Azure or Microsoft Graph token.

Give the trust-root service principal Microsoft Graph Application.ReadWrite.OwnedBy with administrator consent.

Make that service principal an owner of only the Firecall Entra applications it manages.

Do not grant Application.ReadWrite.All.

Record the Graph request IDs and credential object IDs without recording tokens or request bodies containing credentials.

A scheduled protected job runs read-only comparison and alerts on drift.

Before activation, exchange one real GitHub OIDC token from each exact reusable workflow.

Also prove that the wrong workflow, repository identifier, owner identifier, ref, and missing claim each fail.

24.1.1 Trust-root bootstrap and recovery

The trust-root workflow cannot create its own first federated credential.

Use one owner-run bootstrap for only that circular dependency.

Place it at scripts/ops/firecall-trust-bootstrap.sh.

Group A adds the closed bootstrap mechanism to ADR-0012 and CLAUDE.md.

Its authority source is platform/firecall/contracts/owner-bootstrap-operations.json.

Group A publishes that manifest with no active operation. It grants no live authority.

Each future active row binds one exact operation ID, script, subcommand, scopes, roles, mutation class, and one-time or recovery state.

A Terraform mutation also binds its exact root, state container and key, and saved-plan digest. A non-Terraform mutation requires those fields to be null.

Adding or activating a row requires the same protected change to amend ADR-0012 and CLAUDE.md.

Both documents contain one exact hidden authority marker. It binds the raw manifest SHA-256, generation, source commit, and sorted active operation IDs. Credentialless CI requires both markers to equal the canonical manifest. Activation and consumption therefore change the manifest, ADR, and operating contract together.

A missing, incomplete, disabled, or mismatched row denies the operation.

The following list contains candidate operations. It does not authorize them:

  • The exact tfstate-bootstrap/prod saved apply for dedicated Firecall backend containers.
  • The first standing-control saved apply.
  • Trust seed, adoption, observer consent, and recovery scripts.
  • The first Firecall AKS child-resource saved apply and pool rotation.
  • The two fixed Policy-definition saved applies.
  • The first exact firecall-policy/dev assignment saved apply.
  • The exact Firecall budget saved applies.
  • Edge-secret state adoption and zero-change verification.
  • Key, data, legal-hold, checkpoint, and manual credential bootstraps.

The mechanism permits no generic root, discovery loop, unsaved apply, auto-approve, Terraform destroy, wildcard scope, unapproved deletion, or unrelated resource change.

Each Terraform operation requires a prior protected script commit, exact root allowlist, remote state, read-only lock file, and saved-plan digest.

The manifest's source_commit pins one prior source commit and its complete repository tree. A later protected activation commit carries the rows and cannot reference its own commit SHA. Its fixed source_execution_mode is sandboxed_source_commit_archive.

The wrapper extracts only the fixed paths required by the selected operation profile from source_commit. It rejects Git metadata. It executes the pinned script inside a filesystem sandbox. The pinned wrapper maps each exact (script, subcommand, mutation_class) tuple to one closed sandbox profile. The caller cannot provide a path, mount request, tool list, credential destination, or API origin. Each profile fixes source paths, host executable paths, credential transport, file descriptor numbers, sandbox destinations, and API origins. An absent mapping fails. The wrapper rejects a missing, linked, or caller-writable executable. It rejects an undeclared input, inherited credential agent, or unlisted environment variable. The owner host operating system, kernel, system libraries, tools, DNS, and network remain explicit privileged trust boundaries. A runtime-generated digest of that same host cannot remove this trust. The sandbox exposes the fixed source paths read-only and one empty operation scratch directory. It exposes no repository checkout, .git, home directory, host temporary directory, unrelated credential cache, or container socket. The control-repository profile exposes only the wrapper, bootstrap script, and complete manifest-bound control seed from the source commit. The wrapper reads protected main through authenticated GitHub REST against numeric repository ID 510317576. It also verifies that current protected main contains the active row and matching ADR-0012 and CLAUDE.md text before it enters the sandbox.

It requires source_commit to be an ancestor of the current protected main. Credentialless CI requires the wrapper and every active script to exist at that commit and match its exact bytes. It enforces the match while the current or prior protected manifest contains an active row. Consume the row in a separate protected change before the wrapper or active script changes.

The embedded tests change an undeclared activation-checkout file and prove it is absent from the fixed source archive. They also prove that an injected untracked file, .git, the host home, and the container socket are absent. They prove that an inherited credential agent, caller-writable fixed tool, missing active row, wrong marker, wrong ancestry, and script drift fail.

Every manifest change increments generation by exactly one. Existing rows never disappear or change their operation bindings.

A new row can start disabled or active. Existing rows can move only from disabled to active or active to consumed. An active row keeps the same source_commit until consumption. A consumed row is a permanent immutable tombstone.

Consumption binds the prior active generation. An operation that did not run uses outcome revoked_without_execution with a matching zero-change readback.

The authenticated owner must inspect each plan and give the approval required by the repository operating contract.

The script records source, plan, state, actor, role, and live readback digests without state or secret content.

Every temporary role expires or is removed, and the script waits for issued tokens before routine automation starts.

After bootstrap, only an active exact recovery or rotation row keeps this exception.

The script requires the verified organization and infrastructure repository numeric IDs.

Use this bootstrap and adoption order:

  1. Apply the protected tfstate-bootstrap/prod change for the dedicated backend containers.
  2. Run the first firecall-control/dev saved plan with the authenticated owner.
  3. Create the control, session, and image-build groups through that final root.
  4. Create its storage, Key Vault, gallery, VNets, subnets, locks, and other standing objects through that final root.
  5. Run firecall-trust-bootstrap.sh seed for the circular objects and required custom role definitions.
  6. Import every Terraform-owned seed object into firecall-trust/prod.
  7. Verify a zero-change plan for every adopted object.
  8. Create temporary exact-role assignments for the key, data, and manual credential scripts.
  9. Pre-create every key, App-key, HMAC, Cloudflare-token, tunnel-token, audience, status, and checkpoint object.
  10. Populate the initial App and Cloudflare primary slots through the manual credential JIT path.
  11. Apply the two owner-only Policy roots and their final states.
  12. Import the four adopted edge secrets and prove their zero-change state.
  13. Apply a trust-root plan that creates the trust-observer application but omits its FIC.
  14. Let a directory administrator grant and prove the one exact observer Graph consent.
  15. Apply a second trust-root plan that creates the trust-observer FIC only after consent evidence passes.
  16. Run each consumer canary and select only proved primary slots.
  17. Remove every temporary bootstrap assignment and wait for each issued token to expire.

The final trust plan includes exact secret-object, Policy-assignment, backend, standing-resource, observer, and monitor assignments.

Use scripts/ops/firecall-data-bootstrap.sh for fixed secret objects and the initial security-status object.

Its checked-in platform/firecall/contracts/data-bootstrap.json manifest contains every exact secret, container, Blob path, placeholder schema, and expected owner.

Give the authenticated owner a time-limited vault-parent custom role.

That role permits vault metadata read and secret metadata, get, and set only.

It grants no key, certificate, sign, decrypt, unwrap, export, delete, purge, backup, restore, policy, or role operation.

The script creates each absent secret with one random nonfunctional placeholder and content_type=firecall-bootstrap-placeholder.

It rereads each exact object and records only its versionless ID, initial version ID, ETag, content type, and value digest.

It never prints, artifacts, or records a placeholder value.

It fails on a wrong content type, unexpected fixed-prefix object, existing real value, or manifest mismatch.

Give the owner six separate time-limited Blob roles.

Five roles cover the security, Trivy, ClamAV, node-tuple, and access-policy current-object paths in section 15.

The sixth role covers only firecall-image-build-operations/operation.lock.

Each role permits direct read and create on one object, but no list, overwrite, delete, lease, tag, or sibling path.

Give the owner one separate temporary ARM role on only the image-build-operations container.

That role permits only container read and setLegalHold/action.

The script creates each generation-zero uninitialized record with If-None-Match: *.

It creates the immutable empty operation-lock document with generation zero and If-None-Match: *.

It applies legal-hold tag firecalloperationlock before any routine lock role exists.

It rereads each ETag, schema, canonical digest, container policy, and exact role condition.

It also rereads the operation-lock ETag, schema, canonical digest, container policy, and exact role condition.

An existing exact canonical object is success; any other existing content fails bootstrap.

After the four edge secrets enter final edge state, remove the owner's vault, seven Blob, legal-hold, and edge-state assignments.

Wait for every issued Key Vault and Storage token to expire before the protected edge or rescan workflow starts.

The owner receives time-limited Blob access to the standing-control state key for step 2.

That first apply creates resources directly in their final state and requires no ARM import.

Remove the owner's standing-state access after trust assigns the routine standing identity.

The script seed mode creates or verifies only:

  • The firecall-trust Entra application and service principal.
  • The exact firecall-trust-privileged.yml flexible federated credential.
  • The fixed Firecall custom role definitions required by the trust root.
  • The trust identity's exact backend and resource-scoped role assignments.
  • Microsoft Graph administrator consent for Application.ReadWrite.OwnedBy.

It does not create a resource group, state container, operational storage object, Key Vault, Policy definition, or Policy assignment.

The final Terraform roots create those objects directly.

Store an exact adoption manifest under terraform/stacks/azure/firecall-trust/prod/bootstrap-imports.json.

Each entry binds one Terraform address, Azure or Graph object ID, deterministic name, scope, expected tags, and source digest.

The authenticated owner receives time-limited Blob access to only the trust state key during adoption.

The adoption script verifies every live ID, tenant, application ID, principal ID, role digest, scope, and deterministic assignment GUID.

It imports these Terraform-owned seed objects:

  • The trust application.
  • The trust service principal.
  • Every fixed custom role definition.
  • Every trust-root Azure role assignment, including its exact backend assignment.

Set prevent_destroy on each adopted application, service principal, custom role, and bootstrap role assignment.

Remove the owner's temporary state access after the zero-change adoption plan.

The flexible federated credential and Graph administrator consent remain script-owned.

The Graph reconciler verifies both on every trust apply.

No object can remain jointly owned by the script and Terraform.

The bootstrap never grants Application.ReadWrite.All, Owner, or subscription Contributor.

It creates no other flexible credential.

It writes one sanitized, signed evidence file with object IDs, conditions, digests, and Graph request IDs.

It records no access token, private key, state content, or secret value.

After bootstrap, firecall-trust.yml calls only firecall-trust-privileged.yml on protected infrastructure main.

That workflow creates the other Firecall applications as objects it owns.

It creates and verifies their checked-in flexible credentials through Microsoft Graph.

It assigns only fixed Firecall roles at their exact checked-in resource scopes.

It cannot create or change a custom role definition or Azure Policy definition.

It has no subscription-scope role-assignment authority.

Every role-assignment scope and permitted role-definition ID appears in a checked-in closed set.

Adding a scope or role definition requires another owner-run bootstrap change.

The protected trust workflow uses a reviewed saved plan bound to stack, commit, lock file, and manifest digest.

An owner reads that plan and approves the protected environment before apply.

To revoke trust, an owner disables the service principal and removes only its exact federated credential.

Wait for existing Azure tokens to expire before claiming complete revocation.

After adoption, recovery runs the bootstrap in verify-only mode.

It never creates, replaces, or deletes an adopted Terraform object.

A lost adopted object uses a reviewed Terraform recovery plan and its final state.

The script can recreate only its script-owned flexible credential after explicit owner approval.

A mismatch stops recovery and requires a new reviewed bootstrap plan.

24.1.2 Dedicated applies and Terraform state

Exclude every Firecall v2 root from .github/workflows/terraform.yml discovery and dispatch.

The exclusions cover:

terraform/stacks/azure/firecall-control/dev
terraform/stacks/azure/firecall-aks/dev
terraform/stacks/azure/firecall-trust/prod
terraform/stacks/azure/firecall-policy-definition/prod
terraform/stacks/azure/firecall-image-build-policy/prod
terraform/stacks/azure/firecall-budget/prod
terraform/stacks/azure/firecall-policy/dev
terraform/stacks/cloudflare/firecall/dev
terraform/stacks/cloudflare/firecall-zone-policy/prod

Group A removes pull_request from the current generic Terraform workflow.

E1 source removes the plan FIC, its application and service principal, every role assignment, ARM_CLIENT_ID_PLAN, its outputs, and its input wiring.

Keep the control-drift entry until live readback proves the complete surface absent. Remove that entry in the protected evidence change.

Do not claim live pull-request credentiallessness from the source change.

Apply that trust reduction through the authenticated owner-only ci-identities saved-plan path, then wait for every issued token to expire.

After live readback proves the complete set absent, no pull-request identity can plan or apply a Terraform root.

The generic protected-main identity cannot plan or apply these Firecall roots.

Each Firecall pull request runs only credentialless format, validation, policy, and mocked contract checks.

Every Firecall AzureRM backend block sets:

use_oidc          = true
use_azuread_auth  = true

No backend can request an account key, account SAS, service SAS, or user-delegation key.

The exact owner-bootstrap wrapper overrides only use_oidc=false for Azure CLI authentication.

It keeps use_azuread_auth=true and rejects any shared-key backend field or environment variable.

Every Firecall AzureRM provider block sets:

resource_provider_registrations = "none"

No Firecall identity can register a resource provider.

Preflight requires registered state for Microsoft.Authorization, Microsoft.Compute, Microsoft.Consumption, Microsoft.ContainerService, Microsoft.Insights, Microsoft.KeyVault, Microsoft.ManagedIdentity, Microsoft.Network, Microsoft.Resources, and Microsoft.Storage.

An unregistered or registering provider blocks the applicable saved plan before mutation.

Pin hashicorp/azuread to 3.9.0 and commit its complete linux_amd64 lock entry.

The protected trust workflow enables AzureAD OIDC authentication only after the exact GitHub OIDC Azure login.

It rejects AzureAD client secrets, certificates, managed identity, and Azure CLI fallback.

The owner trust bootstrap uses only a verified interactive Azure CLI session in the exact tenant.

It disables AzureAD OIDC for that local process and rejects every client-secret or certificate variable.

Both modes read back tenant, client, principal, and application ownership before any Graph mutation.

Use these exact protected backend owners:

Use existing account sttfstateopenmined in rg-tfstate-prod-eastus.

Extend the existing terraform/stacks/azure/tfstate-bootstrap/prod root with every dedicated Firecall state container.

That existing root is the sole Terraform owner of those containers and their lifecycle protection.

Set prevent_destroy on each container and keep account versioning and 30-day container soft delete enabled.

Apply this protected state-storage change before any Firecall root initializes its backend.

Do not let a Firecall stack own or create its own backend container.

Replace any inherited account-scope Blob data assignment for generic CI with existing-container assignments before container creation.

The generic apply identity keeps backend access only to tfstate and tfstate-prod.

It receives no data-plane role on a dedicated Firecall container.

The state-storage root can manage the container ARM resources without reading their Blob contents.

Activation checks effective role assignments at the account and every parent scope.

An inherited Blob reader, contributor, owner, or wildcard data role blocks activation.

Root Container and key Writer
Standing control firecall-standing-state/azure/firecall-control/dev.tfstate Standing-resource identity
Firecall AKS pool firecall-standing-state/azure/firecall-aks/dev.tfstate Standing-resource identity
Trust root firecall-trust-state/azure/firecall-trust/prod.tfstate Trust-root identity
Fixed policy definition firecall-policy-definition-state/azure/firecall-policy-definition/prod.tfstate Owner bootstrap only
Image-build policy and assignment firecall-policy-definition-state/azure/firecall-image-build-policy/prod.tfstate Owner bootstrap only
Firecall budgets and cost alerts firecall-policy-definition-state/azure/firecall-budget/prod.tfstate Authenticated owner only
Policy assignment firecall-policy-state/azure/firecall-policy/dev.tfstate Policy-apply identity
Cloudflare edge firecall-edge-state/cloudflare/firecall/dev.tfstate Cloudflare-apply identity
Cloudflare zone policy firecall-zone-policy-state/cloudflare/firecall-zone-policy/prod.tfstate Zone-policy-apply identity

Each writer receives read, write, and lease authority on only its listed key.

Section 15.1 exact-path conditions apply to every backend assignment.

The AzureRM backend also lists workspaces during terraform init.

Permit Blob.List only when the request prefix equals the exact state-key bytes followed immediately by env:.

For standing control, express the condition as concat("azure/firecall-control/dev.tfstate", "env:").

Create one equivalent condition for each table key.

Direct Blob access remains StringEquals on only the base state key.

Do not permit writes under an env: prefix.

Firecall roots use only Terraform's default workspace.

Qualification runs real terraform init and plan with every dedicated identity.

The protected tfstate-bootstrap/prod apply pre-creates each state container.

The exact cluster-owner bootstrap creates all three Kubernetes Lease objects before any protected workflow can run.

No edge or credential operation Lease exists in Azure Storage.

It applies prevent_destroy, versioning, and soft delete to every state container. No age-delete rule applies to a Terraform state key. The separate saved-plan prefix has the bounded lifecycle defined below.

The generic Terraform identities receive no Blob data role on these containers.

The standing-resource identity accepts only firecall-standing-privileged.yml.

It manages only the two standing roots and their two exact state keys.

Its custom Azure roles cover standing Firecall resource groups, the exact dev agent-pool child scope, and exact peering scopes.

A separate custom assignment covers only read and join actions on the live AKS subnet resource ID from section 21.6.

Give this identity Microsoft.Compute/locations/usages/read at subscription scope.

Give it Microsoft.Network/locations/usages/read at subscription scope.

It uses those actions only for East US BS-family, Total Regional vCPU, and Standard public-address activation checks.

It receives no quota-write or sibling Compute-write action at that scope.

The agent-pool and AKS-side peering child scopes do not exist before first creation.

Do not grant parent-cluster or VNet management actions to the standing identity. After the owner bootstrap, the only parent-cluster exception is Cluster User credential retrieval for the immutable admission-proof readback below.

Use scripts/ops/firecall-standing-bootstrap.sh once after the Firecall VNet exists.

Before its first AKS agent-pool plan, the script verifies both live node-isolation admission bindings.

It verifies the reviewed system-DaemonSet tuple digest and exact scheduler username.

It runs a denied tenant toleration probe and a denied nonscheduler Binding probe.

It stops before any Azure write when one precondition fails.

Grant an authenticated owner temporary backend authority for only azure/firecall-aks/dev.tfstate.

That authority includes exact-key read, create, write, and lease operations.

Its only list authority uses concat("azure/firecall-aks/dev.tfstate", "env:").

The authenticated owner runs the reviewed firecall-aks/dev Terraform plan against that remote state.

That first apply creates only the exact Firecall pool and exact AKS-side peering.

It does not create a role assignment.

After both children exist, run the protected trust apply against its final trust state.

That apply creates the standing identity assignments directly at only those two child resource IDs.

No owner-created assignment needs import or adoption.

Record the child IDs, deterministic assignment IDs, trust-state digest, and live role comparison.

Remove the owner's exact backend assignment after a zero-change AKS plan and live child and assignment verification.

Wait for the owner's issued storage token to expire before routine standing automation starts.

The script rereads the state, child IDs, parent resources, role assignments, and unrelated sibling digests.

It proves that no existing agent pool or peering changed.

Later protected standing applies can update only those exact children.

They accept only plans that update the existing child resource IDs in place.

They reject replacement, deletion, a temporary pool name, a sibling pool, or any write at the parent cluster.

VM size, subnet, disk, availability zones, and similar rotation fields require an active exact pool-rotation operation row and its bound runbook section.

Use this rotation order:

  1. Disable creation and close every Firecall session.
  2. Verify family and Total Regional vCPU quota for the old pool, temporary sibling, and upgrade surge.
  3. Merge exact admission tuples and node-prefix rules for both planned pool generations.
  4. Give the authenticated owner temporary access to only the AKS state key and required parent agent-pool operations.
  5. Apply one saved plan with the reviewed provider rotation name and no unrelated cluster change.
  6. Verify workloads, system tuples, resolved image digests, peering, subnet, taint, label, and node capacity on the replacement.
  7. Run final trust to create the final child assignments and remove every stale deleted-child assignment.
  8. Remove the temporary admission generation and prove a zero-change AKS plan.
  9. Remove the owner's temporary assignments and wait for every issued token to expire.
  10. Run full dark preflight before creation resumes.

The owner rotation never grants the standing identity a parent-cluster or sibling-pool role.

The owner does not create a role assignment outside the protected trust state.

If either child is deleted, routine automation fails closed.

Only the owner bootstrap can recreate it and restore child-scoped authority.

It cannot manage role assignments, policy, federated credentials, Key Vault secret values, or CA signing keys.

terraform/stacks/azure/firecall-aks/dev owns only the new Firecall node pool and AKS-side Firecall peering.

The shared aks-platform/dev state never owns that pool.

No state migration is required because the pool does not exist before this stack applies.

Every protected apply acquires the backend lease and rejects another active operation.

Each apply uses one saved plan bound to exact source commit, root, lock file, and manifest digest.

No workflow uploads a plan that contains a secret value.

Activation tests concurrent lock denial and unauthorized reads by every other Firecall identity.

24.1.3 AKS authorization and protected verifier identities

The live dev AKS cluster uses Azure RBAC authorization.

AKS supports native Kubernetes RBAC and Azure RBAC on the same cluster.

Use native Kubernetes RBAC for these fine-grained, single-cluster permissions.

Give each external Entra application Azure Kubernetes Service Cluster User Role only on the exact dev cluster.

That Azure role permits credential retrieval only.

Give it no Azure Kubernetes DataAction role.

Create one native Role or ClusterRole per external workflow and bind its exact Entra service-principal object ID.

Store the exact expected Kubernetes username, application client ID, object ID, and tenant ID in the protected trust manifest.

Before activation, each identity runs SelfSubjectReview and a fixed positive and negative authorization canary.

The observed username must equal the protected value and map to the expected object ID.

A username, tenant, client, or object mismatch blocks activation.

Use resourceNames for fixed-name get, update, patch, and delete operations.

Use narrow namespace scope and exact verbs when a dynamic list is required.

Use ValidatingAdmissionPolicy as a second control for every external create or mutation with a dynamic name.

Bind each policy to the exact observed Entra username and protected contract digest.

The preflight policy permits only fixed fixture Pods and one fixed synthetic Binding probe.

The CA policy permits only the named certificate-rotator Lease and its exact transition schema.

The edge policy permits only the named rollout ConfigMap, operation Lease, and one Deployment annotation.

The credential policy permits only the fixed rotation Job prefix, ServiceAccount, image, command, arguments, volumes, resources, deadline, and owner labels.

All bindings use Deny, compiled expressions, and failurePolicy: Fail before any external identity obtains a credential.

The GitOps source owns these Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, admission policies, and bindings.

The trust manifest binds every source digest and RoleBinding subject to the matching Entra object.

Qualification logs in as each real identity and tests its complete positive and negative permission matrix.

Create one dedicated preflight Entra application and service principal.

Grant it Azure Kubernetes Service Cluster User on only the exact dev cluster.

Bind it to narrow Kubernetes Roles in only the two fixture namespaces.

The two fixture namespaces copy the exact tenant and preview isolation labels.

They contain no application, credential, service, or trusted ServiceAccount.

Default-deny policy, restrictive quota, and Pod Security apply in both fixture namespaces.

The Roles can create fixed preflight Pods and both Binding request forms.

It can get and delete only the fixed preflight Pod names.

Kubernetes returns the complete object for each permitted Pod GET.

It cannot list or watch Pods, read logs or Secrets, or use exec, attach, or port-forward.

It cannot read or change a Deployment, Service, policy, role, ConfigMap, Lease, or application Pod.

One ValidatingAdmissionPolicy binds its exact username to reviewed test names, namespaces, image digests, commands, resources, and no-token ServiceAccounts.

A second independent policy limits its Binding attempts to one fixed synthetic Pod UID and nonexistent Firecall-prefix node.

The scheduler-only policy must also deny that Binding attempt.

Both policies and bindings must report compiled status before the workflow obtains a cluster credential.

The flexible OIDC claim, short token lifetime, exact RoleBindings, and admission policy bind authority to that workflow.

The workflow polls only its fixed Pod names and records sanitized status in a keyless-signed artifact.

A missing policy, unexpected diff, or failed callback blocks activation.

Create one separate activation-verifier Entra application and service principal.

Grant it Cluster User on only the exact dev cluster.

Give it read-only Kubernetes RBAC for exact Firecall namespaced objects and reviewed cluster policy names.

It can get exact Certificates, Issuers, ConfigMaps, Leases, Services, RBAC, quotas, and policies.

It can get and list Deployments, ReplicaSets, Pods, EndpointSlices, and PodDisruptionBudgets only in both Firecall namespaces.

It can get and list DaemonSets and Pods only in kube-system for the reviewed tuple check.

It can get and list Nodes because Firecall node names do not exist before activation.

Those reads return complete objects, including Pod specifications, node addresses, commands, and literal environment values.

Firecall and system workloads must not contain a literal secret in those fields.

The verifier cannot read Secrets, logs, exec, attach, port-forward, events, or another application namespace.

It cannot create, update, patch, delete, bind, approve, or mutate any Kubernetes object.

It records only digests, names, UIDs, safe status fields, and resolved image IDs in signed evidence.

Standing, image, trust, and edge workflows verify their own Azure or Cloudflare resources.

The activation verifier does not receive Azure resource-read authority beyond Cluster User credential acquisition.

Create one separate load-qualification Entra application and service principal.

Its only FIC accepts firecall-load-qualification-privileged.yml through the owner-approved firecall-live environment.

Grant it Cluster User on only the exact dev cluster.

Bind it to one Kubernetes Role in firecall for only Pod firecall-node-credit-qualification.

It can create, get, watch, and delete only that Pod.

It cannot list Pods or read logs, Secrets, events, exec, attach, port-forward, or any other Kubernetes object.

One ValidatingAdmissionPolicy permits only exact drain and sustain Pod contracts.

Both contracts fix name, namespace, ServiceAccount, image digest, command, node selector, toleration, and security fields.

The Pod has no ServiceAccount token, network egress, host namespace, host mount, capability, writable root, or persistent volume.

Both use a 100m CPU request, 32Mi memory request, 1800m CPU limit, and 64Mi memory limit.

Drain mode targets 1800m CPU and has an eight-hour active deadline.

Sustain mode targets 700m CPU and has a three-hour active deadline.

The workflow changes mode only by deleting the old Pod, verifying absence, and creating the other exact contract.

The fixed helper performs only bounded CPU work and has no runtime control input.

Start-drain requires protected qualification_mode=true, no nonowner session, and a current node-tuple verdict before credential acquisition.

Switch-sustain requires the exact drain Pod and authenticated node credits below 15 for 15 minutes.

Stop accepts only the current qualification epoch and remains available after qualification-mode expiry.

It deletes the Pod at completion or cancellation and waits for verified absence.

The node-tuple observer supplies authenticated CPU-credit evidence; this identity receives no Azure metrics or resource authority.

Revoke it by removing its FIC, Cluster User assignment, RoleBinding, and admission binding, then wait for token expiry.

24.1.4 External control monitor

Create one dedicated control-monitor Entra application and service principal.

Its flexible credential accepts only the exact external infrastructure monitor workflow.

Grant it get on only the primary and secondary control-observer App-key secret objects.

Grant it read, exact-prefix list, and create-only append access to firecall-audit/control-workflow-monitor/.

Grant it read-only exact-prefix access to request receipts and lifecycle status.

Grant it read, conditional write, and lease authority on only firecall-control-monitor/checkpoint.json.

It receives only the two exact prefix-list grants above and no container-root or sibling list.

It cannot overwrite audit, delete Blob data, change a session, or manage a role, network, cluster, Gallery, or Cloudflare resource.

The monitor acquires the checkpoint lease before it reads GitHub or changes a checkpoint.

Before scheduling starts, an authenticated owner runs firecall-control-monitor-bootstrap.sh once.

Give that owner time-limited create-only access to the exact checkpoint and monitor-audit prefix.

The script creates schema-valid sequence zero with If-None-Match: * and appends its correlated immutable audit event.

It reads back the checkpoint, ETag, digest, and audit object before success.

An existing checkpoint makes bootstrap fail without a write.

Remove the temporary assignment and wait for the issued token to expire before the first scheduled poll.

The checkpoint stores a monotonic sequence, prior checkpoint digest, per-workflow high run number, observed run IDs, and evidence-retention deadlines.

It contains no email, subject key, reason, namespace, SSH address, token, log body, or artifact body.

Every update requires the prior ETag, increments the sequence by one, and writes one immutable correlated audit event first.

The monitor reads back the event, checkpoint, ETag, sequence, and digest before lease release.

An unreadable or malformed checkpoint, receipt, status prefix, or prior audit prefix fails closed and alerts.

It never initializes a replacement checkpoint when one should already exist.

Keep the checkpoint and its Blob versions for the Firecall service lifetime.

Expire a detailed run entry only after its GitHub evidence-retention deadline and one complete successful poll.

Keep each workflow's high run number permanently.

The monitor mints a read-only installation token and performs the checks in section 10.13.

It writes one immutable result with workflow states and evidence-presence digests for each run.

It never copies a GitHub log or artifact body into Azure audit.

Alert delivery uses the existing platform notification path outside the control repository.

24.1.5 Automated observer and reduction identities

Do not attach a scheduled FIC to a standing, trust-root, builder, sealer, edge, or zone-policy service principal.

Each scheduled mode uses a separate Entra application, service principal, FIC, custom role, and revocation path.

The trust observer runs the scheduled firecall-trust-drift.yml trigger every five minutes.

That trigger calls firecall-trust-drift-schedule-privileged.yml through its exact FIC.

A separate manual job calls firecall-trust-drift-manual-privileged.yml through its exact FIC.

Each five-minute scheduled run checks the declared machine-identity trust facts.

Give it this Microsoft Graph application permission with administrator consent:

  • Application.Read.All.

Give it no Graph write permission.

This permission exposes directory-wide application and service-principal metadata.

Give it Microsoft.Authorization/roleDefinitions/read at subscription scope.

Give it Microsoft.Authorization/roleAssignments/read only at every exact Firecall scope in the closed trust manifest.

It can read subscription role-definition metadata and Firecall assignment subjects, roles, scopes, conditions, and IDs.

It cannot list role assignments at an unlisted resource scope or perform any authorization write.

Owner acceptance of that read exposure is an activation gate.

Give it get on only the primary and secondary settings-observer GitHub App key objects.

The settings-observer App is installed only on OpenMined/infrastructure.

Give that App Metadata read, Contents read, Actions read, and Administration read.

Give it no write, secret, deployment, organization, or other repository permission.

The job compares every Firecall application, owner, password, certificate, FIC, role digest, and assignment with the closed trust manifest.

It also verifies the exact firecall-live environment ID, reviewer IDs, prevent_self_review=true, and protected-main deployment rule.

It requires branch protection and repository rules to forbid administrator bypass for the exact Firecall paths.

It verifies the exact Firecall machine applications, service principals, federated credentials, and declared Azure assignments.

Expose POST /api/v1/access-policy-canaries on the portal only to a current platform owner.

Require the exact portal Origin, Sec-Fetch-Site: same-origin, and current portal CSRF token.

The route accepts no target identity and starts one three-audience batch for the authenticated owner.

It returns three same-origin handoff URLs with the signed handoff value in the URL fragment.

Portal, workspace, and preview JavaScript clear that fragment and post it to /.firecall/v1/access-policy-canary.

Each target route accepts exactly one JSON object containing only the signed handoff.

Limit that decoded body to 2 KiB.

Require the exact media type, Origin, Sec-Fetch-Site: same-origin, current Access identity, route audience, and batch subject key.

The signed one-use handoff is the route-specific CSRF and idempotency credential.

Its random challenge identifier is the conditional-create object name.

The same valid handoff returns the prior result, and different bytes for its identifier fail.

No target route accepts a portal CSRF token, caller target, audience, subject key, batch, or object path.

The handoff binds batch, subject key, audience, issue time, and 15-minute expiry and permits one conditional challenge creation.

Use the shared Blob rate limiter for one outstanding batch per subject key and four batches per subject key each day.

Cap the service at 100 batches and 300 immutable challenge objects each UTC day.

Reject another outstanding batch, an unauthorized user, cross-subject-key use, duplicate audience, replay, stale handoff, and every limit excess.

Once each 24 hours, a platform owner completes fresh email one-time PIN authentication against the portal, workspace, and preview canary routes.

Before each canary, invalidate or isolate the Cloudflare global SSO session in a separate browser profile.

Each canary must cause a distinct Cloudflare authentication event.

The broker writes one create-once challenge under firecall-access-policy-evidence/challenges/.

It records only a random challenge digest, subject key, email subject key, identity-nonce key, audience, Access iat, access time, schema, and expiry.

It records no email, display name, raw subject, raw identity nonce, token, cookie, IP address, PIN, or authentication body.

The challenge expires after 15 minutes and remains immutable under the locked 90-day evidence policy.

The observer lists only the bounded challenges/ prefix and reads the three selected exact paths.

It requires each email subject key to equal the immutable challenge subject key. It requires three distinct audiences and identity-nonce keys.

Each Access iat and challenge receipt time must fit the configured clock window. The evidence does not contain the canonical email.

It conditionally creates consumed/<challenge_sha256>.json for each first correlation.

An existing exact consumption is valid only for the same immutable challenge, subject key, audience, and policy digests.

A conflicting reuse, reused identity nonce, expired challenge, or cross-audience challenge blocks the verdict and alerts.

It publishes a keyless-signed verdict under firecall-access-policy-evidence/checks/<epoch>/<digest>.json.

It uses If-None-Match: *, rereads that exact path, and verifies bytes, digest, signature, schema, and immutable-container state.

It never lists the checks/ prefix and cannot overwrite or delete evidence.

It leases and conditionally advances only firecall-access-policy-status/current.json.

It advances the pointer only after the exact evidence reread succeeds.

The verdict is valid for ten minutes and binds the last three successful human canaries, each no older than 24 hours.

The broker and control runner deny new sessions when that verdict is missing, stale, malformed, unsigned, or failed.

Close, sweep, break glass, and zero-resource proof do not use this gate.

It appends one immutable trust-drift/ audit object and alerts on any missing, extra, or changed field.

The final trust apply creates the observer application without directory consent.

An authenticated directory administrator then runs scripts/ops/firecall-trust-observer-bootstrap.sh.

The script verifies the exact application ID and grants only Application.Read.All.

It records consent object IDs and live read-only canaries, then enables no workflow itself.

The protected trust apply activates the observer FIC only after that evidence passes.

Recovery first revokes the observer FIC and its one consent, then waits for Graph tokens to expire.

The trust observer cannot repair drift, read a secret value except its App slots, or change an Azure or GitHub object.

The image-idle reconciler runs firecall-image-idle.yml every 30 minutes.

Its reusable workflow is firecall-image-idle-privileged.yml and its only event is schedule.

It can read, write, and lease only firecall-image-build-operations/operation.lock.

It can read the image-build resource inventory and read or replace only nsg-firecall-image-build.

It can restore only the complete checked-in idle NSG document after lease and active-run denial checks.

It cannot create or delete a VM, NIC, disk, address, Gallery object, lock, role, policy, or registry artifact.

An orphan resource alerts and requires the owner orphan-cleanup path.

The image-idle identity can deny a later build by corrupting the fixed NSG, but it cannot open inbound SSH beyond Policy.

The image-inventory observer runs firecall-image-inventory.yml every 30 minutes.

Its reusable workflow is firecall-image-inventory-privileged.yml and its only event is schedule.

Give it read-only Gallery-version and exact-lock actions plus shared-registry AcrPull.

Give it create-only access to firecall-security-evidence/image-inventory/* and firecall-audit/image-inventory/*.

It publishes signed quarantine and deletion proposals but cannot seal, pin, delete, or change an image.

Shared-registry read exposure matches the image-pin residual.

The Cloudflare drift observer runs firecall-cloudflare-drift.yml daily.

Its reusable workflow is firecall-cloudflare-drift-privileged.yml and its only event is schedule.

Give its Entra identity get on only the read-only Cloudflare observer-token secret.

That Cloudflare token has account Tunnel read, Access application and policy read, and exact-two-zone DNS read.

It also has SSL and Certificates Read on only those two zones.

Give it Access: Organizations Read on only the OpenMined account.

That permission exposes Zero Trust organization settings, including the global session duration.

The zone permission exposes certificate packs, SAN coverage, issuers, validity, and SSL settings for those zones.

It also has exact-two-zone Cache Rules read, Account Rulesets read, and Account Filter Lists read.

It has no Cloudflare edit, revoke, purge, token, billing, membership, firewall, or zone-settings permission.

Store primary and secondary observer-token slots, use 90-day expiry, and alert 14 days before expiry.

Rotate through the manual JIT path and require the old token to fail after canary selection.

Account read scopes expose unrelated tunnel, Access, ruleset, and filter-list metadata.

The job compares the complete edge contract and both cache-phase documents, then runs cache-bypass HTTP probes.

It appends one immutable cloudflare-drift/ audit object and cannot apply or repair drift.

Create one node-tuple observer Entra application and service principal.

Run firecall-node-tuple-monitor.yml every five minutes and permit a separate manual recovery dispatch.

The two event-specific jobs call the commit-pinned firecall-node-tuple-monitor-schedule-privileged.yml and firecall-node-tuple-monitor-manual-privileged.yml workflows through separate FICs.

Grant Azure Kubernetes Service Cluster User only on the exact dev cluster.

After pool creation, grant exact VMSS read and Azure Monitor metric-read actions at only the resolved Firecall VMSS resource ID.

Pool rotation changes that ID through final trust and invalidates the old assignment before creation resumes.

Kubernetes RBAC permits get and list for Nodes cluster-wide and DaemonSets and Pods only in kube-system.

Those operations expose complete node addresses, Pod specifications, commands, environment values, and system image references.

Give it no Secret, log, event, exec, attach, port-forward, workload-write, Binding, or admission authority.

It binds each Firecall node name, UID, pool identity, subnet identity, and exact InternalIP inside the reviewed CIDR.

It also binds the exact workload=firecall label and workload=firecall:NoSchedule taint.

It binds allocatable CPU, memory, Pod, and ephemeral-storage facts used by the capacity proof.

It compares the protected tuple-manifest digest with every reviewed DaemonSet UID, Pod UID, template, request, and resolved image digest.

A missing or changed label, taint, address, pool, subnet, capacity fact, or system tuple publishes blocked and alerts.

It keyless-signs one immutable result under firecall-node-tuple-evidence/checks/<epoch>/<digest>.json.

It leases and conditionally advances only firecall-node-tuple-status/current.json.

The current record binds generation, prior digest, source commit, tuple-manifest digest, evidence path, verdict, evaluation time, and valid_until.

Set valid_until no later than ten minutes after the Azure Storage response time.

A mismatch publishes blocked; an execution failure leaves the prior record to become stale.

The broker and control runner verify the signed current result before a new request or reservation.

A missing, stale, malformed, unsigned, wrong-source, or blocked tuple result denies new sessions.

It never blocks close, sweep, break glass, or zero-resource proof.

The identity can forge a tuple verdict or suppress its own alert, but it cannot change the cluster.

Disable a compromised observer service principal, remove its FICs and exact roles, and wait for issued tokens to expire.

24.2 Normal control identity

Create one dedicated normal control OIDC identity.

Its authority is:

  • A custom Firecall Session Operator role on rg-firecall-sessions-dev-eastus.
  • State read, write, list, delete, and lease authority on firecall-sessions.
  • Request-receipt read and delete authority on firecall-requests, with no content write.
  • Status authority on firecall-status.
  • Prefix-conditioned read and create-once authority on firecall-tombstones, without list or delete.
  • Read, lease, and conditional write authority on firecall-capacity, without Blob delete.
  • Exact read-only authority on firecall-capacity-bootstrap/bootstrap.json, without list.
  • Rate-counter authority on firecall-ratelimits is not granted to the control runner.
  • Probe-request write and delete, plus probe-result read and delete authority.
  • Read-only authority on public proxy certificates in firecall-trust.
  • Read-only authority on exact profile-pinned root, lock, and bundle objects in firecall-root-trust.
  • Revocation read and delete authority on firecall-revocations.
  • Invitation read and delete authority on firecall-invitations, with no create authority.
  • Immutable audit append authority on firecall-audit.
  • Read access for the exact AKS network facts needed by preflight.
  • Subscription-scope Microsoft.Insights/eventtypes/values/read for Azure Activity Log reconciliation.
  • Microsoft.Insights/metrics/read on only rg-firecall-sessions-dev-eastus for session qualification.
  • Read access for the exact Compute Gallery image version and its deterministic seal lock.
  • policyAssignments/read at only the exact session assignment resource ID.
  • policyDefinitions/read at only the exact session definition resource ID.
  • A custom network-attachment role on the exact session subnet.
  • Conditional role-assignment authority at separately assigned dev AKS scopes.

Give it Microsoft.Compute/locations/usages/read at subscription scope.

Give it Microsoft.Network/locations/usages/read at subscription scope.

It uses those actions only for East US BS-family, Total Regional vCPU, and Standard public-address session checks.

It receives no quota-write or sibling Compute-write action at that scope.

Azure authorizes the Activity Logs REST endpoint at subscription scope.

The resource-group and time filters are application controls, not Azure RBAC boundaries.

The reviewed runner wrapper always applies the exact session resource-group ID and recorded operation interval.

A compromised control identity can read unrelated subscription Activity Log metadata within Azure retention.

Give it no diagnostic-setting, log-profile, alert, export, write, or delete action.

Session metric reads expose CPU and credit time series for only Firecall session resources.

Owner acceptance of this read-only subscription metadata exposure is an activation gate.

The custom session role permits only required operations for:

  • Virtual machines.
  • Azure VM Run Command on session machines.
  • Managed disks.
  • Network interfaces.
  • Standard static session public IPs with required tags.
  • Session NSGs and their associations.
  • Session user-assigned managed identities.

At only rg-firecall-sessions-dev-eastus, also permit these linked-resource actions:

  • Microsoft.Network/networkSecurityGroups/join/action.
  • Microsoft.Network/publicIPAddresses/join/action.
  • Microsoft.Network/networkInterfaces/join/action.
  • Microsoft.ManagedIdentity/userAssignedIdentities/*/assign/action.

Grant no linked action at the subscription, another resource group, a parent VNet, or an unrelated resource type.

Before first creation, read effective permissions at the exact resource group and require all four actions.

Repeat the permission read at a sibling resource group, subscription, VNet, and unrelated resource and require absence.

Policy validation rejects a wrong resource type, extra attachment, and every cross-session attachment before mutation.

The first Group F owner canary supplies the positive live linked-resource proof through the normal session path.

The custom role permits parent networkSecurityGroups/read, write, and delete only.

It omits networkSecurityGroups/securityRules/write and networkSecurityGroups/securityRules/delete.

Terraform sends every rule through the complete parent NSG PUT.

Azure Policy evaluates that parent PUT before Azure accepts a rule change.

This avoids the Azure Policy limitation that does not evaluate DELETE operations.

Azure VM Run Command executes as root and cannot restrict the command text through Azure RBAC.

Normal Firecall code uses it only to read the generated public server certificate and fixed readiness facts.

A compromised control runner can instead execute arbitrary root commands on any active session VM.

It can then use that box's managed identity and resolved namespace Reader and Writer authority until cleanup or token expiry.

The role scope limits this authority to the Firecall session resource group.

This is an explicit owner-approved residual boundary, not a least-privilege claim.

The separate network-attachment role permits only:

  • Read and join on the exact standing session subnet.
  • Read the exact Firecall and AKS VNet peering facts.

The session resource-group role alone cannot cross those resource scopes.

Every session resource has these tags:

  • owner with the requester email.
  • project=firecall.
  • expiry with the exact UTC expiry.
  • session_id.
  • profile.

Names and tags support discovery and cleanup.

They do not replace state or authorization checks.

The custom role cannot modify:

  • The parent resource group.
  • The standing Firecall VNet, subnet, or peerings.
  • Virtual networks or peering.
  • Storage accounts or containers.
  • Key Vault.
  • AKS configuration.
  • DNS.

Assign role-management authority separately at these scopes:

  • The exact dev cluster scope, conditioned to Cluster User only.
  • Each reviewed namespace scope, conditioned to RBAC Reader and RBAC Writer only.

The runner assigns exactly one namespace role for each resolved namespace. It assigns Reader by default. It assigns Writer only when effective namespace access is write.

Give every role assignment a deterministic UUIDv5 name.

Derive it from a checked-in Firecall namespace UUID, session identifier, exact scope, and role-definition ID.

Status records each full role-assignment resource ID before VM readiness.

Break glass can derive the same IDs without state or principal lookup.

Delete role assignments before deleting the managed identity during normal cleanup.

Azure role conditions cannot prove that the target principal belongs to the shared v2 parent resource group.

A compromised runner can assign these approved roles to an unrelated principal.

It can remove that assignment before the 15-minute live assignment poll.

The unrelated principal can retain an already issued AKS token until that token's signed exp value.

Activation records the live maximum token lifetime and blocks when it exceeds 60 minutes.

Assignment removal does not revoke an already issued token.

The assignment auditor also reads subscription Activity Log every five minutes with a 24-hour overlapping watermark.

It appends each observed create and delete operation to immutable Firecall audit.

The control runner cannot delete or shorten Azure Activity Log retention.

Activity Log reduces short-lived assignment evasion but does not revoke a token or prevent first use.

The design considered pre-created shared slot identities.

That option creates standing namespace authority and weakens identity isolation between sessions.

The design also considered a privileged role-assignment broker.

That option moves the same assignment power to a new service and adds a new availability dependency.

Azure has no condition that binds these dynamic assignments to a managed identity inside one resource group.

The initial release therefore keeps dynamic identities and records this authority as an owner-approved residual.

Before activation, use one disposable identity with no credential or federated credential to demonstrate this residual safely.

Prove the conditioned assignment accepts only Cluster User at the exact cluster. Prove it accepts only Reader or Writer at each exact namespace scope.

Prove every other role, parent scope, sibling namespace, subscription scope, and role-definition identifier fails.

Remove the disposable assignment and identity, then require Activity Log and immutable audit proof of both operations.

Use one new managed identity for each session.

Never reuse session identities because old Azure tokens can remain valid briefly.

The normal control identity receives no subscription Contributor role.

It receives no Key Vault access.

It receives no production cluster access.

24.2.1 Independent assignment inventory

Run assignment-audit.yml every 15 minutes and by manual dispatch.

Its event-specific reusable workflows are privileged-assignment-audit-schedule.yml and privileged-assignment-audit-manual.yml on protected control main.

Use a separate assignment-auditor OIDC identity.

Grant only Microsoft.Authorization/roleAssignments/read at the exact dev cluster and reviewed namespace scopes.

Give it read-only access to the exact capacity ledger and lifecycle-status prefix.

For each active slot, derive the expected box principal, role, scope, and deterministic assignment GUID.

List assignments at each exact scope and compare the complete closed set.

Alert on an unexpected principal, role, scope, GUID, missing assignment, or assignment without an active session.

The workflow cannot create, update, or delete an assignment, status object, capacity value, identity, or Azure resource.

This check detects trust expansion but cannot prevent a compromised normal control runner from creating it.

24.3 Reduction-only break glass

Create one separate break-glass OIDC identity.

It can:

  • Read session state and lifecycle facts when storage works.
  • Read exact break-glass approval-consumption objects.
  • Create one immutable approval-consumption object with a conditional write.
  • List Firecall resources in the shared v2 parent resource group.
  • Deallocate and delete deterministic session resources.
  • Delete deterministic Cluster User and namespace Reader or Writer assignments.
  • Append a break-glass audit event.

It cannot:

  • Create or update a VM, disk, NIC, public IP, NSG, or managed identity.
  • Use Azure VM Run Command.
  • Join a subnet or change VNet peering.
  • Create or update any role assignment.
  • Write Terraform state, the capacity ledger, profiles, or request receipts.
  • List, overwrite, or delete break-glass approval-consumption objects.
  • Publish an image.

Break glass uses a direct reduction script, not Terraform.

When status is unavailable, it checks deterministic IDs at every reviewed profile scope.

Normal control later records cleanup proof and releases capacity.

24.3.1 Classic reduction identity

Create one replacement Classic reduction Entra application and service principal before cutover.

Its schedule and manual FICs accept only firecall-classic-reduction-schedule-privileged.yml and firecall-classic-reduction-manual-privileged.yml at one approved 40-character commit each.

Each pinned workflow accepts only its exact event, current protected infrastructure source, and first run attempt.

Give it exact-key read, write, and lease authority on only these objects:

  • tfstate/azure/firecall/dev.tfstate.
  • tfstate/azure/firecall/dev-slot2.tfstate.
  • tfstate/azure/firecall/dev-slot3.tfstate.

Give it read and delete actions only for deterministic Classic VM, NIC, disk, public address, NSG, and managed-identity resources. Each resource must belong to rg-firecall-dev-eastus or an exact state-derived group matching rg-firecall-dev-<session_id>-eastus.

Give it role-assignment read and delete, but no write, at only the fixed Classic cluster and namespace scopes.

Give it no VM create, update, Run Command, network join, role-assignment write, Key Vault, Gallery, registry, Firecall-v2 state, or policy authority.

The workflow accepts only close, sweep, or zero-proof against the merged open-rejection manifest.

It refuses a resource not present in the deterministic Classic inventory.

The historical Classic identity loses its FIC before v2 capacity becomes three.

Prove this identity can reduce all current Classic states and cannot authenticate from a historical open workflow.

24.4 Image-builder identity

Use a separate image-builder OIDC identity.

This one identity builds both runtime containers and the Packer VM image.

There is no separate container-builder principal.

Give it build-resource authority only in rg-firecall-image-build-dev-eastus.

Its custom role permits deployment-wrapper, VM, NIC, disk, public-address, and exact fixed-NSG operations.

It grants these deployment actions only in the image-build resource group:

Microsoft.Resources/deployments/read
Microsoft.Resources/deployments/write
Microsoft.Resources/deployments/delete
Microsoft.Resources/deployments/validate/action
Microsoft.Resources/deployments/operations/read
Microsoft.Resources/deployments/operationStatuses/read
Microsoft.Resources/subscriptions/resourceGroups/read

That permission cannot restrict the wrapper name.

The fixed Azure Policy constrains every nested VM, NIC, NSG, disk, and public address.

It can read and join only snet-firecall-image-build.

It can read and replace only nsg-firecall-image-build through a parent NSG PUT.

It cannot delete that NSG or write a child security-rule resource.

It cannot write the standing build VNet, subnet, route table, peering, DNS, or another network.

It cannot use a VM SKU other than Standard_D2ds_v5.

Grant Microsoft.Compute/locations/usages/read at subscription scope to this identity only.

Grant Microsoft.Network/locations/usages/read at subscription scope to this identity only.

Those actions read DDSv5-family, Total Regional vCPU, and Standard public-address quota.

It cannot change quota or Compute resources.

Policy fixes one VM, NIC, disk, and standing NSG name but cannot enforce one prefixed public address.

A compromised identity can create multiple policy-compliant public addresses and deployment wrappers.

Subscription quota and budget alerts limit or detect that residual spend; they do not prevent it.

Give it gallery read and image-version publish authority only in the Firecall gallery.

Give it AcrPush on the shared platform registry for Firecall containers.

The current registry cannot scope AcrPush to selected repository paths.

Digest-only GitOps pins prevent a pushed image from deploying without a reviewed pull request.

Delete temporary build resources after each build.

The protected workflow checks DDSv5-family and Total Regional vCPU quota before creation.

It proves zero temporary resources at completion.

Give it no AKS role-assignment or session-state authority.

Give it no Azure resource-lock read, write, or delete action.

Give it policyAssignments/read at only the exact builder assignment resource ID.

Give it policyDefinitions/read at only the exact builder definition resource ID.

Do not rely on the image-build resource-group scope for the subscription-level definition read.

24.4.1 Fixed image-build Policy

Use a separate Azure Policy definition and assignment for rg-firecall-image-build-dev-eastus.

Store both under terraform/stacks/azure/firecall-image-build-policy/prod/.

Use backend key azure/firecall-image-build-policy/prod.tfstate in firecall-policy-definition-state.

Only the authenticated owner bootstrap can read, plan, or write that state.

Do not create a routine OIDC identity or flexible credential for this root.

The root owns only its fixed definition and exact assignment, both named firecall-image-build-deny.

Their exact scopes are:

/subscriptions/f1ade722-e418-4647-8094-3f73664dd6a5/providers/Microsoft.Authorization/policyDefinitions/firecall-image-build-deny
/subscriptions/f1ade722-e418-4647-8094-3f73664dd6a5/resourceGroups/rg-firecall-image-build-dev-eastus/providers/Microsoft.Authorization/policyAssignments/firecall-image-build-deny

Set Policy mode to All and assignment enforcement mode to Default.

Set notScopes, overrides, resource selectors, exemptions, and assignment identity to empty.

Use fixed checked-in parameter values without a routine parameter input.

The Policy allows the standing build VNet, subnet, and fixed subnet NSG only with their exact names and properties.

It allows only the exact temporary VM, NIC, and disk names.

It allows Standard public-address names with prefix fcimgip and exact length 12.

Those patterns match the pinned official plugin's five-character random suffix.

It validates East US, Standard_D2ds_v5, the 32-GiB Standard SSD, and the reviewed versioned Ubuntu source image identifier.

It validates the exact subnet-to-fixed-NSG attachment, NIC subnet, address SKU, static allocation, and supported evidence tags.

Azure Packer plugin v2.6.3 creates the OS disk implicitly through the VM resource.

Its ARM template has no separate Microsoft.Compute/disks resource and cannot tag that implicit OS disk.

Do not require evidence tags on that disk.

Validate its fixed name, Standard SSD SKU, 32-GiB size, and ownership by the exact builder VM.

Keep its exact name in success and failure cleanup inventory.

It validates every fixed-NSG rule name, priority, direction, protocol, address prefix, and port.

The inbound SSH source must be either one syntactically valid public IPv4 /32 or the fixed idle documentation /32.

The workflow supplies that one /32 as the only per-run deployment value.

The Policy requires a valid IPv4 /32, but it cannot prove that address belongs to the current runner.

The protected workflow proves its address through an outbound callback before the parent NSG PUT.

It compares the complete live NSG with that callback address immediately after the PUT and again before Packer starts.

Azure Policy cannot bind the dynamic /32 to that callback evidence.

A compromised image-builder job can select another public /32 while it holds the lease.

The one-build lease, signed callback evidence, post-PUT comparison, and idle reconciliation detect but do not prevent that misuse.

The builder custom role omits child securityRules/write and securityRules/delete actions.

The protected workflow submits the complete fixed NSG through a parent PUT, which Policy evaluates.

Azure Packer plugin v2.6.3 must not submit or attach a temporary NSG for this existing-VNet build.

The Policy denies another VM, NIC, disk, or NSG name, another SKU, another subnet, a broad SSH source, and an extra rule.

It denies a public address outside its exact prefix and length contract.

It cannot deny multiple distinct public addresses that each satisfy the allowed random-name pattern.

It denies every unapproved resource type in the image-build resource group.

Azure Policy cannot evaluate the ARM deployment wrapper or its template digest.

The owner bootstrap uses this order:

  1. Create the empty image-build resource group through the standing control root.
  2. Create the standing build VNet, subnet, fixed NSG, attachment, and idle rules through the standing control root.
  3. Apply the fixed builder Policy definition and exact assignment.
  4. Reread the assignment, definition, parameters, enforcement mode, and exclusions.
  5. Prove a broad SSH source, changed outbound rule, extra NSG, and child-rule write fail.
  6. Run one denied wrong-SKU deployment and one permitted disposable build.
  7. Enable the protected image workflow only after all proofs pass.

The image workflow reads the exact live assignment and definition before every build.

It compares their canonical digests with protected source and signed bootstrap evidence.

A missing, changed, disabled, excluded, or noncompliant assignment blocks the build.

Changing or removing this Policy requires a new owner-run reviewed bootstrap plan.

24.5 Image-sealer identity

Use a separate image-sealer Entra application and service principal.

It can:

  • Read Firecall Gallery image definitions and versions.
  • Read Azure resource locks at that exact Gallery scope.
  • Pull signed evidence from the shared registry through AcrPull.
  • Create or update the deterministic seal lock.
  • Append an image-seal audit event.

It cannot:

  • Create, update, or delete a Gallery image definition or version.
  • Delete a resource lock.
  • Push an ACR artifact.
  • Read the image-pin GitHub App key.
  • Manage a session, AKS role, network, Key Vault, or storage policy.

Use a deterministic lock named from the exact image-version resource ID.

Set its level to ReadOnly.

Store the signed evidence digest and sealed resource ETag in its notes and seal statement.

The sealer can change a lock and can cause Gallery denial of service.

It cannot change image content without a separate image-builder compromise.

24.6 Image-pin identity

Use a dedicated Entra application and service principal for image-pin proposals.

Limit its Key Vault authority to the exact image-pin GitHub App private-key secret.

Give it AcrPull on the shared registry.

Give it exact Firecall Gallery-version and resource-lock read authority.

Give it policyAssignments/read at only the exact session assignment resource ID.

Give it policyDefinitions/read at only the exact session definition resource ID.

Give it no ACR push, Gallery write, lock write, build-resource, session, AKS, network, storage, or key-signing authority.

The current registry cannot limit AcrPull to firecall/evidence.

Therefore, each protected sealer and pin identity can read every artifact in that shared registry.

This read-only residual needs owner approval before activation.

The privileged image-pin workflow verifies build and seal bundles before it requests that secret.

It mints a short-lived GitHub App installation token in memory.

It can propose reviewed digest and Gallery-version pins but cannot merge them.

24.6.1 Security-rescan identity

Use a separate Entra application and service principal for recurring security scans.

Its two flexible credentials accept only firecall-rescan-schedule-privileged.yml or firecall-rescan-manual-privileged.yml on infrastructure main.

Each credential fixes the supported subject, repository IDs, and fully qualified reusable-workflow commit.

Each reusable workflow fixes its event and first-attempt guard.

Give this identity AcrPull on the shared registry.

The registry cannot limit that role to Firecall repositories.

Give it read-only access to Firecall Gallery versions and their exact seal locks.

Give it only the scanner-data read, security-status, security-evidence, and security-rescan audit Blob operations in section 15.1.

It receives no ACR push, Gallery write, lock write, Key Vault, Kubernetes, network, policy, session, role, or Terraform-state authority.

Use contents: read and id-token: write for its exact privileged job.

The job obtains one Azure token and one keyless signing certificate from separate OIDC requests.

Accept only these rescan certificate identities:

https://github.com/OpenMined/infrastructure/.github/workflows/firecall-rescan-schedule-privileged.yml@<approved-40-character-commit>
https://github.com/OpenMined/infrastructure/.github/workflows/firecall-rescan-manual-privileged.yml@<approved-40-character-commit>

Require the exact infrastructure repository, numeric identifiers, protected main ref, source commit, and caller workflow.

The rescan identity can forge a passing verdict because it owns the current verdict and its signing identity.

Owner approval of this trust transfer is an activation gate.

24.6.2 Scanner-database-publisher identity

Use a separate Entra application and service principal for the Trivy and ClamAV database publisher.

Its schedule and manual credentials accept only firecall-scanner-db-update-schedule-privileged.yml and firecall-scanner-db-update-manual-privileged.yml.

Give it direct read and create-only authority under separate firecall-scanner-data/trivy-db/v2/* and clamav/* assignments.

Give it read, conditional write, and lease authority on only the two exact Trivy and ClamAV current pointers.

It receives no list, scanner-data overwrite, delete, security-status, rescan-evidence, audit, ACR, Gallery, Key Vault, Kubernetes, network, or Terraform authority.

Its workflow permissions are only contents: read and id-token: write.

It creates the database object, signed statement, and Sigstore bundle with If-None-Match: *.

It rereads every exact path and proves locked immutability before success.

The scheduled and manual credentials use separate supported-claim FICs.

Each reusable workflow requires its exact event and first attempt.

A pull-request, wrong workflow, wrong event, wrong SHA, rerun, or sibling Blob path cannot authenticate or write.

24.6.3 Tool-cache-publisher identity

Use one separate Entra application and service principal for public tool acquisition.

Its push and manual credentials accept only firecall-tool-acquire-push-privileged.yml and firecall-tool-acquire-manual-privileged.yml on infrastructure main.

Each FIC fixes the supported subject, repository IDs, and fully qualified workflow commit.

Each reusable workflow fixes its event and first-attempt guard.

Give it only the firecall-tool-cache read and create operations in section 15.1.

It receives no list, write, delete, policy, Key Vault, ACR, Gallery, Kubernetes, network, session, role, or Terraform-state authority.

The manual mode can repair one missing cache object from the unchanged merged lock.

It cannot accept a caller URL, version, digest, signer, path, or lock override.

Revoke it by removing both FICs and its three cache assignments, then wait for its issued storage token to expire.

The trust observer verifies its application, service principal, FICs, role conditions, and absence of other credentials every six hours.

24.6.4 Retention-reconciler identity

Use one separate Entra application and service principal for reference-aware retention.

Its schedule and recovery credentials accept only firecall-retention-schedule-privileged.yml and firecall-retention-recovery-privileged.yml on infrastructure main.

Each FIC fixes the supported subject, repository IDs, and fully qualified workflow commit.

Each reusable workflow fixes its event and first-attempt guard.

Give it only the exact read, list, delete, and retention-audit operations in section 15.1.

The manual recovery mode can retry one recorded incomplete batch.

It accepts only an immutable retention operation ID and no caller-supplied container, prefix, path, version, age, or reference override.

It receives no current-pointer write, permanent delete, policy, legal-hold, state, Key Vault, ACR, Gallery, Kubernetes, network, session-resource, or role authority.

Soft-delete restoration remains an owner-approved operator action outside this identity.

Revoke it by removing both FICs and every exact storage assignment, then wait for its issued token to expire.

The trust observer verifies its application, service principal, FICs, role conditions, and absence of other credentials every six hours.

Before the first schedule, run positive canaries on a disposable expired unreferenced object.

Run negative canaries against a current pointer, referenced object, young object, WORM object, sibling prefix, state, marker, and tombstone.

24.7 Workflow rules

The normal control workflows accept only signed envelopes.

The scheduled sweep can close expired sessions without an envelope.

The break-glass workflow is close and cleanup only.

Break-glass can never open a session or expand a profile.

24.8 Standing policy and apply path

Assign Azure Policy to rg-firecall-sessions-dev-eastus.

Name both its definition and assignment firecall-session-deny.

Their exact scopes are:

/subscriptions/f1ade722-e418-4647-8094-3f73664dd6a5/providers/Microsoft.Authorization/policyDefinitions/firecall-session-deny
/subscriptions/f1ade722-e418-4647-8094-3f73664dd6a5/resourceGroups/rg-firecall-sessions-dev-eastus/providers/Microsoft.Authorization/policyAssignments/firecall-session-deny

The policy permits only:

  • The approved East US location.
  • Standard_B2s session VMs named vm-fc-<session_id>.
  • Microsoft.Compute/disks named disk-fc-<session_id> with the approved SKU and maximum size.
  • Microsoft.ManagedIdentity/userAssignedIdentities named id-fc-<session_id> with required session tags.
  • The approved VM OS-disk SKU and maximum size.
  • A reviewed allowlist of exact sealed Firecall Gallery version resource IDs.
  • NICs named nic-fc-<session_id> on the exact session subnet.
  • Standard static IPv4 public addresses named pip-fc-<session_id> with required session tags.
  • One exact deterministic session NSG on every session NIC.
  • The eleven mandatory NSG rules from section 21.3.
  • The optional SSH rule only for an approved exact IPv4 /32.
  • Required owner, project, expiry, session_id, and profile tags on every permitted resource except the implicit VM OS disk.

The implicit VM OS disk is the only exception to required session tags.

The AzureRM VM resource has no separate OS-disk tag field, and Azure does not copy VM tags to that disk.

For that disk, require the exact deterministic name, approved SKU, 32-GiB size, session resource group, and exact owning VM managedBy resource ID.

Reject an unattached disk or a disk owned by another VM.

The resource-group budget includes this untagged disk cost.

Deny unapproved resource types in that resource group.

Use the deterministic NSG name nsg-fc-<session_id>.

Require each VM to have exactly one network-interface reference.

That reference must equal nic-fc-<session_id> in the fixed session resource group.

Require each VM identity type to be exactly UserAssigned with one userAssignedIdentities entry.

That identity resource ID must equal id-fc-<session_id> from the same VM session tag.

Require each NIC to have exactly one IP configuration, one subnet reference, and one public-address reference.

The public-address reference must equal pip-fc-<session_id> in the fixed session resource group.

Require each NIC networkSecurityGroup.id to equal the deterministic nsg-fc-<session_id> resource ID.

Derive the ID from the fixed resource-group ID and the NIC session_id tag.

An absent, empty, differently named, or differently tagged NSG attachment is denied.

An absent, detached, additional, cross-session, cross-resource-group, or changed VM, NIC, public-address, NSG, or identity relationship is denied.

Before Policy apply, read and pin every Azure Policy alias used for VM identities, VM NICs, NIC IP configurations, subnets, public addresses, and NSGs.

A missing, changed, wrong-resource-type, or duplicate required alias blocks the saved Policy plan.

Store ssh_mode and ssh_source_cidr on the NSG and NIC.

Use ssh_source_cidr=none when SSH is disabled.

The policy initiative validates every rule name, priority, direction, access, protocol, prefix, and port.

It also validates the custom-rule count as eleven or twelve according to ssh_mode.

Apply the same checks to inline NSG rules and child securityRules resources.

Deny an additional rule, a missing rule, an open 8443, an unpinned 22, or a changed deny rule.

Do not rely on Terraform source validation for this contract.

Permit at most two exact image IDs: the active version and one reviewed rollback version.

Do not accept an image definition, version-name pattern, prefix, tag, or latest alias.

Store the fixed policy definition under terraform/stacks/azure/firecall-policy-definition/prod/.

Store only its exact assignment and image-ID parameters under terraform/stacks/azure/firecall-policy/dev/.

Store its image allowlist in protected infrastructure source.

The protected image-pin process uses this order:

  1. Verify the new image evidence and exact ReadOnly seal.
  2. Open an infrastructure pull request for the exact policy allowlist.
  3. Merge and apply that policy pull request after review.
  4. Read the live assignment and prove the exact new ID is present.
  5. Open the control-repository profile-pin pull request.
  6. Enable the profile only after its reviewed pin merges.

The image-pin App cannot merge either pull request or apply Azure Policy.

The policy stack refuses an unsealed ID, more than two IDs, or a mutable alias.

The control profile validator requires its exact image ID in the live policy allowlist.

The normal runner repeats that check before it reserves capacity.

Remove an old allowlist ID only after no live session uses it.

Azure Policy cannot enforce the reservation count.

A compromised control runner can create multiple allowed VMs until subscription quota stops it.

Budget alerts are detection, not a hard spend boundary.

Declare standing Firecall resources in terraform/stacks/azure/firecall-control/dev/.

Declare the fixed definition in terraform/stacks/azure/firecall-policy-definition/prod/.

Declare its exact Firecall-group assignment in terraform/stacks/azure/firecall-policy/dev/.

The dedicated firecall-trust/prod stack owns Firecall identities, credentials, custom roles, and assignments.

The current infrastructure apply identity cannot write Azure Policy.

Keep that limitation.

Create a dedicated Firecall policy-apply Entra application and service principal.

Its flexible GitHub credential accepts only firecall-policy-privileged.yml on infrastructure main.

The owner-only definition root creates firecall-session-deny and applies prevent_destroy.

The owner performs the first assignment apply through the final firecall-policy/dev state.

That first apply creates the exact assignment before routine assignment-scoped RBAC exists.

The routine policy-apply identity cannot write a policy definition.

Give it one Firecall Policy Assignment Manager role at only the exact session assignment resource ID.

That role permits exact policy-assignment read and write, with no delete action.

Give it a separate policyDefinitions/read assignment at only the exact session definition resource ID.

Give policy-compliance and resource-group read through separate read-only assignments at the session group.

This role permits no policy delete, exemption, set definition, remediation, deployment, support, resource write, or role management.

The policy-apply identity receives no Contributor, Key Vault, storage-data, AKS, Gallery, network, session, or role-management authority.

Its workflow can read and write only the Firecall policy-assignment state.

It can change only parameters on the exact named Firecall assignment because its write scope is that assignment resource.

It cannot read or write another policy definition or assignment.

Apply in this order:

  1. Create the standing Firecall resource group with session creation disabled.
  2. Apply the owner-only fixed definition root.
  3. Give the owner time-limited access to the policy-assignment state key.
  4. Apply the exact assignment once through its final protected state.
  5. Apply the routine assignment-scoped authority through the trust-root workflow.
  6. Remove the owner's temporary state access.
  7. Read the live definition and assignment through an independent check.
  8. Keep session creation disabled until the policy denial tests pass.

The dedicated firecall-trust/prod stack remains the only protected apply path for Firecall identity authority.

A live apply changes trust, secrets, or spend and requires explicit owner approval.

24.9 Cloudflare apply authority

Manage the Firecall edge through terraform/stacks/cloudflare/firecall/dev/.

Do not use dashboard changes for Terraform-owned Firecall resources.

Require FIRECALL_APEX to equal the apex of its complete Cloudflare full-setup zone.

Require FIRECALL_PREVIEW_APEX to equal the apex of its separate complete Cloudflare full-setup zone.

Cloudflare Universal SSL then covers each zone apex and exactly one hostname level below it.

That coverage includes portal.FIRECALL_APEX, *.FIRECALL_APEX, and *.FIRECALL_PREVIEW_APEX.

Use Universal SSL only for the initial release.

Do not enable Total TLS or buy an Advanced Certificate Manager certificate for this design.

The initial certificate increment is USD 0 when both required zones use their included Universal SSL service.

Before any Access policy, ingress route, or connector activation, require both Universal SSL certificates to be active.

Read and record each exact zone, certificate authority, SAN set, status, not-before time, expiry, and chain digest.

Require at least 30 complete days before expiry.

Run live TLS 1.3 SNI probes for the portal, one canonical workspace canary host, and one canonical preview canary host.

Each probe must validate the public chain and exact hostname through system trust without an insecure option.

The Cloudflare drift observer receives SSL and Certificates Read on only the two zones.

The edge mutation token receives no SSL, certificate, Total TLS, or certificate-pack edit permission.

A missing certificate, wrong SAN, incomplete chain, failed SNI, TLS downgrade, or short expiry keeps the edge closed.

If either apex cannot equal its zone apex, stop initial activation.

That alternate topology needs an owner-approved Advanced certificate, cost, permissions, renewal design, and plan amendment.

Pin the Cloudflare provider to 5.23.0 and AzureRM provider to 4.81.0.

Commit both complete linux_amd64 lock entries and reject a different selected version.

Use these provider objects:

  • cloudflare_zero_trust_tunnel_cloudflared for one remotely managed tunnel.
  • cloudflare_zero_trust_tunnel_cloudflared_token as the sensitive connector-token data source.
  • cloudflare_zero_trust_tunnel_cloudflared_config for ordered ingress and final 404 fallback.
  • cloudflare_dns_record across two zones for the portal, workspace wildcard, and preview wildcard.
  • cloudflare_zero_trust_access_identity_providers to select exactly one existing development email one-time PIN provider.
  • cloudflare_zero_trust_access_application for three separate applications and their inline application policies.
  • azurerm_key_vault_secret for the exact connector-token and audience delivery objects.

Each Access application has type=self_hosted, its exact hostname, one-hour session duration, and one exact audience.

Set allowed_idps to only the existing development email one-time PIN identity-provider ID.

Enable automatic redirect to that one identity provider.

Render each steady application with exactly one inline application-scoped policy at precedence one.

Do not use cloudflare_zero_trust_access_policy for these policies.

In provider 5.23.0, that standalone resource represents a reusable account policy.

CI renders and compares the complete inline policy order for all three applications.

That policy has decision allow and includes only email domain openmined.org.

Its only Require selector is the exact email one-time PIN login method.

Cloudflare's unmatched default denial is the only fallback.

Permit no bypass, service-token, non-identity, reusable-policy, everyone, alternate-IdP, group, country, IP, or emergency rule.

The drift observer compares the complete ordered policy set, the one Require selector, application fields, identity-provider type and ID, and implicit-deny expectation.

An extra, missing, reordered, changed, standalone, reusable, or fallback policy blocks new sessions.

The deployable closed state keeps each Access application's inline policy list empty.

Cloudflare unmatched default denial then prevents application-token issuance.

The closed state also omits the three tunnel ingress rules.

Before the safety interval, prove every host denies access and no application token can reach either origin.

After that interval, one saved edge plan adds all three steady inline policies and their fixed ingress rules.

The workflow applies no DNS, tunnel, identity-provider, cache, or unrelated Access change in that transition.

Create these exact DNS records:

Host Type Content Proxied
portal.FIRECALL_APEX CNAME <tunnel-id>.cfargotunnel.com true
*.FIRECALL_APEX CNAME <tunnel-id>.cfargotunnel.com true
*.FIRECALL_PREVIEW_APEX CNAME <tunnel-id>.cfargotunnel.com true

The first edge apply and daily drift check compare each record's zone, name, type, content, proxy flag, and tunnel ID.

Do not manage a zone Cache Rules entry-point ruleset in the Firecall edge state.

Cloudflare permits one entry-point ruleset per zone phase, and that object contains every rule in the phase.

Create the canonical zone-owner root at terraform/stacks/cloudflare/firecall-zone-policy/prod/.

That root owns the complete http_request_cache_settings entry-point ruleset document for each exact zone.

It owns every existing rule, order, ref, action, expression, parameter, version, and description in those two phase documents.

No other Terraform state or dashboard process can own either phase after adoption.

Store its state at firecall-zone-policy-state/cloudflare/firecall-zone-policy/prod.tfstate.

Set backend versioning, 30-day soft delete, exact-key roles, and no standing human access.

Add firecall-zone-policy.yml and commit-pinned firecall-zone-policy-privileged.yml.

The mutation workflow uses the firecall-live environment and workflow_dispatch only.

Create a separate zone-policy Entra application, service principal, exact FIC, and state role.

It can get only the selected zone-policy Cloudflare token slot and read, write, and lease only its state key.

The zone-policy Cloudflare token has these permissions:

  • Zone Cache Rules Edit on only the portal and preview zones.
  • Account Rulesets Edit on the OpenMined account.
  • Account Filter Lists Edit on the OpenMined account.

Cloudflare currently requires both account-wide edit permissions to manage Cache Rules through the API.

A stolen token can change unrelated account rulesets and filter lists.

Owner acceptance of that broad authority is required before adoption.

Give the token no DNS, tunnel, Access, billing, membership, API-token, firewall, purge, or zone-settings permission.

Store its primary and secondary slots through the section 24.10.1 path.

Set each zone-policy token to expire after 90 days and alert 14 days before expiry.

Rotate it through the inactive slot, full-phase read canary, saved no-change plan, and old-token denial.

Before adoption, inventory every phase ruleset, rule identifier, order, action, expression, parameter, version, and canonical digest.

Store that reviewed inventory in platform/firecall/contracts/cloudflare-cache-phase-imports.json.

The protected workflow imports both complete phase documents into their final Terraform addresses.

It requires a zero-change saved plan before it proposes the Firecall rule.

If another owner already manages either phase, integrate that state into this canonical root or stop activation.

Never create a second entry-point ruleset or use dual ownership.

The plan adds ref firecall-cache-bypass-v1 with action set_cache_settings and cache=false.

Its expression matches only portal.FIRECALL_APEX, *.FIRECALL_APEX, and *.FIRECALL_PREVIEW_APEX.

The saved plan must preserve every unrelated rule byte and order.

Apply that saved plan only after owner review through firecall-live.

Set prevent_destroy on both complete ruleset resources.

Accept only this keyless signer identity:

https://github.com/OpenMined/infrastructure/.github/workflows/firecall-zone-policy-privileged.yml@<approved-40-character-commit>

Publish each canonical statement and Sigstore bundle create-once at firecall-security-evidence/cloudflare-cache/<epoch>/<digest>.json.

Upload only its exact non-secret Blob reference as a 30-day GitHub artifact.

No workflow writes this evidence to repository source.

Name that artifact firecall-zone-policy-reference-<run-id>-1.

Its only file is canonical JSON with the infrastructure repository ID, workflow ID, run ID, run attempt, source SHA, Blob path, and Blob digest.

The public edge workflow requires four immutable dispatch inputs:

  • zone_policy_run_id.
  • zone_policy_run_attempt, which must equal 1.
  • zone_policy_source_sha.
  • zone_policy_artifact_sha256.

Its cache-evidence consumer job receives actions: read, contents: read, and id-token: write only.

The job retrieves only that run and exact artifact through the GitHub Actions API.

It requires the infrastructure repository ID, exact zone-policy workflow ID, workflow_dispatch, successful conclusion, attempt one, and matching head SHA.

It computes the downloaded artifact digest before parsing and compares all four immutable inputs.

It rejects a latest-run lookup, another run, another attempt, an expired artifact, a duplicate name, or extra artifact content.

It then reads the exact referenced Blob without a list operation and verifies its content digest, signer, schema, source, prior digest, freshness, and live state.

The edge evidence binds the consumed zone-policy run, artifact digest, and Blob digest.

The evidence binds both zone IDs, phase and ruleset IDs, versions, full pre- and post-digests, rule order, exact Firecall fields, workflow SHA, and live readback.

It also binds schema version, source commit, prior evidence digest, evaluation time, and valid_until no later than 24 hours.

Keep Blob evidence under the locked 90-day policy.

The Firecall edge workflow verifies that evidence and performs live HTTP cache-bypass probes.

Give it read-only exact-prefix access to firecall-security-evidence/cloudflare-cache/* with no list, write, lease, or delete.

It has no Cloudflare Rulesets, Filter Lists, or Cache Rules permission.

The separate read-only Cloudflare drift observer repeats both complete inventories and HTTP probes daily.

A missing, changed, unsigned, stale, conflicting, or partially owned cache proof blocks activation.

Rollback disables Firecall traffic first, then removes only the Firecall rule through a reviewed full-phase plan.

It never restores a pre-adoption state file or discards unrelated phase changes.

The first edge apply cannot scope a Key Vault role to a secret object that does not exist.

Use this one-time adoption order for the connector token and three audience secrets:

  1. An authenticated owner creates each exact secret with a random nonfunctional placeholder value.
  2. Do not create the corresponding ExternalSecret objects yet.
  3. The trust root assigns the edge identity exact-object get and set DataActions.
  4. Give the owner temporary access to only cloudflare/firecall/dev.tfstate and its keyenv: prefix.
  5. Import all four secret objects into their final edge-stack Terraform addresses.
  6. Store their IDs and initial version IDs in bootstrap-imports.json.
  7. Run a zero-change refresh for those adopted objects.
  8. Remove the owner's state assignment and wait for its storage token to expire.
  9. Let the protected edge identity update each object with its real value.
  10. Prove no current version contains a placeholder before secret delivery starts.

Set prevent_destroy on all four adopted secret resources.

The bootstrap script never becomes a second owner after import.

The tunnel resource does not export its connector token.

The provider data source reads that token after tunnel creation or refresh.

Terraform marks it sensitive, but the saved plan and dedicated edge state still contain its value.

The stack owns only Firecall records, applications, policies, and the dedicated tunnel.

It does not own the Cloudflare zone, OpenMined identity provider, organization settings, or unrelated routes.

Store its backend state in a dedicated firecall-edge-state Blob container.

Use backend key cloudflare/firecall/dev.tfstate.

Restrict routine state read and write to the protected edge apply identity.

Give no human or operator standing Blob access.

Development creates no PIM group or standing human state assignment.

When the protected edge workflow cannot run, use only the exact bounded owner operation from ADR-0012. That operation binds one protected source commit, one saved plan, and only cloudflare/firecall/dev.tfstate plus its keyenv: prefix.

Remove its temporary assignment and wait for its issued Azure token to expire. This recovery authority is a secrets and trust operation. It is not a development session gate.

Read access exposes the connector token and can start a rogue connector until token rotation.

Write access can corrupt edge state, while versioning and soft delete support recovery.

Use this bounded recovery path only when the protected edge workflow cannot run.

State contains the sensitive connector token.

Treat every state version as secret material.

Enable versioning, soft delete, private Azure RBAC access, and state locking.

Create one Cloudflare API token with only these scopes:

  • Tunnel read and write on the exact OpenMined account.
  • Access application and policy read and write on that account.
  • DNS read and write on the exact portal zone and separate preview zone.

Cloudflare applies the first two permissions to every tunnel and Access application in that account.

It cannot restrict them to Firecall resources.

Therefore, a stolen token can change unrelated OpenMined tunnels and Access applications.

The DNS permission remains limited to those two exact zones.

Grant no Rulesets, Filter Lists, Cache Rules, zone-settings, firewall, member, billing, API-token, or other permission category.

Set the token expiry to 90 days and alert 14 days before expiry.

Store primary and secondary Cloudflare API-token slots at two exact Key Vault secret scopes.

The token has no API-token management permission and cannot rotate or revoke itself.

Use this normal rotation:

  1. A Cloudflare account owner creates a new token and uses the section 24.10.1 setter for the inactive slot.
  2. Verify its account, two zone scopes, expiry, and exact permission map.
  3. Run permitted tunnel, Access, and DNS canaries for the exact scopes.
  4. Prove Rulesets, Filter Lists, Cache Rules, billing, member, token-management, extra-zone DNS, firewall, and zone-settings calls fail.
  5. Merge the protected non-secret slot selector.
  6. Run the edge plan, apply, and complete live comparison with the new token.
  7. The Cloudflare owner revokes the old token immediately after that success.
  8. Call the verification endpoint with the old slot and require denial.
  9. Use the separate section 24.10.1 disabler on the old Key Vault version after denial evidence exists.
  10. Keep the disabled version only under the approved secret-retention policy.

For suspected disclosure, revoke the suspected token before the replacement apply.

Accept the resulting edge-management outage while the owner creates and proves a replacement.

Never restore a revoked token to shorten that outage.

The owner must approve the account-wide tunnel and Access residual authority before activation.

Store that API token in the dedicated Firecall Key Vault.

Create a dedicated Cloudflare-apply Entra application and service principal.

Its flexible GitHub credential accepts only firecall-cloudflare-privileged.yml on infrastructure main.

Its Azure authority is limited to:

  • Read the exact Cloudflare API-token secret.
  • Read and write the dedicated edge-state container.
  • Read the exact kv-firecall-dev-eastus vault parent through Microsoft.KeyVault/vaults/read.
  • Get and set only the exact current tunnel-token secret in Key Vault.
  • Get and set only the three Access-audience secrets in Key Vault.
  • Obtain AKS Cluster User credentials for the dev cluster.

The AzureRM provider needs the vault-parent read before it can manage one secret object.

Assign that control-plane action at only the exact Key Vault resource ID.

Grant no vault write, delete, access-policy, role, network, private-endpoint, or sibling-resource action.

Vault metadata disclosure is the maximum added authority.

Its Kubernetes Role can read one ExternalSecret status and update one pre-created cloudflared-rollout-epoch ConfigMap.

It can get, update, and patch only the pre-created firecall-edge-operation Lease.

It can patch only the exact named cloudflared Deployment.

It can get and watch only that named Deployment.

Every watch uses a metadata.name field selector.

It can list read-only ReplicaSet and Pod metadata and status only in firecall-edge.

It rejects every ReplicaSet not owned by the exact Deployment UID.

A ValidatingAdmissionPolicy binds that identity to one annotation-only Deployment change.

The old and new objects must differ only at firecall.openmined.org/tunnel-token-epoch.

It cannot read a Kubernetes Secret or change another Deployment field.

It receives no session, Gallery, ACR, network, policy, or Azure role authority.

The protected workflow obtains the API token in memory and masks it before provider initialization.

Pull-request jobs run terraform validate and a credentialless mocked contract test only.

They receive no Cloudflare, Azure, or state credential.

The protected main workflow runs a saved plan and apply after the required edge approval.

Permit one explicit bootstrap_no_connector mode before any cloudflared replica starts.

That mode requires the live and protected desired replica counts to equal zero.

It creates and compares the tunnel, three DNS records, three Access applications, and three audiences.

Each application has an empty inline policy list.

The tunnel configuration contains only the final 404 fallback and no Firecall origin ingress rule.

It writes the first tunnel token and three audiences to their adopted Key Vault objects.

It skips only connector and origin-reachability proof because no connector exists.

After any connector has started, reject bootstrap_no_connector permanently.

Every routine apply and recovery then requires two current connectors and both origin checks.

It acquires the same Kubernetes firecall-edge-operation Lease before planning.

It holds that lease through apply and live verification.

The plan job writes the sensitive plan to one versioned Blob under the dedicated firecall-edge-plan container.

The apply job downloads that exact Blob version and verifies the source, lock, plan, run, Blob-name, Blob-version, and Kubernetes Lease-generation bindings before apply.

The edge identity can read and write that container. It cannot list or delete plans. Azure lifecycle soft-deletes active plan versions after two days. The shared state account retains recoverable bytes for its existing 30-day soft-delete period. The edge identity cannot recover them.

GitHub stores only the sanitized manifest for one day. It never receives the plan file because the plan can contain sensitive values.

When protected source opens the edge, both jobs require the current tunnel ExternalSecret, Ready broker and proxy EndpointSlices, and exactly two current Ready connector Pods. They read no Secret data or logs.

After apply, a zero-diff Terraform refresh rereads all managed Cloudflare and Azure resources. The job repeats the Kubernetes checks before Lease release.

In bootstrap mode, it compares hostnames, audiences, empty policy lists, session durations, DNS targets, and fallback-only ingress exactly.

Section 21.7 step 68 is the only initial transition that adds the three policies and three origin routes.

After that transition, routine applies compare origin Services, policy includes, route order, and the final fallback exactly.

The apply validates both hostnames through a pinned Public Suffix List.

It fails unless their effective registrable domains differ.

It writes the tunnel connector token and three audience values directly to their exact Key Vault objects.

It never prints those values or places them in a workflow artifact.

The separate External Secrets objects deliver the tunnel token and audience values to only their intended Pods.

Tunnel-token rotation uses the same protected edge state and apply identity.

The rotation workflow acquires the same Kubernetes Lease before it reads state.

It holds that lease through state reconciliation and connector verification.

It first requires a no-change Terraform plan and an exact live API comparison.

It then refreshes the token through the Cloudflare API.

This API call is the only unavoidable out-of-Terraform edge mutation.

Existing connectors remain active, but the old token cannot start a new connection.

In the same protected job, Terraform refreshes the connector-token data source.

It creates one saved plan that can change only the exact Key Vault tunnel-token resource and sensitive data-source state.

The workflow rejects that plan if another resource, output, provider, or dependency changes.

It applies that exact saved plan without uploading it.

The azurerm_key_vault_secret resource writes the refreshed value as a new Key Vault version.

This apply stores the same refreshed token in the dedicated Terraform state.

A second full plan must return exit code zero before any Kubernetes rollout starts.

The workflow also repeats the exact live Cloudflare comparison.

If failure occurs after API rotation, a reconcile_only recovery uses the new data-source value.

Recovery must not rotate the token a second time.

The edge-operation lease prevents a normal edge apply or another rotation during recovery.

External Secrets updates the fixed Kubernetes Secret without logging its value.

The edge workflow waits for a newer successful ExternalSecret refresh time.

It then writes only the non-secret Key Vault version GUID to cloudflared-rollout-epoch.

The live Reloader ignores Secret changes, so Firecall does not use it for this rollout.

After the ConfigMap update, the edge workflow patches the exact Deployment metadata epoch annotation.

The admission policy rejects a patch to an image, command, environment, volume, identity, label, or replica count.

The annotation is outside the Pod template and does not start a new rollout.

The ConfigMap contains no token, audience, account identifier, or credential.

The Deployment uses two replicas, maxUnavailable: 1, and maxSurge: 0.

The workflow deletes one connector Pod and waits for its same-ReplicaSet replacement.

The replacement Pod must report a healthy Cloudflare connection before the workflow deletes the second Pod.

The workflow verifies two new connectors before success.

It requires observedGeneration to equal the patched Deployment generation.

It also requires updatedReplicas=2, readyReplicas=2, and availableReplicas=2.

It identifies the unchanged ReplicaSet through its Deployment revision and Pod-template digest.

It requires exactly two Ready, non-terminating Pods owned by that ReplicaSet UID.

Each ready Pod must come from the exact Deployment revision with the expected token-epoch annotation.

The sidecar readiness must pass for that Pod.

It reads the complete tunnel connector inventory through the Cloudflare API.

Require exactly two healthy connector IDs and no stale, foreign, or disconnected connector.

Both connector run_at values must follow the rollout start and token epoch.

This jointly proves two Ready current Pods and two current Cloudflare connections.

It does not create a cryptographic one-to-one mapping from a Pod UID to a connector ID.

Record that correlation limit as an accepted connector-proof residual.

The Cloudflare connector run_at must be later than the rotation start time.

It releases the edge-operation lease only after that proof.

Emergency rotation also deletes every tunnel connection after both new Pods become ready.

The new Pods then reconnect and prove that only the refreshed token works.

A scheduled protected workflow runs a read-only Terraform plan and live API comparison each day.

Any drift fails and alerts; it does not auto-apply.

Set prevent_destroy on the tunnel, Access applications, and DNS records.

Edge deletion needs a separate reviewed removal change and explicit owner approval.

Import a conflicting existing Firecall object before the first apply.

Never create a second object to bypass a name conflict.

24.10 Key Vault and secret delivery

Create a dedicated kv-firecall-dev-eastus Key Vault in the standing Firecall resource group.

Use the Key Vault standard SKU.

Use non-exportable software-backed P-256 keys because Firecall does not require managed HSM keys.

Record the SKU, region, tenant, RBAC mode, network mode, soft-delete period, and purge-protection state in live evidence.

Set Terraform prevent_destroy on the Key Vault parent.

No Terraform lifecycle rule applies to the six operational key objects because runtime workflows add their versions.

The first owner-run standing-control apply creates lock-firecall-keyvault-delete on the vault.

Use the Azure CanNotDelete lock level.

The lock starts in the protected standing-control state and needs no import.

The standing identity can read the lock but cannot create, update, or delete a resource lock.

It also lacks Microsoft.KeyVault/vaults/delete.

Only an owner-approved retirement change can remove that lock and the parent lifecycle protection.

Use Azure RBAC and disable legacy access policies.

Enable 90-day soft delete and purge protection.

Keep its public endpoint because protected GitHub-hosted image jobs need the exact image-pin secret.

Require Entra authorization for every operation.

Store non-exportable keys separately for request and CSRF signing.

Store the four fixed non-exportable CA key objects defined in section 10.12.

Before final trust, run scripts/ops/firecall-key-bootstrap.sh as the authenticated owner.

Give that owner a time-limited vault-parent custom role for key create, read, and disable operations only.

The role grants no sign, verify, decrypt, unwrap, export, delete, purge, backup, restore, secret, certificate, policy, or role operation.

The script creates only these six exact ECDSA P-256 key names:

  • firecall-client-root-a.
  • firecall-client-root-b.
  • firecall-origin-root-a.
  • firecall-origin-root-b.
  • firecall-request-signing.
  • firecall-csrf-signing.

It requires each new initial version to be non-exportable, disabled, and limited to sign and verify operations.

It fails when another unexpected Firecall key exists or any exact object has different properties.

It writes one sanitized manifest with each versionless ID, placeholder version ID, public digest, and Key Vault ETag.

The final trust apply uses those exact versionless IDs for all child-scoped role assignments.

Remove the owner's vault-parent assignment and wait for every issued token to expire before any real key version starts.

The CA-root and credential-rotator identities create real candidate versions only after that revocation proof.

Neither identity can create a seventh key name because its key DataActions have only exact child scopes.

Use exact key-scope custom data roles with public-key read and sign actions only.

Do not grant decrypt, unwrap, export, delete, backup, restore, policy, or role-management actions.

Store fixed exportable primary and secondary slots for:

  • Broker GitHub App private keys.
  • Image-pin GitHub App private keys.
  • Root-publication GitHub App private keys.
  • Classic-observer GitHub App private keys.
  • Control-observer GitHub App private keys.
  • Settings-observer GitHub App private keys.
  • Session-token HMAC secrets.
  • Edge Cloudflare API tokens.
  • Read-only Cloudflare observer API tokens.
  • Zone-policy Cloudflare API tokens.

Store the Cloudflare tunnel token as one versioned secret.

Use one cluster-scoped ClusterSecretStore named firecall-azure-kv.

The firecall Argo Application is its sole GitOps owner.

Its namespace conditions name only firecall and firecall-edge.

Both namespaces carry the exact matching store-use label.

No third namespace can reference the store.

Apply the store after its workload identity and before either namespace receives an ExternalSecret.

Do not add either namespace to the general platform store.

The existing External Secrets controller uses its workload identity to read Firecall secret values.

This controller is an explicit trust holder for every exportable Firecall secret.

It cannot export the Key Vault private signing-key material.

For suspected External Secrets compromise, disable creation and stop its Firecall reconciliation first.

Revoke its Firecall-vault data role and workload-identity binding.

Wait for every issued Azure token to expire.

Rotate both slots for every GitHub App key, HMAC, Cloudflare API token, tunnel token, and other exportable Firecall secret.

Restart each exact consumer on a new selected version and run its complete permission canary.

Verify every non-exportable Key Vault key version, operation set, and role assignment remained unchanged.

Restore reconciliation only after old values fail, all consumers pass, and immutable incident evidence exists.

Materialize one primary or secondary Kubernetes Secret per component and credential.

Mount private values as read-only files.

Do not inject them through environment variables or a shared Secret.

Give Firecall application ServiceAccounts no Kubernetes Secret read permission.

The certificate rotator updates fixed leaf-key Secret slots through narrow Kubernetes RBAC.

It does not use External Secrets for generated leaf keys.

Rotate the inactive slot, roll the consumer, verify it, and then change the active slot.

Alert before either slot loses its safe overlap window.

Rotate the non-exportable request key in this order:

  1. Create a new disabled key version.
  2. Export only its public key and identifier.
  3. Merge that public key into the protected control repository.
  4. Enable the key and record one authenticated request_key_switched_at value.
  5. Deploy the broker with the new active identifier and stop old-key signing.
  6. Set request_previous_valid_until to exactly 22 minutes after that value.
  7. Keep the old public key through that complete interval.
  8. Reject an old-key envelope issued after request_key_switched_at.
  9. Reject every envelope after its 20-minute expiry or 15-minute GitHub queue allowance.
  10. Query receipts and GitHub runs for every old-key accepted dispatch.
  11. Require zero queued, waiting, requested, or in-progress old-key run in two reads 60 seconds apart.
  12. Require every old-key receipt to be terminal or rejected before reservation.
  13. Disable the old private key only after step 11, step 12, and the 22-minute interval pass.
  14. Keep the old public key in control until every retained receipt no longer needs signature diagnosis.

The 22-minute interval contains the 20-minute maximum envelope age and 120-second verification skew.

The 60-second broker dispatch limit and 15-minute GitHub queue limit remain inside that envelope age.

An unreadable receipt, GitHub run, authenticated time source, or key selector delays retirement.

A rotation retry resumes from the signed switch record and never creates a second switch epoch.

Never let the broker sign with a key identifier that control main cannot verify.

24.10.1 Manual external-credential JIT path

Use scripts/ops/firecall-manual-secret-slot.sh for every GitHub App private key and Cloudflare API token.

JIT means just in time.

Keep its closed manifest at platform/firecall/contracts/manual-secret-slots.json.

The manifest names every credential, primary secret, secondary secret, consumer, canary, active selector, and retention rule.

It includes broker, image-pin, root-publication, Classic-observer, control-observer, and settings-observer App keys.

It also includes edge, read-only observer, and zone-policy Cloudflare API tokens.

Create two custom Key Vault data roles.

The exact-slot setter permits only:

Microsoft.KeyVault/vaults/secrets/readMetadata/action
Microsoft.KeyVault/vaults/secrets/setSecret/action

The exact-version disabler permits only:

Microsoft.KeyVault/vaults/secrets/readMetadata/action
Microsoft.KeyVault/vaults/secrets/update/action

Assign each role at one exact existing child-secret resource ID.

Do not assign either role at the vault parent.

Neither role permits getSecret, list, delete, purge, recover, backup, restore, role, key, certificate, sign, decrypt, or export actions.

For initial bootstrap, an authenticated owner receives one time-limited assignment for only the selected primary child.

The owner repeats that assignment separately for each closed credential.

Development creates no PIM group and grants no standing human assignment. Each later rotation uses one explicit time-limited setter assignment for only the selected inactive slot. Secret and trust approval remains required under the repository operating contract.

The script accepts an exact manifest credential name, slot, and local descriptor or mode-0600 file.

It never accepts a secret value as a command-line argument or environment variable.

It disables shell tracing, Azure debug output, command echo, and response-body output before reading the value.

It sends one Key Vault set request, discards the returned secret body, and records only the new version ID and metadata.

It clears process buffers, closes the descriptor, and securely removes only its exact temporary file when one exists.

It cannot verify a value by reading it back.

The exact consumer reads the candidate slot and runs its permission canary.

The selector changes only after that canary proves identity, installation or account, repository or zone scope, expiry, and complete permission map.

For the first edge token, prove exact tunnel, Access, and two-zone DNS permissions before the edge workflow selects it.

For the first observer token, prove all declared reads and every edit denial.

For the first zone-policy token, prove the broad ruleset permissions and all unrelated permission denials.

After selection, remove the temporary setter assignment and wait for its issued Key Vault token to expire.

Rotation uses only the inactive slot and follows the same canary and selection sequence.

After the overlap, grant the separate time-limited disabler assignment for only the retired slot.

The script pins one recorded old version ID and sends only enabled=false through the Key Vault update API.

The disabler can disable any version of that one secret while active, but it cannot read or replace a value.

Remove that assignment immediately. Prove both old consumer use and old-version metadata selection fail.

Write sanitized assignment IDs, operation times, version IDs, selectors, canary digests, revocation proof, and token-expiry proof to immutable audit.

24.10.2 Deferred privileged human identity

The former development PIM group design is deferred with staging and production. Development creates no PIM group, Conditional Access policy, eligible membership, PIM status object, or PIM evidence store.

The inactive contract and scripts remain reference material only. A later ADR must replace or confirm that design before staging or production implementation.

24.11 Application credential rotation

Use firecall-credentials.yml as the public caller.

It calls only firecall-credentials-privileged.yml on protected infrastructure main.

Dormant Kubernetes runtime status on 2026-09-02

Current source implements bootstrap-prepare, bootstrap-activate, and bootstrap-prove.

Protected source must still supply each reviewed bootstrap argument before activation can start.

Normal and emergency rotation require the exact completed credential-bootstrap status.

The dormant credential Job uses normal mode, an all-zero image digest, and suspend: true.

After a Key Vault write, the rotator patches only the exact ExternalSecret force-sync annotation.

The annotation digest binds the requested slot, derived key identifier, and exact Key Vault version.

The rotator then requires a newer Ready refresh and the exact materialized Secret resource version.

It validates both canonical HMAC values and requires the candidate in only the requested slot.

This proof requires get on only the firecall-session-token-hmac Secret.

The rotator cannot read another Kubernetes Secret.

The ExternalSecret refresh interval is one minute, below the bounded rotation deadline.

The proxy reads one strict selector file and two fixed HMAC files from whole projected volumes.

It accepts a canonical expired previous deadline, which permits recovery after a long outage.

It still rejects a previous deadline more than 24 hours and two minutes in the future.

All certificate consumers mount firecall-certificate-active/data/active.json as one whole atomic projection.

They reject an inert, partial, hybrid, malformed, swapped, escaped, exposed, or unstable selector read.

Certificate bootstrap starts only from the exact inert selector and known fixed Secret states.

It issues all five first certificates after candidate-only consumer Pods start.

It publishes one complete immutable evidence object before one selector resource-version update.

An evidence failure leaves the selector inert.

The selector binds each exact mounted root-bundle byte sequence, not only its decoded certificate.

A selector race preserves prior immutable evidence and permits a later retry to converge.

Every post-bootstrap rotation still requires proof from every expected consumer Pod.

Activation creates the new immutable firecall-runtime-v3 ConfigMap instead of changing the live baseline.

Broker, proxy, and readiness templates bind its canonical data digest in one rollout annotation.

Activation also creates immutable egress version v1 and DNS-policy version v2 ConfigMaps.

Each rotator uses its dedicated DNS listener and dedicated HTTP CONNECT proxy.

The certificate rotator can reach only its Azure login, Key Vault, and Blob hosts through that proxy.

The activation foundation excludes all application ExternalSecrets.

The post-prepared composition adds those ExternalSecrets only after credential status becomes prepared.

The live node-binding validator continues to use the unchanged immutable baseline.

Both rotators use one 14-minute end-to-end deadline.

Their Leases last 14 minutes and 30 seconds.

Their suspended Jobs have a 15-minute deadline and no completion TTL.

The lack of a completion TTL prevents Argo from recreating a completed versioned Job.

The dark Application keeps its legacy selector fields and its existing ignore pointers.

Use two protected source changes for the selector migration.

The first change removes the three legacy ignore entries and selects the activation foundation.

Argo then replaces each legacy field set with one exact JSON document at wave -60.

Verify that each live selector ConfigMap contains only its one expected JSON key.

The second change adds the three complete-document ignore pointers.

Do not seed and ignore a new JSON document in the same sync.

RespectIgnoreDifferences=true would omit the ignored seed before Argo applies it.

The final activated Application ignores only each complete selector document, not separate selector fields.

No live dev overlay references any activation composition.

Every runtime replica remains zero, and every new Job image remains an all-zero digest.

Target activation contract

The dedicated GitHub OIDC identity receives only AKS Cluster User on the exact dev cluster.

Its Kubernetes Role can create one admission-bound credential-rotation Job.

It can get and watch only that Job and its exact Deployment health objects.

It cannot read logs, Secrets, exec, attach, port-forward, or patch a credential ConfigMap.

A cluster ValidatingAdmissionPolicy fixes the Job namespace, name prefix, ServiceAccount, image digest, command, arguments, volumes, and resources.

The admission contract permits explicit bootstrap-prepare, bootstrap-activate, bootstrap-prove, normal, and emergency modes.

Bootstrap mode requires creation_enabled=false, zero v2 sessions, and no existing completed bootstrap record.

bootstrap-prepare acquires the credential Lease and verifies each fixed placeholder object and selector placeholder.

It generates two independent HMAC values and writes new versions to the two fixed slots.

It creates the first real CSRF candidate version and one disabled request-signing candidate version under the existing exact key objects.

It writes no active selector yet.

It publishes only the request public key, identifiers, digests, and candidate record for a protected control pull request.

It releases the Lease before human review.

bootstrap-activate reacquires the Lease after that exact pull request merges.

It verifies the protected control commit, public-key digest, candidate record, and all Key Vault version identifiers.

It enables the request key and atomically writes complete initial HMAC, CSRF, and request selector ConfigMaps.

It atomically changes firecall-credential-bootstrap-status.data.state from uninitialized to prepared and releases the Lease.

Only then can source add the exact HMAC and application ExternalSecrets.

After broker and proxy replicas start, bootstrap-prove reacquires the Lease.

It verifies each replica reports the exact selector resource version, key identifiers, public digest, and successful canary.

It atomically changes that record from prepared to complete with generation=1.

The fixed firecall-credential-bootstrap-status ConfigMap is a non-secret Kubernetes status record.

GitOps pre-creates it with state=uninitialized, generation=0, and empty digest and time fields.

The credential rotator alone can patch its five exact data keys through one resource-version comparison.

Admission permits only uninitialized to prepared to complete and rejects regression, deletion, another generation, or field removal.

The complete record binds the protected control commit, a digest of both selector ConfigMaps, and the API-server completion time.

After complete, admission permits only an identical object.

The rotator also creates immutable signed evidence at firecall-trust-evidence/credential-bootstrap/complete.json.

Argo ignores only those five ConfigMap data pointers and continues to own every other field.

The certificate-bootstrap Job has name-based get on this ConfigMap and no update operation.

Initial certificate bootstrap accepts only state=prepared and generation=1.

It requires a nonzero reviewed control commit and a digest equal to both live selector ConfigMaps.

Normal and emergency certificate rotation accept only the exact immutable complete status.

Normal and emergency modes reject a missing or noncomplete bootstrap status record.

Bootstrap modes reject a second completed bootstrap and never print or artifact a secret value.

The Job uses the dedicated firecall-credential-rotator workload identity.

That workload identity can read and write only the fixed HMAC secret slots.

At each exact CSRF and request-key resource ID, grant only these Key Vault DataActions:

Microsoft.KeyVault/vaults/keys/read
Microsoft.KeyVault/vaults/keys/create/action
Microsoft.KeyVault/vaults/keys/update/action
Microsoft.KeyVault/vaults/keys/sign/action
Microsoft.KeyVault/vaults/keys/verify/action

keys/read permits direct version reads and version listing for only those two key objects.

keys/update/action permits version enable and disable operations on only those two key objects.

Grant both actions at each exact key resource scope, not at the vault scope.

Do not grant key delete, purge, recover, import, backup, restore, release, decrypt, unwrap, or rotation-policy authority.

The rotator can disable an active fixed key and stop broker operations.

It can also enable an older version of a fixed key.

The broker still rejects a request-key selector absent from the attested protected control commit.

It cannot read a GitHub App private key or another Firecall secret.

It can read, lease, and conditionally write only request-key-index.json in firecall-trust.

It can read and create only the fixed request-key candidate and status prefixes in firecall-trust-evidence.

It cannot list, overwrite, or delete those records.

Its Kubernetes Role can read exact ExternalSecret status objects, one materialized HMAC Secret, and candidate health metadata.

It can update only firecall-session-token-keys, firecall-broker-key-epochs, and firecall-credential-rotation.

A ValidatingAdmissionPolicy permits only the listed section 21.5 data keys and strict value schemas.

It can read only firecall-session-token-hmac to bind materialized bytes to the requested slot and key identifier.

The Job acquires the exact credential-rotation Lease before any mutation.

It generates secret bytes only in process memory and writes them directly to Key Vault.

It forces one exact ExternalSecret refresh and waits for a newer successful refresh.

It then reads the exact target Secret and verifies both slot encodings, key identifiers, and resource-version change.

It probes every proxy or broker replica through separate headless credential-health Services.

Use TCP 9544 for proxy credential health and TCP 9545 for broker credential health.

Set clusterIP: None and publishNotReadyAddresses: true on only these credential-candidate Services.

Map every EndpointSlice target UID to one expected current Deployment Pod UID before accepting a response.

Keep normal broker and proxy Services limited to ready endpoints.

Those routes accept only the credential-rotator ServiceAccount and its protected image tuple.

Certificate-health routes on 9444 and 9445 reject the credential rotator.

Credential-health routes reject the certificate rotator.

Every response must bind the candidate kid to the exact Pod UID.

Only then can it replace the reviewed selector document through one resource-version comparison.

The session selector stores active and previous slots, their kid values, the overlap deadline, and the emergency denied kid list.

The proxy reads the selector from one whole ConfigMap volume.

It reads both 32-byte HMAC values from one whole Secret volume.

Kubernetes atomic projection links permit either complete generation and reject a mixed read.

All fields use a strict schema and contain no key value.

After bootstrap, the workflow supports only normal and emergency rotation modes with one fixed credential target.

It records key identifiers, digests, replica proof, actor, workflow, and timestamps only.

GitHub App private-key creation and Key Vault entry remain manual App-owner actions.

This workflow never creates or downloads a GitHub App private key.

Request-signing rotation uses two idempotent phases.

Phase one acquires the Kubernetes Lease and the Blob lease on request-key-index.json.

If the index is absent, phase one creates the empty generation through a conditional request before lease acquisition.

The index permits only one pending request-key candidate.

It stores generation, pending kid, exact version ID, public-key digest, candidate path, creation time, and workflow run.

The rotator lists versions only under the exact request key.

It compares them with the active, previous, and pending identifiers.

Phase one creates one disabled Key Vault key version with candidate and workflow tags in the same request.

It writes one immutable, non-secret candidate record under firecall-trust-evidence/request-key-candidates/<kid>.json.

That record binds public key, key-version ID, digest, workflow, and creation time.

It then adds that candidate to the index through its Blob lease and an ETag comparison.

If a crash occurs before the index write, the next run finds the tagged version through exact-key version listing.

It reads the deterministic candidate path directly and repairs the index under both leases.

An unindexed disabled version without a valid candidate record becomes orphaned and can never activate.

The Job releases both leases and ends before human review starts.

A human opens the reviewed control pull request with only that public key and identifier.

After merge, phase two starts as a new Job and reacquires both leases.

The broker uses its read-only control App permission to attest the exact current control commit and public-key digest.

The Job rereads the candidate and revalidates key version, digest, control commit, and broker attestation.

Only then does it enable the candidate version.

It atomically patches all request-key fields in firecall-broker-key-epochs through one resource-version comparison.

The patch sets active and previous kid values, overlap deadline, control commit, and public-key digest.

Every broker replica must report the new ConfigMap resource version, kid, control commit, and digest.

If that proof fails, the Job restores the prior complete selector and disables the candidate.

It writes an immutable activated status object and releases both leases.

A candidate older than seven days cannot activate.

A scheduled cleanup acquires both leases and reads the one pending index entry.

It also lists exact request-key versions to find a disabled tagged version lost before index creation.

It disables an unused indexed version and writes an immutable stale status object.

It writes an immutable orphaned status for each recovered version without a valid candidate record.

It then clears the pending index through its Blob lease and an ETag comparison.

It never needs Blob list authority.

Candidate and status records remain non-secret audit evidence.

If the Job fails, the prior active identifier remains unchanged.

Emergency failure disables new opens and joins while close and sweep remain available.

Every signed token includes an explicit kid key identifier.

Each consumer signs with one active key and verifies only the reviewed overlap set.

Use these maximum lifecycles:

Credential Planned rotation Warning Normal overlap
Broker GitHub App private key 90 days Day 75 and day 83 24 hours
Image-pin GitHub App private key 90 days Day 75 and day 83 24 hours
Root-publication GitHub App private key 90 days Day 75 and day 83 24 hours
Classic-observer GitHub App private key 90 days Day 75 and day 83 24 hours
Control-observer GitHub App private key 90 days Day 75 and day 83 24 hours
Settings-observer GitHub App private key 90 days Day 75 and day 83 24 hours
CSRF ES256 key version 90 days Day 75 and day 83 One hour
Session-token HMAC key 30 days Day 23 and day 27 24 hours
Request-signing ES256 key version 90 days Day 75 and day 83 Keep the prior selector for exactly 22 minutes after rotation

GitHub App private keys do not expire automatically.

The 90-day value is an enforced Firecall operating limit.

For each GitHub App key:

  1. An App owner creates one new private key while the prior key remains active.
  2. The owner uses the exact-slot setter and script in section 24.10.1 for the inactive Key Vault slot.
  3. No workflow input, repository secret, artifact, terminal transcript, or issue contains the PEM.
  4. The exact consumer refreshes or reads only its inactive slot.
  5. A canary mints one installation token and verifies the exact installation, repository IDs, and permission map.
  6. The component's protected configuration selects the new key ID.
  7. The component reloads the selection and repeats its exact canary.
  8. The system observes successful token minting for 24 hours.
  9. The App owner revokes the old private key in GitHub.
  10. The operator uses the separate exact-version disabler, then retains its metadata for 30 days.

The canary never prints an App JWT, installation token, private key, or token response body.

The root-publication canary reads infrastructure and creates no branch or pull request.

The Classic-observer canary reads current main and the fixed workflow's recent runs.

The control-observer canary reads required control workflow files, states, runs, logs, and artifact metadata.

It proves workflow enable, disable, dispatch, cancel, rerun, run-delete, log-delete, artifact-delete, and content writes fail.

It also proves access to infrastructure and every uninstalled repository fails.

The settings-observer canary reads the exact environment, protection, rules, workflow, and source settings in infrastructure.

It proves every settings write, content write, workflow mutation, secret read, organization API, and uninstalled-repository request fails.

Each canary proves that an uninstalled repository and a forbidden API write fail.

The image-pin workflow reads only the active and candidate exact App-key objects.

It receives read-only AcrPull separately for signed evidence.

For the CSRF key, set a 15-minute token lifetime and allow two minutes of clock skew.

Create a new non-exportable Key Vault key version before changing the active kid.

Verify one canary signature through the Key Vault public key.

Then sign new CSRF tokens with the new version.

The broker verifies the prior version for one hour and then rejects it.

After that overlap, disable signing through the old Key Vault version.

An emergency CSRF rotation rejects the old kid immediately.

It forces users to refresh the portal before another state change.

Generate each session HMAC as 32 random bytes from the operating-system cryptographic random source.

Treat that value as a master key, not a direct signing key.

Derive six independent HMAC-SHA-256 subkeys with HKDF-SHA-256.

Use salt firecall-session-token-v1.

Use exactly these info values:

join-pending
join-ack
workspace-csrf
cursor
preview-handoff
preview-ack

Each route selects its expected subkey before it parses payload fields.

It never selects a subkey from an attacker-controlled purpose.

Each token has exactly three unpadded base64url segments.

Its protected header contains only alg=HS256, known kid, and its purpose-specific versioned typ.

Each payload contains fixed iss=firecall-proxy, exact route audience, purpose, version, iat, exp, and required purpose fields.

Use these exact types and audiences:

Purpose typ aud
Join pending firecall-join-pending+jwt;v=1 firecall:workspace:join-pending
Join acknowledgement firecall-join-ack+jwt;v=1 firecall:workspace:join-ack
Workspace CSRF firecall-workspace-csrf+jwt;v=1 firecall:workspace:csrf
Cursor firecall-cursor+jwt;v=1 firecall:workspace:cursor
Preview handoff firecall-preview-handoff+jwt;v=1 firecall:preview:handoff
Preview acknowledgement firecall-preview-ack+jwt;v=1 firecall:preview:ack

Reject padding, extra segments, unknown fields, duplicate JSON keys, an unprotected header, and an oversized token.

Reject a wrong type, issuer, audience, purpose, version, algorithm, key ID, time, or purpose field before authorization.

The workspace CSRF token binds the derived subject key, session identifier, and exact workspace host.

It expires after 15 minutes and permits two minutes of clock skew.

The browser requests it through same-origin GET /api/v1/csrf and keeps it only in memory.

The proxy requires it in X-Firecall-CSRF on every workspace POST and DELETE route.

The proxy also requires exact Origin and Sec-Fetch-Site: same-origin on those routes.

A route rejects a stale token or a token for another subject key, session, host, purpose, or key.

Write it to the inactive fixed Key Vault slot.

External Secrets refreshes that slot without logging its value.

Every proxy replica validates the candidate and reports its kid through the internal health endpoint.

Only then change the non-secret active-key ConfigMap.

The proxy signs each new token with only its purpose-derived subkey.

It verifies the prior HMAC for 24 hours.

The longest possible Firecall token remains below the eight-hour session ceiling.

After 24 hours, disable the old Key Vault version and remove its verifier on the next safe reload.

An emergency HMAC rotation adds the old kid to an immediate deny list.

It closes active proxy streams, revokes invitations, and forces new acknowledgements.

It does not preserve overlap for a known compromised key.

A protected rotation workflow records only key IDs, versions, timestamps, tests, and actors.

It never records credential values or signature payloads containing personal data.

Normal rotation failure leaves the prior active key selected.

Emergency rotation failure disables new opens and joins, preserves close and sweep, and pages the platform owner.

25. Terminal design

25.1 Terminal state

The agent manages one shared tmux session.

Named windows are browser terminal tabs.

Each window stores one immutable allowed namespace in agent metadata.

The top namespace selector affects new views and new terminals only.

The agent creates one read-only kubeconfig context file for each window.

That context sets the window's namespace as the Kubernetes default.

The agent exports that exact file through KUBECONFIG before the PTY starts.

The file contains only the random box-local proxy token.

It contains no Azure, Entra, or Kubernetes API bearer token.

A user can still name another profile namespace explicitly.

Azure RBAC, not the tab label, remains the authority boundary.

Refreshes do not remove windows.

Closing the browser detaches only.

End terminal destroys one selected window after confirmation.

The profile sets a maximum of eight windows.

25.2 Shell prompt

Show the active context:

firecall[dev/sf-aigw]$

Set KUBECONFIG before the first shell starts.

The prompt namespace and kubectl config view --minify namespace must match the tab label.

25.3 k9s

The user selects one allowed namespace.

The launcher uses:

k9s -n sf-aigw

Never generate k9s -A.

25.4 fc helper

Install a small helper with exact commands:

fc scope
fc contexts
fc pods sf-aigw
fc events sf-aigw
fc logs sf-aigw pod/aigw-68cb67856f-7jhk5 aigw
fc follow sf-aigw pod/aigw-68cb67856f-7jhk5 aigw
fc describe sf-aigw deployment/aigateway
fc k9s sf-aigw

The helper rejects namespaces outside the profile.

Azure RBAC remains the final authorization control.

25.5 Terminal safety

Use these limits:

  • 1 MiB maximum WebSocket message.
  • 1 MiB/s terminal-output rate per stream with a 2 MiB burst.
  • 2 MiB/s aggregate terminal-output rate per actor with a 4 MiB burst.
  • 4 MiB/s aggregate terminal-output rate per session with an 8 MiB burst.
  • 512 KiB/s live-log rate per stream with a 1 MiB burst.
  • 1 MiB/s aggregate live-log rate per actor with a 2 MiB burst.
  • 2 MiB/s aggregate live-log rate per session with a 4 MiB burst.
  • 3 MiB/s aggregate terminal-and-log rate per actor with a 6 MiB burst.
  • 8 MiB/s aggregate data rate per box with a 16 MiB burst.
  • 8 MiB/s aggregate data rate per proxy replica.
  • 10-second box dial timeout.
  • 30-minute idle timeout.
  • Bounded buffers.
  • Backpressure for slow clients.
  • No WebSocket compression.
  • 64 KiB paste maximum.
  • Confirmation above 8 KiB.

Run all interactive firecall processes in one systemd user slice.

Set these initial box limits:

  • MemoryMax=3G for the complete interactive-user slice.
  • TasksMax=128 for the complete interactive-user slice.
  • LimitNOFILE=256 for each interactive process.
  • A theoretical aggregate maximum of 32,768 descriptors across the interactive slice.
  • An 8-GiB dedicated ext4 home filesystem for the firecall user, backed by one reserved image file on the OS disk.
  • A hard bound of 131,072 inodes in that home filesystem.
  • A combined 1-GiB tmpfs limit for /tmp and /var/tmp.
  • A 256-MiB journal size limit.

Give the root-owned agent and expiry services separate slices.

Keep at least 4 GiB and 262,144 inodes outside the interactive user's home filesystem for root services and recovery.

The box does not use kernel disk quotas. Azure Ubuntu kernels build the VFS quota formats as modules that ship only in linux-modules-extra, which the marketplace image omits, so quotaon fails there. The dedicated home filesystem enforces the same byte and inode bounds and reserves its bytes before any session starts.

Give those services higher CPU and I/O weights than the interactive slice.

Reserve at least 256 MiB for the agent slice through cgroup v2 controls.

Do not give the shell a hard CPU quota when the agent remains healthy.

The agent reports a safe resource-limit error when it cannot create another terminal.

An interactive out-of-memory event must not stop the agent or expiry timer.

Disable OSC 52 clipboard control.

Disable automatic terminal hyperlinks.

Treat all terminal output as hostile bytes.

The agent uses separate bounded control and data queues.

It permits at most four attached terminals, four live logs, and six combined data streams per actor.

It counts requester, platform owner, and each collaborator by their exact subject key.

The proxy binds that digest to the acknowledged authorization record before its authenticated agent attach request.

The agent holds the shared per-actor counters across both proxy replicas.

Reserve 256 KiB of control-queue capacity and process expiry, revoke, heartbeat, and close frames before data frames.

Rate-limited output cannot consume the control queue.

26. Command palette

V2 has no global command-palette shortcut.

Common browser and operating-system shortcuts must keep their normal behavior.

Show a permanent command-palette button in the workspace toolbar.

Do not intercept readline or terminal shortcuts while xterm has focus.

Initial actions are:

  • New shell.
  • Open k9s.
  • Switch namespace.
  • Show unhealthy pods.
  • Show warning events.
  • Tail workload logs.
  • Follow workload logs.
  • Open a pod shell.
  • Open SigNoz logs.
  • Open SigNoz traces.
  • Copy the join link when the current human can mint one.

Read actions can execute through structured APIs.

Mutation actions open a visible terminal command.

The interface never silently mutates Kubernetes resources.

Generated commands contain exact names.

They contain no placeholders.

Examples:

kubectl -n sf-aigw get pods -o wide
kubectl -n sf-aigw get events --sort-by=.lastTimestamp
kubectl -n sf-aigw describe deployment aigateway
kubectl -n sf-aigw logs pod/aigw-68cb67856f-7jhk5 -c aigw --tail=200
kubectl -n sf-aigw logs -f pod/aigw-68cb67856f-7jhk5 -c aigw --tail=200
kubectl -n sf-aigw rollout status deployment/aigateway

27. Workload view

Show:

  • Pods.
  • Deployments.
  • StatefulSets.
  • Jobs.
  • Services.

The table shows:

  • Name.
  • Ready state.
  • Status.
  • Restarts.
  • Age.
  • Image.
  • Namespace.

The detail drawer shows:

  • Summary.
  • Conditions.
  • Containers.
  • Recent events.
  • Approved application labels.
  • Replica and readiness counts.
  • Container names and redacted image references.
  • Resource requests and limits.
  • Service ports.
  • Environment variable names and source types without values.
  • View logs.
  • Choose pod and container.
  • Open pod shell after that explicit choice.
  • Copy kubectl command.

Do not return arbitrary YAML, annotations, commands, arguments, or environment values.

Do not show Secret data or Secret names in the graphical response.

Redact URI user information from every image reference.

Do not add Secrets, RBAC, or ServiceAccounts to the graphical explorer.

The terminal has the session's resolved Reader and Writer authority.

The terminal can show only data permitted by that authority. Reader cannot view Secrets, roles, or role bindings. Writer can view Secrets and run Pods as namespace ServiceAccounts.

Controllers and Services show their matching Pods.

Firecall never chooses one replica silently.

Shell and log actions require one exact Pod and container.

28. Logs and events

28.1 Logs

The logs page provides:

  • Namespace selector.
  • Workload or pod selector.
  • Container selector.
  • Current or previous logs.
  • Tail count.
  • Timestamps.
  • Text filter.
  • Line wrap.
  • Pause and resume.
  • Follow mode.
  • Copy selected lines.
  • Open in terminal.
  • Open historical logs in SigNoz.

Current and previous Kubernetes logs require one exact Pod and container.

Do not select a Deployment replica automatically.

Kubernetes supplies current logs through the box identity.

Escape logs as plain text.

Do not store log lines in browser storage.

The agent counts current, previous, and follow-log bytes against the section 25.5 live-log and box limits.

The proxy enforces the lower per-replica aggregate limit before it forwards another data frame.

28.2 Events

Show warning events first.

Show:

  • Type.
  • Reason.
  • Object.
  • Message.
  • Count.
  • First seen.
  • Last seen.

Every request includes one explicit allowed namespace.

Never issue a cluster-wide event list.

29. SigNoz

Firecall opens SigNoz in a new tab.

Firecall does not embed SigNoz.

Firecall holds no SigNoz API key.

Use the resource attribute with periods:

k8s.namespace.name="sf-aigw"

Do not use:

k8s_namespace_name

Add a test that rejects the underscore form.

Call the URL a prefilled filter.

Do not call it namespace authorization.

SigNoz remains its own authorization boundary.

The canary must verify the current SigNoz logs, traces, and metrics URL syntax.

30. HTTP web previews

Web previews are part of the complete v2 release.

ADR-0016 fixes the exact-Pod resolution and private process boundary.

ADR-0017 fixes the separate-domain authentication, handoff, HTTP transport, and browser boundary.

The current ADR-0017 source slice supports bounded HTTP and HTTPS requests. It rejects WebSockets and application Authorization forwarding. It does not include the access-policy canary, external-redirect confirmation, or reusable upstream connection pools. The applicable statements below describe the complete v2 target, not the current source slice.

Use a different registrable domain from the portal and workspace.

A registrable domain is the domain boundary where a site can set parent cookies.

FIRECALL_APEX and FIRECALL_PREVIEW_APEX must have different registrable domains.

The preview apex must use a separately registered domain in its own Cloudflare zone.

Another openmined.org subdomain does not satisfy this rule.

The broker, proxy, Cloudflare apply, and CI use the same pinned Public Suffix List validation.

Use:

https://<preview_id>--<session_id>.FIRECALL_PREVIEW_APEX

Generate preview_id only on the server from 128 cryptographically random bits.

Encode those 16 bytes as exactly 26 lowercase unpadded Base32 characters.

Require canonical trailing bits and the alphabet [a-z2-7].

Do not accept a user-supplied preview identifier.

Create the preview record conditionally and regenerate after a collision, with at most five attempts.

Parse the leftmost host label as exactly <26-character-preview-id>--<validated-session-id>.

Require one separator, lowercase ASCII, the exact preview apex, and exact record bindings.

Reject an extra separator, IDNA, percent encoding, trailing dot, mixed case, port suffix, empty label, or noncanonical Base32.

Require HTTP Host, HTTP/2 :authority, Cloudflare route, preview record, and authenticated session to match exactly.

Strip every client forwarding header before the proxy creates its fixed upstream headers.

Do not enable previews on the portal's registrable domain.

This prevents an untrusted application from setting portal or workspace parent-domain cookies.

The user selects:

  • Allowed namespace.
  • Service or pod.
  • Declared port.
  • HTTP or HTTPS upstream protocol.
  • Optional display name.

For a Service preview, Firecall resolves the exact backing Pod before creation.

It lists only EndpointSlices with the exact kubernetes.io/service-name label.

Each slice must have:

  • A matching owner reference to the selected Service UID.
  • endpointslice.kubernetes.io/managed-by=endpointslice-controller.k8s.io.
  • The same namespace as the Service.
  • Only TCP ports.

Every endpoint must have a Pod targetRef with name and UID.

Firecall fetches that Pod and requires the same UID, namespace, address, and complete Service selector match.

Every target Pod must be not terminating and permitted by the preview security rules.

Its controller-owner chain must resolve by UID to a Deployment, StatefulSet, DaemonSet, or Job in that namespace.

Firecall rejects the complete Service when any endpoint is stale, mixed, external, unresolved, or unowned.

It offers only endpoints with ready=true and a Ready Pod.

It also rejects zero ready endpoints.

Firecall maps the selected Service port to one exact declared Pod port.

A named target port must resolve to one unambiguous declared container port.

The UI shows all eligible backing Pods and requires one explicit Pod choice.

The preview record binds the Service and selected Pod UIDs and resource versions.

It also binds each EndpointSlice UID and resource version, workload identity, port mapping, and protocol.

The agent resolves those facts before it starts a loopback port forward to that exact Pod.

After kubectl reports readiness, the agent resolves them again and compares the complete target.

The agent exposes the loopback endpoint only when both resolved targets match.

It never asks kubectl port-forward service/... to choose a replica.

If the Pod or slice changes, the forward ends and the user must select a current backing Pod.

Use the Kubernetes appProtocol value only as the initial protocol choice.

The user must confirm the final protocol.

HTTPS validates the upstream certificate with system and reviewed profile CA roots.

Do not offer an insecure certificate bypass.

For a Service, set upstream Host and TLS ServerName to:

<service>.<namespace>.svc.<profile-dns-domain>

Direct Pod previews support HTTP only in v2.

Reject direct Pod HTTPS because it has no reviewed stable TLS identity.

The agent starts a loopback-only Kubernetes port forward.

The proxy forwards HTTP, HTTPS, and WebSocket traffic.

Rules are:

  • Profile namespaces only.
  • Service and pod targets only.
  • Declared target ports only.
  • Managed Pods only.
  • Services with selectors only.
  • Endpoints owned by the selected workload only.
  • No ExternalName Services.
  • No hostNetwork, hostPort, hostPath, or privileged target Pods.
  • No arbitrary IP address.
  • No arbitrary hostname.
  • No raw TCP browser tunnel.
  • Three previews per session.
  • Sixty new HTTP requests per subject and preview each minute.
  • One hundred twenty new HTTP requests per subject and session each minute.
  • Three hundred new HTTP requests per complete session each minute.
  • Eight active HTTP requests per subject.
  • Sixteen active HTTP requests per preview.
  • Thirty-two active HTTP requests per session.
  • Eight queued requests per preview and sixteen per session.
  • One-second maximum queue wait.
  • Fifteen-minute preview idle timeout.
  • 16 MiB maximum request body.
  • 64 MiB maximum non-streaming response body.
  • 32 KiB maximum upstream response headers.
  • 1 MiB maximum WebSocket message.
  • Two WebSocket connections per preview.
  • Thirty-minute maximum WebSocket lifetime.
  • Five-second upstream dial timeout.
  • Fifteen-second upstream response-header timeout.
  • Sixty-second idle timeout.
  • Five-minute total non-WebSocket request timeout.
  • Thirty-second request-body read deadline.
  • Thirty-second upstream response-progress deadline.
  • 2 MiB/s per-actor preview byte rate with a 4 MiB burst.
  • 4 MiB/s per-preview byte rate with an 8 MiB burst.
  • 8 MiB/s aggregate preview byte rate per session with a 16 MiB burst.
  • 1 GiB cumulative transfer per preview and 3 GiB per session.
  • Reject CONNECT and TRACE.
  • Strip hop-by-hop and client-supplied forwarding headers.
  • Set trusted Forwarded values from the validated preview request.
  • All previews stop at close or expiry.
  • Remove the Cloudflare Access cookie before the box.
  • Strip every application request cookie before the target.
  • Reject every application Set-Cookie response.
  • Rewrite safe same-preview redirects back to the preview origin.
  • Intercept external HTTPS Location redirects and show a trusted confirmation.
  • Reject external non-HTTPS redirects.
  • Reject redirects to private or loopback hosts.
  • Strip Cloudflare identity headers.
  • Use Cache-Control: no-store, private.
  • Add worker-src 'none' to the response policy.
  • Treat applications that require service workers as unsupported.
  • Show a trusted Untrusted application preview interstitial before opening.

Do not inject a banner into application HTML.

The proxy cannot intercept JavaScript navigation, meta refresh, or normal link clicks.

The separate untrusted domain remains the security boundary for those navigations.

Cookie-dependent preview applications are unsupported in v2.

Applications can use explicit Authorization only when that preview enables forwarding.

The proxy strips it for every other preview.

JavaScript can create parent-domain cookies inside the untrusted preview domain.

The proxy strips those cookies, so they never reach any preview target.

The separate preview Access application uses the current Cloudflare Access session when that session still satisfies its one-time PIN policy.

Cloudflare can complete this step without another credential prompt.

The browser can still show an Access transition because the preview audience differs.

The proxy validates the preview audience before it contacts the agent.

The proxy derives one opaque per-session actor handle from the subject key.

The box receives that handle, not the subject key, raw subject, email, Access token, or browser identity.

The agent enforces actor, preview, and session totals across all proxy replicas.

Count request bodies, response bodies, and both WebSocket directions against the same byte budgets.

When a byte-rate budget empties, pause reads with backpressure before a bounded timeout.

When a cumulative budget expires, close HTTP streams and WebSockets with a safe quota reason.

The proxy also enforces lower local defensive limits before it opens an agent stream.

Set the upstream pool to 16 active and four idle connections per preview.

Set pool idle expiry to 30 seconds.

Never share one upstream pool across preview targets.

An acknowledged workspace mints a 60-second signed preview handoff.

The handoff binds subject key, session, profile digest, preview identifier, target hash, authorization kind, authorization identifier, owner-list revision, and expiry.

Authorization kind is exactly requester, platform_owner, or invitation.

The first two kinds use their fixed subject-key-derived authorization identifier.

The invitation kind uses the exact invitation ID that authorized the workspace join.

Place the handoff in the preview URL fragment.

Reserve this preview-origin route:

POST /.firecall/v1/preview-acknowledgements

Also reserve POST /.firecall/v1/access-policy-canary for the preview-audience Access identity canary.

Neither reserved route can reach preview content or the box.

The trusted preview interstitial reads the fragment.

It removes the fragment with history.replaceState before it posts the handoff in a JSON body.

It reloads the same path after the acknowledgement succeeds.

The acknowledgement route never forwards to preview content.

It requires JSON, the exact preview Origin, and the preview Access audience.

That endpoint derives the preview Access JWT subject key and requires it to match the handoff subject key.

The proxy then sets __Host-firecall_preview_ack on that preview host.

The signed cookie repeats the authorization kind, exact authorization identifier, and owner-list revision.

The browser, not the proxy, clears the fragment through the trusted interstitial.

The cookie is Secure, HttpOnly, SameSite Strict, host-only, and scoped to /.

Its expiry cannot exceed the session expiry.

Each request also requires the current preview record.

Effective access therefore cannot exceed the current preview idle expiry.

The proxy serves application content only after that cookie validates.

For each HTTP request, it rechecks requester, platform-owner, or exact invitation authorization before contacting the agent.

For each preview WebSocket, it repeats that check at most every five seconds.

It closes the WebSocket immediately when the invitation, session, or subject-key authorization fails.

A preview URL without a valid handoff directs the human to the workspace preview view.

A collaborator must join the workspace first, then open the preview with their own handoff.

Target checks reduce accidental exposure.

They cannot make a write-selected namespace safe from server-side request forgery.

A Writer can deploy an allowed Pod that proxies other reachable services.

Do not call the complete v2 release ready without the separate preview domain.

The core workspace can canary before preview activation.

31. Browser security

Use separate portal and workspace origins.

The workspace origin exposes no portal control API.

Every broker and proxy response uses Cache-Control: no-store, private.

This rule includes HTML, JSON, errors, redirects, interstitials, terminal upgrades, and preview content.

Parse and validate the inbound host before JWT audience, cookie, route, redirect, or CSP selection.

Accept only exact portal.FIRECALL_APEX, <validated-session-id>.FIRECALL_APEX, or the section 30 preview host.

HTTP/1.1 requires one canonical Host header.

HTTP/2 requires one canonical :authority and rejects a mismatching Host header.

Reject duplicate hosts, a port suffix, trailing dot, IDNA, percent encoding, mixed case, empty label, unknown session, or cross-route host.

Bind the selected Access audience, host-only cookies, Origin check, route class, session, and CSP substitution to that parsed host.

Never construct a response header, redirect, cookie, or policy from the unparsed request host.

Validate the raw request path before identity or route authorization.

Accept only strict ASCII route patterns declared in this plan.

Reject encoded slash, encoded backslash, encoded dot, NUL, invalid escape, invalid UTF-8, semicolon parameter, or dot segment.

Reject duplicate slashes and any mismatch between the raw and decoded path.

Do not clean, normalize, or redirect an invalid path.

Reserve the complete /.firecall/ prefix for Firecall before any preview request can reach an application.

Apply the same parser to portal, workspace, preview acknowledgement, API, and WebSocket upgrade routes.

Enforce these limits before JWT parsing:

  • 8 KiB maximum HTTP/1.1 request line.
  • 32 KiB maximum decoded request headers.
  • 64 maximum header fields.
  • 8 KiB maximum one header field.
  • Five-second header and TLS-handshake deadlines.
  • Thirty-second request-body read deadline.
  • Sixty-second idle connection timeout.
  • 512 pre-authentication connections per replica.
  • 16 pre-authentication connections per validated Cloudflare source address.
  • 32 concurrent HTTP/2 streams per connection.

Reject duplicate Host, :authority, Access assertion, Authorization, Cookie, Origin, CSRF, fetch-site, content-length, and transfer-encoding fields.

Reject a content-length and transfer-encoding combination, conflicting lengths, unsupported transfer coding, and HTTP/2 connection-specific headers.

Apply the route body limit during streaming and stop reading immediately after the first excess byte.

Cloudflare Cache Rules bypass caching for every portal, workspace, and preview hostname.

For an original HTTP request, return a fixed HTTPS 308 before request-body parsing or application routing.

Trust the original-scheme signal only on the authenticated Cloudflare-to-origin path.

A direct or malformed scheme signal fails closed.

Every HTTPS response adds this header:

Strict-Transport-Security: max-age=31536000; includeSubDomains

HSTS means HTTP Strict Transport Security.

Do not add either registrable domain to the browser preload list without a separate owner decision.

Every Firecall HTML response also uses:

Cache-Control: no-store, private
X-Content-Type-Options: nosniff
Referrer-Policy: no-referrer
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Resource-Policy: same-origin

Use this exact Permissions Policy on every portal, workspace, interstitial, and preview response:

Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), bluetooth=(), camera=(), clipboard-read=(), clipboard-write=(self), geolocation=(), gyroscope=(), hid=(), magnetometer=(), microphone=(), midi=(), payment=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), usb=(), xr-spatial-tracking=()

Use this portal Content Security Policy:

default-src 'none'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; script-src 'self'; style-src 'self'; style-src-attr 'unsafe-inline'; img-src 'self' data:; font-src 'self'; connect-src 'self'; worker-src 'none'; child-src 'none'; frame-src 'none'; manifest-src 'none'; media-src 'none'; report-to firecall-csp

Use this workspace Content Security Policy after substituting the validated current host:

default-src 'none'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; script-src 'self'; style-src 'self'; style-src-attr 'unsafe-inline'; img-src 'self' data:; font-src 'self'; connect-src 'self' https://<workspace-host> wss://<workspace-host>; worker-src 'none'; child-src 'none'; frame-src 'none'; manifest-src 'none'; media-src 'none'; report-to firecall-csp

Use this trusted preview-interstitial Content Security Policy:

default-src 'none'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; script-src 'self'; style-src 'self'; img-src 'self'; connect-src 'self' https://<preview-host>; worker-src 'none'; child-src 'none'; frame-src 'none'; manifest-src 'none'; media-src 'none'; report-to firecall-csp

Use this untrusted preview-content policy after substituting the validated preview host:

default-src 'none'; base-uri 'none'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; script-src 'self'; style-src 'self'; style-src-attr 'unsafe-inline'; img-src 'self' data: blob:; font-src 'self' data:; connect-src 'self' https://<preview-host> wss://<preview-host>; worker-src 'none'; child-src 'none'; frame-src 'none'; manifest-src 'none'; media-src 'self' blob:; report-to firecall-csp

No policy permits inline script, eval, an external script, an iframe, or a service worker.

The proxy removes an upstream preview CSP and applies the fixed untrusted policy.

Applications that need inline scripts, external APIs, framing, or workers are unsupported in v2.

Set one same-origin Report-To group named firecall-csp.

Its endpoint is /.firecall/v1/csp-reports on the current origin.

Accept only bounded CSP report JSON and store only directive, route class, and safe reason counters.

Never store a blocked URL, source snippet, subject, token, or application payload.

Rate-limit that endpoint before parsing.

Use host-only cookies where Firecall creates cookies.

Use the broker ES256 CSRF token for portal mutations.

Use the proxy HMAC workspace-CSRF token for workspace mutations.

Use exact Origin and Sec-Fetch-Site checks.

Enforce these initial request limits:

  • 16 KiB maximum portal request body.
  • Five create attempts per subject in ten minutes.
  • Twenty close attempts per subject in ten minutes.
  • Thirty-two workspace streams per session.
  • Eight live log streams per session.
  • Eight terminal windows per session.
  • Four attached terminal streams per actor.
  • Four live log streams per actor.
  • Six combined terminal and live-log streams per actor.
  • Two presence connections per actor.
  • Eight presence connections per session, with the final-slot fairness rule from section 8.4.

The box agent enforces every per-session stream, terminal, and preview total.

Proxy replicas also enforce local defensive limits.

Proxy-local counters never replace the agent's shared total.

Return 429 with a bounded retry time.

Use one Blob lease per subject and action for create and close limits.

This keeps the limit exact across both broker replicas.

Cloudflare edge rate limits provide an additional IP-based burst control.

The atomic capacity reservation remains the hard session limit.

Do not use local storage for:

  • Access tokens.
  • Session status.
  • Terminal data.
  • Kubernetes data.

On terminal and structured-agent routes, the proxy removes:

  • Every Cookie value.
  • Authorization.
  • Cf-Access-Jwt-Assertion.
  • CF-Connecting-IP.
  • All inbound X-Firecall-* headers.

On preview routes, the proxy removes:

  • The complete Cookie header after it validates Firecall and Cloudflare cookies.
  • Cf-Access-Jwt-Assertion.
  • CF-Connecting-IP.
  • All inbound X-Firecall-* headers.

They can also forward application Authorization after the preview warning.

Never log application cookie or authorization values.

Terminal and structured responses reject every upstream Set-Cookie.

Preview responses reject every upstream Set-Cookie.

Request cookies contain no host-only or Domain metadata.

Do not attempt to infer cookie scope at the proxy.

The proxy does not forward browser identity to the box.

32. Audit and observability

Audit remains session-level.

Use firecall-audit as the durable audit sink.

Create one object for each event with If-None-Match: *.

Canonicalize event JSON with RFC 8785 before hashing.

Derive one logical event UUIDv5 from event type, session or HTTP request, and operation epoch.

Put the lowercase SHA-256 payload digest in the immutable object path.

Create the durable receipt or revocation-marker audit fields before the append attempt.

An append succeeds on object creation or a precondition response for that exact immutable digest path.

Every other storage response is a failure.

Update the receipt or marker to committed through its ETag only after that result.

An independent protected audit-verify.yml control workflow runs every thirty minutes.

Its event-specific privileged-audit-verify-schedule.yml and privileged-audit-verify-manual.yml jobs use a dedicated read-only OIDC identity.

That identity can list and read audit objects and read correlated request, status, and tombstone facts.

It cannot create, update, or delete any storage object or Azure resource.

The workflow checks path digests, schemas, logical-event duplicates, and required event order.

It alerts on one logical event with different payload digests.

It cannot create, overwrite, or delete an audit object.

It never prints email, raw subject, subject key, namespace lists, or complete event payloads to GitHub logs.

Apply a time-based immutable retention policy after the owner approves retention.

Lock that policy after the canary proves append, read, alert, and retention behavior.

Give each event producer create and append actions only.

Do not give event producers read, list, overwrite, delete, or policy authority.

A compromised producer can add false events.

It cannot change an event after immutable storage accepts it.

Keep Azure Storage resource logs outside the Firecall producer identities.

Record outside the box:

  • Identity acceptance or denial.
  • Open acceptance or denial.
  • Capacity reservation.
  • Every lifecycle transition.
  • Readiness result.
  • Join, leave, and denial.
  • Close request and actor.
  • Expiry.
  • Cleanup result.
  • Rollback failure.
  • Certificate mismatch.
  • Status-read failure.
  • Capacity denial.

Every event contains:

  • Schema version.
  • Event identifier.
  • Event type.
  • Producer component.
  • UTC time.
  • Result.
  • Safe reason code.

The strict event schema has event-specific oneOf variants.

Correlation fields are request, workflow run, session, subject key, normalized email, profile, and namespaces.

An event includes each known correlation field and uses JSON null only where its event variant permits it.

Pre-authentication denial permits all identity and session fields to be null.

It never records an unverified token claim as identity.

Open acceptance requires request, session, verified subject key, email, profile, and namespaces.

Dispatch acceptance also requires the workflow run identifier.

Capacity and lifecycle events require every correlation field.

Join and close events require session, verified acting subject key, acting email, and actor role.

Cleanup events require session, workflow, terminal mapping, and zero-resource-proof digest.

CI validates each event variant against audit-event.schema.json.

Never log:

  • JWTs.
  • Cookies.
  • GitHub installation tokens.
  • Private keys.
  • Signed request envelopes.
  • Raw join capabilities and preview handoffs.
  • Application Authorization values.
  • Terminal input.
  • Terminal output.
  • Kubernetes logs.
  • Secret values.

Export service metrics and traces through OpenTelemetry.

Use one closed telemetry attribute allowlist at every SDK and again at the collector.

Permit only:

  • service.name.
  • service.version.
  • deployment.environment.
  • firecall.operation from a fixed enum.
  • firecall.result from a fixed enum.
  • firecall.reason_code from a fixed enum.
  • firecall.state from the lifecycle enum.
  • http.request.method.
  • A templated http.route with no identifier.
  • http.response.status_code.
  • network.protocol.version.

Drop every unlisted resource, span, event, link, metric label, and baggage attribute before export.

Disable request and response bodies, headers, cookies, query strings, URLs, user agents, baggage, and exception messages.

Do not export subject, email, reason, session, request, workflow, namespace, Pod, terminal, preview target, authorization, or Kubernetes-log data.

Use fixed route templates and enum values to bound metric and trace cardinality.

Do not use a dynamic value as a metric label.

The collector repeats the allowlist and rejects a batch with a forbidden key.

Add dashboards for:

  • Create latency.
  • Ready latency.
  • Occupied capacity-ledger slots.
  • Session state counts.
  • Join failures.
  • Proxy connection failures.
  • Cleanup latency.
  • Sweep lag.
  • rollback_failed count.

Keep the closed notification contract at platform/firecall/contracts/notification-targets.json.

Do not use an unnamed or unverified existing notification path.

The contract pins one Azure Monitor Action Group resource ID and one SigNoz contact-point UID.

It also pins the Azure subscription, resource group, action-group name, SigNoz organization ID, notification-policy UID, and owner group object ID.

Record only receiver type, enabled state, redacted destination digest, source ETag, last test receipt ID, and sanitized readback digest.

Keep receiver addresses, webhook URLs, Slack tokens, and other destination secrets outside Git.

The named platform notification owner supplies and controls those secret receiver values.

Give the trust observer Microsoft.Insights/actionGroups/read at only the exact Action Group resource ID.

Give no Firecall runtime, control, or budget identity Action Group mutation authority.

The budget root can reference only the exact merged Action Group ID.

Firecall holds no SigNoz API key.

An authenticated SigNoz administrator runs scripts/ops/firecall-notification-verify.sh in readback and test modes.

The script accepts only the merged organization, contact-point, notification-policy, and safe test identifiers.

It changes no receiver, route, contact point, organization setting, dashboard, or alert rule.

The owner records a sanitized signed readback and one real receipt from each destination.

Send service alerts through only that exact SigNoz contact point and notification policy.

Also fail the applicable GitHub workflow and publish its safe job summary.

Test delivery for rollback failure, sweep lag, storage failure, certificate expiry, budget thresholds, and each trust observer.

Do not claim operator notification until alert delivery passes a live test.

A missing Action Group, contact point, policy, owner, permission, destination digest, or current receipt blocks activation.

33. Supply chain

Every release needs verified source, build identity, and immutable content.

  1. Pin every GitHub Action by full commit SHA.
  2. Pin every container image by digest.
  3. Pin the Go toolchain.
  4. Commit go.sum.
  5. Commit the front-end lock file.
  6. Vendor xterm.js assets and licenses into built artifacts.
  7. Fetch no browser asset from a CDN.
  8. Pin the Azure Compute Gallery image version.
  9. Produce SBOMs for every binary, container, and VM image.
  10. Produce build provenance.
  11. Scan binaries, containers, and the VM image.
  12. Block image publication from pull requests.
  13. Give pull-request workflows no Azure OIDC credential.
  14. Forbid pull_request_target in the control repository.
  15. Protect workflows, profiles, Terraform, and public keys with CODEOWNERS.
  16. Permit only reviewed actions in the control repository.
  17. Pin the exact Ubuntu marketplace publisher, offer, SKU, and version.
  18. Pin a dated Ubuntu package snapshot for image builds.
  19. Pin every installed tool version and verify its published checksum.
  20. Pin Terraform and Packer CLI versions, checksums, release signatures, and verification keys.
  21. Commit .terraform.lock.hcl for every root module and required runner platform.
  22. Run terraform init -lockfile=readonly in validation and control jobs.
  23. Pin Packer required_plugins and load only the verified cached Azure plugin from an isolated directory.
  24. Install no package during session boot.
  25. Sign every container manifest, SBOM, provenance statement, and scan result.
  26. Publish signed VM-image evidence as an OCI artifact in ACR.
  27. Verify all release evidence before an image-pin pull request can open.
  28. Verify the pinned VM evidence again before a session can reserve capacity.

Keep direct tool artifacts in platform/firecall/toolchain/toolchain.lock.json.

Keep the canonical union at platform/firecall/toolchain/acquisition.lock.json.

Use these initial Linux AMD64 entries:

Tool Version and artifact SHA-256 or package integrity Required verification
Terraform CLI 1.15.9, terraform_1.15.9_linux_amd64.zip 76edd0b22d2f27d3d2e097cd793209646f719cf60f02ff3af626b07361137da1 HashiCorp signed checksum manifest
Packer CLI 1.15.4, packer_1.15.4_linux_amd64.zip 15f97a6a99645c7d5308c609973b5280837b38e112beac413ccbce80da927cf1 HashiCorp signed checksum manifest
Azure Packer plugin 2.6.3, packer-plugin-azure_2.6.3_linux_amd64.zip, commit 7f7e7b4aabf41dc853d6d65c494d95c258150bc4 fac73884ea60eadf63b1c14105b588997e22e68b0c4f46a62d7277677c3ef189 HashiCorp signed checksum manifest, exact required_plugins, and isolated cached binary
Azure CLI 2.89.1, azure-cli_2.89.1-1~noble_amd64.deb 3608abe64c5c1a407d468b790a2187b26d4df7f709f747414d19143d909e45da Microsoft-signed APT InRelease and package index
Go 1.27.0, go1.27.0.linux-amd64.tar.gz 675c26c449cbb18fc24b74650de1eabbae6e16f64326fd85a283fb3b58280685 Official Go release checksum
Node.js LTS 24.19.0, node-v24.19.0-linux-x64.tar.xz 14b342e71204f811bde6153be8e04b62aef63c236fef92b55f9c83154b409647 Node-signed SHASUMS256.txt.sig
pnpm 11.23.0, npm package sha512-8ACC5bKDoZm3Tgedoo0VXACP4jL0TIoG6n3foBTs9xn8Ni95DsxnsoEG3awq+yTEmpoHkVnaVgss59Hpjv0Rrw== npm signature key SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U and tarball SHA-1 0a15e2d3f345118a5fe6850597fba5c09c036f3b
kubectl 1.35.6, bin/linux/amd64/kubectl 5d11e2ba01ea68ffd053f56e27738e2b4330013ee67f7e46c6da6c585d3c9926 Official dl.k8s.io checksum; version must match live AKS
kubelogin 0.2.19, kubelogin-linux-amd64.zip ebaeff02aa899c5cae6a2b954b64fc02738185319df2570f7dc053451efa4b2f Official GitHub release checksum asset
k9s 0.51.0, k9s_Linux_amd64.tar.gz c3752ad51a5a4015a113819c4eeb6e55a4d0e4b8e652494797532f6fc8161dd7 Release manifest whose digest is dda56946b28b50a56edc43a1b7e24563308f8e1ef5070b1febfc567d63b66b96
Cosign 3.1.3, cosign-linux-amd64 4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71 Sigstore bundle digest e16547fbee348eb23bd7e5a4d542b540395faea2e7bb1d18da01bbc3cc74d57d
Trivy 0.73.0, trivy_0.73.0_Linux-64bit.tar.gz 2edd39da482bb4e9831962487b68f68e3928ec3137794757f54d00383d79547b Sigstore bundle digest ca61b1a42a1ccd92cbdb7a94fbb4cba2b7aba0051d3d0fda4b5bcde63fa18fbd
ClamAV 1.5.3+dfsg-0ubuntu0.24.04.1, Noble AMD64 package 655057edd5efa990e791e9b6c0ffde2f38e587a4a6afd021e94523bae1d2e965 Ubuntu Noble signed package index and pinned snapshot

Add each exact lock entry before the group that introduces its consuming source merges:

Class Required logical entries Closure stage
VM direct tools stern, helm, yq, fzf, NATS CLI, and grpcurl Group C
VM Ubuntu packages tmux, jq, postgresql-client, redis-tools, curl, dnsutils, openssl, socat, less, vim, chrony, nftables, openssh-server, ca-certificates, cloud-init, Azure Guest Agent, and every transitive package Group C
Container build Rootless BuildKit image and buildctl, ORAS, Syft, Cosign, Trivy, ClamAV, Go, Node.js, and pnpm Group C for runtime inputs; Group D for web inputs
Test build Playwright package, Chromium, Firefox, WebKit, each browser dependency, axe package, and test fonts Group D
Operator browser canary firecall-browser-token-helper, its vendored Go modules, pinned Playwright Chromium, browser dependencies, and fonts Group C; Group E executes the live identity canary
Infrastructure build Terraform, Packer, Azure Packer plugin, Azure CLI, kubectl, and kubelogin, plus every provider package used by a merged root Group C for direct tools; Group E for E-owned provider packages
Acquisition bootstrap Exact GitHub runner image plus bash, curl, gpgv, sha256sum, tar, unzip, xz, zstd, jq, git, strace, and container-runtime executable digests Group C
OCI inputs Every runtime base image, rootless BuildKit image, scanner helper image, and test image manifest, config, and layer The group that introduces each image consumer
Source dependencies Every vendored Go module and license, every pnpm tarball and integrity, and every vendored browser or xterm asset Group C for Go; Group D for web and browser assets

The implementation selects one current supported version for each missing direct tool.

It records exact URL, byte size, SHA-256, release signature, signer, source commit, license digest, platform, and install path.

No latest, moving release URL, package-manager range, tag-only OCI reference, or unverified Git archive can enter the lock.

Use platform/firecall/toolchain/executables.lock.json as the complete executable authority.

Each row binds executable path, owning package or artifact, version, file SHA-256, architecture, license, and permitted build or runtime phase.

Trace each build with execve capture and reject an executable absent from that file or used in the wrong phase.

For Ubuntu packages, bind the complete installed package manifest and each package's shipped executable-file digests.

For the GitHub runner, bind the approved image version, host executable digests, Docker client and daemon versions, and accepted bootstrap residual.

The runner remains a privileged trust holder; the lock detects a changed image but cannot make that host hermetic.

Keep exact GitHub Action repository, commit SHA, source-tree digest, permissions, and license in actions.lock.json.

Reject every unlisted action, Docker action, composite child action, mutable ref, or action that invokes an unlisted executable.

Keep every reviewed OCI index, platform manifest, config, layer digest, signature, provenance, source, and license in oci-images.lock.json.

Reject a platform fallback, foreign layer, tag resolution change, missing signature, or layer absent from the cache.

Pin the HashiCorp release key fingerprint to C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F.

Pin the Microsoft package key fingerprint to BC52 8686 B50D 79E3 39D3 721C EB3E 94AD BE12 29CF.

Pin the Node release key fingerprint to 5BE8 A3F6 C8A5 C01D 106C 0AD8 20B1 A390 B168 D356.

Pin the Ubuntu archive key fingerprint to F6EC B376 2474 EDA9 D21B 7022 8719 20D1 991B C93C.

Vendor those four public keys under platform/firecall/toolchain/keys/.

Verify Cosign with the reviewed Sigstore artifact key and identity keyless@projectsigstore.iam.gserviceaccount.com before using it.

Verify Trivy's bundle with issuer https://token.actions.githubusercontent.com.

Require certificate identity https://github.com/aquasecurity/trivy/.github/workflows/reusable-release.yaml@refs/tags/v0.73.0.

Bind Trivy release commit 40c73e5d6166dcc0346a1ab4e94499d1572854e4.

Each component lock records URL, media type, size, version, platform, digest, signature object, signer, license, and intended install path.

The acquisition lock binds the canonical SHA-256 digest of every component lock and every required artifact record.

Its closed union includes tools, actions, OCI objects, Ubuntu metadata and packages, Terraform providers, Packer plugins, scanner inputs, browsers, modules, packages, and assets.

The Ubuntu lock contains signed snapshot metadata and every direct and transitive .deb digest.

Each Terraform root lock contributes every provider package and verified zh: and h1: value.

The Go inventory binds vendor/modules.txt, every vendored module tree digest, module version, source, checksum, and license digest.

Do not acquire Go module .info responses.

The Go Modules Reference states that these records can change over time.

Go authenticates .mod and .zip content, but it does not authenticate .info content.

The offline proxy creates the minimal {"Version":"<locked-version>"} record from each locked module version.

Keep .mod and .zip verification bound to the reviewed go.sum values.

The pnpm inventory binds every offline-store tarball, package integrity, dependency edge, lifecycle-script policy, and license digest.

The browser inventory binds each Playwright browser archive, extracted-tree digest, required system package, font, and license.

The OCI inventory binds every index, platform manifest, config, layer, signature, provenance statement, and license.

The scanner inventory binds the scanner executables, rules, vulnerability databases, malware databases, current pointers, and publisher evidence.

Group C cannot merge until every member of its source union has a verified immutable pin and its complete acquisition digest passes CI.

The Group C source union contains every build, test, action, executable, OCI, package, module, browser, scanner, and asset input reachable from governed source.

Lock-extension changes now include the merged Group D portal and workspace inputs. The union excludes only planned source that does not exist yet.

Every later consuming change, including later Group B product source, first lands a small firecall-runtime lock-extension pull request when it adds a new artifact, version, executable, path, digest, license, architecture, or derived byte set.

That lock extension adds every new input before the consuming source merges.

A consuming change can add only a new phase to an already pinned executable in its own pull request.

That phase-only authority change must preserve the executable path, size, SHA-256 digest, license, architecture, source artifact, and all existing phases.

The consuming test must verify the pinned executable before it uses the new phase.

Other implementation groups do not directly edit the Group C-owned canonical toolchain tree.

Group E cannot publish an image until the activation union covers every merged Group A through E source and passes offline verification.

Reject an unknown tool, duplicate tool, missing field, moving URL, different platform, unlisted executable, or unverified transitive installer.

Acquire each artifact once through a credentialless review job.

Verify it before it enters the content-addressed build cache.

After acquisition, image and container builds use only the verified cache and make no tool download.

The acquisition workflow can publish several bounded artifacts, but one manifest binds every batch digest and complete membership.

Create one Blob container named firecall-tool-cache.

Apply the same locked 90-day WORM policy before its first writer receives authority.

Store public artifact bytes and signed acquisition evidence at these create-once paths:

artifacts/sha256/<64-lowercase-hex-digest>/<canonical-filename>
evidence/sha256/<artifact-digest>/<canonical-statement-digest>.json
bundles/sha256/<statement-digest>/<sigstore-bundle-digest>.json

Add firecall-tool-acquire.yml and commit-pinned firecall-tool-acquire-push-privileged.yml and firecall-tool-acquire-manual-privileged.yml on infrastructure main.

The first job has no OIDC, GitHub App, Azure, registry, Key Vault, or storage credential.

It downloads only fixed URLs from the merged tool lock and verifies size, SHA-256, signature, signer, and key fingerprint.

It uploads one bounded GitHub artifact with verified bytes, lock digest, and verification report.

The privileged publisher verifies the triggering workflow ID, protected commit, event, branch, run attempt, and artifact digest.

It repeats every verification before Azure login.

Use one dedicated tool-cache publisher application and exact protected-main FIC.

Give it read and create-only Blob data authority on the three exact cache prefixes, without list, write, delete, tag, or policy authority.

It writes through If-None-Match: *, rereads exact objects, and keyless-signs the canonical acquisition statement.

The statement binds source URL, redirect chain, byte size, media type, tool version, platform, digest, signature proof, lock digest, source commit, and workflow run.

An existing exact path succeeds only when its bytes and acquisition statement match completely.

Give image-builder and scanner-publisher identities read-only cache access without list authority.

Each consumer addresses one exact path from the merged lock and verifies bytes and signed evidence again before use.

The GitHub runner downloads verified cache bytes before Packer starts.

Packer transfers those bytes through fixed file provisioners and installs no package from the network.

Packer configures one signed local file: APT repository from the transferred snapshot packages.

It runs APT with --no-download and fails when a requested or transitive package is absent.

The build NSG permits only required Azure platform traffic and denies Internet package mirrors.

Container builds load every base and helper image from the verified OCI cache into rootless BuildKit.

They use only local verified cache inputs in their build context.

Set BuildKit network mode to none for every build step and reject an undeclared frontend or remote context.

Build Go only with -mod=vendor and a read-only verified vendor tree.

Run pnpm with --offline --frozen-lockfile against only the verified offline store.

Set PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 and use only the verified browser tree.

Run Terraform initialization with the exact local provider mirror and -lockfile=readonly.

Run Packer with only its isolated verified plugin directory.

Run every scanner with its exact cached rules and database generation and with updates disabled.

Capture attempted network connections and executable launches for Packer, BuildKit, Go, pnpm, Terraform, Playwright, and scanners.

Fail a build on any unlisted executable, uncached artifact, DNS query, registry pull, package download, module fetch, browser fetch, or scanner update.

Neither a Packer guest nor a container build stage receives a storage token.

Keep an artifact while any current tool lock, allowed image, release bundle, scanner lock, or rollback release references it.

After the final reference ends and WORM expires, the reference-aware retention workflow can delete it under the section 37.3 two-read rule.

Soft delete permits at most 30 days of recovery.

A compromised publisher can add forged new public cache objects but cannot overwrite an existing digest path.

Consumers still require the merged digest and a valid protected-workflow signature.

Keep exact source and version constraints in platform/firecall/packer/plugins.pkr.hcl.

Packer has no plugin lock-file command or .packer.lock.hcl contract.

Publish both the verified Azure plugin archive and matching signed checksum manifest in firecall-tool-cache.

Extract them only into one new mode-0700 temporary directory.

Set PACKER_PLUGIN_PATH to only that directory for packer plugins installed, packer validate, and packer build.

Require packer plugins installed to report only Azure 2.6.3 from the expected source and platform.

Reject another executable, plugin version, source, directory entry, or discovery path.

Run the build with network capture and prove Packer makes no plugin registry or download request.

Keep every Terraform root's .terraform.lock.hcl beside that root.

Keep the exact package manager and integrity in platform/firecall/web/package.json and pnpm-lock.yaml.

Keep VM Ubuntu packages and snapshot metadata in platform/firecall/packer/packages.lock.json.

Keep Trivy, ClamAV, Cosign, and database inputs in platform/firecall/scanners/scanners.lock.json.

Generate each lock file with terraform providers lock -platform=linux_amd64.

Preserve the registry-signed zh: hashes and the verified linux_amd64 h1: hash.

Do not add an untested platform-specific package hash.

Pin each Terraform provider source and version in source constraints and the lock file.

Fail CI when terraform init -lockfile=readonly changes files or requests an unlocked provider.

Use keyless cosign signatures from the protected image workflow.

Keyless means GitHub OIDC proves the signer identity without a stored signing key.

Grant id-token: write only to the reusable privileged image job on infrastructure main.

Accept only this signer identity:

  • Issuer https://token.actions.githubusercontent.com.
  • Repository OpenMined/infrastructure.
  • Repository identifier 510317576.
  • Repository-owner identifier 30185530.
  • Build-signer reference .github/workflows/firecall-images-privileged.yml@<approved-40-character-commit>.
  • Build-config reference .github/workflows/firecall-images.yml@refs/heads/main.
  • Ref refs/heads/main.
  • Source commit equal to the reviewed merge commit.

The exact Cosign certificate identity is:

https://github.com/OpenMined/infrastructure/.github/workflows/firecall-images-privileged.yml@<approved-40-character-commit>

Require a Sigstore bundle with each signature.

The bundle permits offline verification in pull-request CI.

Vendor the reviewed Sigstore trusted-root metadata under platform/firecall/trust/sigstore/.

Bootstrap trust from one checked-in TUF root.json version and exact SHA-256 digest.

TUF means The Update Framework.

Do not download the first trusted root during a build or verification job.

Record its version, threshold key IDs, source URL, retrieval evidence, and digest in tuf-bootstrap.lock.json.

Mirror the same bootstrap root and lock into the control repository through a reviewed pull request.

Use a pinned TUF client library and checked protocol fixtures.

Accept root updates only one sequential version at a time.

Verify each new root with the old and new root signature thresholds.

Then verify timestamp, snapshot, and targets metadata in that order.

Verify role signatures, thresholds, versions, hashes, lengths, expiries, and consistent-snapshot names.

Persist the highest accepted version for every role in reviewed source.

Reject rollback, freeze, mix-and-match, fast-forward root skips, expired metadata, and unknown target paths.

A separate credentialless update job downloads candidate public metadata.

It opens a CODEOWNERS-reviewed pull request with the complete sequential chain and verification report.

Normal builds and runners perform only offline verification against reviewed files.

Update both repository copies before timestamp or snapshot expiry.

The verification report fails when either repository has a different bootstrap or targets digest.

For every keyless signature, require:

  • A Fulcio leaf and complete chain to the reviewed TUF Fulcio roots.
  • The exact workflow certificate identity and OIDC issuer.
  • A valid certificate time and key usage.
  • A valid Certificate Transparency Signed Certificate Timestamp.
  • A Rekor entry bound to the artifact digest and signature.
  • A Rekor Merkle inclusion proof and signed checkpoint from reviewed TUF log keys.
  • A Rekor integrated time inside the Fulcio certificate validity interval.
  • A bundle media type and version from the reviewed allowlist.

Do not accept an inclusion promise without its inclusion proof and signed checkpoint.

Do not use a live Rekor, Fulcio, or TUF response to repair missing bundle evidence during verification.

Pin the Cosign binary version, checksum, and release signature.

Never use --check-claims=false, an insecure transparency-log option, or an identity wildcard.

Pull-request CI receives no Azure or registry credential.

For every container digest, publish:

  • An SPDX JSON SBOM.
  • SLSA v1 build provenance.
  • A vulnerability, secret, and malware scan result.
  • Signatures for the manifest and each attached statement.

Do not deploy the mutable upstream cloudflared image directly.

Build a Firecall mirror from one exact cloudflare/cloudflared source commit and verified release tag.

Store the upstream repository, commit, tag, release-checksum digest, signature proof, and license digest in cloudflared-source.lock.json.

Use the pinned Go toolchain and module lock, a reviewed minimal base digest, and --no-autoupdate in the final command.

Publish only to firecall/third-party/cloudflared@sha256:<digest> in ACR.

The protected Firecall image workflow produces its SBOM, provenance, vulnerability, secret, malware, manifest, and keyless signature evidence.

Qualification compares the compiled version and commit with the source lock and runs tunnel, metrics, and graceful-shutdown tests.

Preserve the upstream license and source notice in the image and release evidence.

The closed-set runtime manifest permits only this verified ACR digest for cloudflared.

The image-pin workflow and recurring rescan treat it exactly like every Firecall-built container.

For every VM gallery version, publish one signed in-toto statement.

Store that statement as an immutable OCI artifact under the Firecall evidence path in ACR.

Use the exact repository firecall/evidence and address every bundle by manifest digest.

Do not pass an ACR tag between workflows.

The protected build workflow uploads one non-secret firecall-evidence-reference GitHub artifact.

That JSON contains the OCI digest, Gallery version ID, source commit, workflow run ID, and canonical document digest.

Set its retention to seven days.

The image-seal workflow starts only from a successful workflow_run event for the exact build workflow on main.

It verifies the workflow numeric ID, event, branch, head commit, conclusion, repository ID, and owner ID.

It downloads the exact reference artifact from that triggering run through its read-only GITHUB_TOKEN.

It exchanges its image-sealer OIDC token for short-lived registry pull access.

It pulls firecall/evidence@sha256:<digest> through AcrPull and verifies every signature before Azure sealing.

After sealing, it uploads one keyless-signed firecall-seal-bundle GitHub artifact.

That bundle binds the build evidence digest, Gallery version, lock, ETag, source commit, and sealer run.

Set seal-bundle retention to 30 days.

The image-pin workflow starts only from that exact successful sealer workflow on main.

It verifies the same immutable run facts and downloads the exact seal bundle from its triggering run.

It uses its own short-lived AcrPull token to fetch the build evidence by the bound digest.

It rereads the exact Gallery version and lock through its read-only Azure identity.

It rejects a missing artifact, tag reference, run mismatch, digest mismatch, stale commit, or unsigned bundle.

Before artifact expiry, the pin workflow copies both verified bundles into each source pull request.

Those checked-in bundles become the durable offline verification path after merge.

Use a new monotonically increasing Gallery version name for every build.

Never reuse a version name, including after retirement.

Publish a new version as unsealed and excluded from latest.

Tag it with the exact source commit, build run, evidence digest, and creation time.

On an in-process failure, the builder deletes only that unsealed version before final cleanup.

The Gallery ReadOnly seal prevents deletion after successful sealing.

Run an audit-only image-seal inventory every 30 minutes.

It lists Firecall versions and deterministic seal locks through the existing read-only sealer authority.

It never creates a seal during an inventory event.

An unsealed version older than two hours enters logical quarantine.

No profile, Policy allowlist, image-pin workflow, or session can use a quarantined version.

The inventory correlates its source commit, build run, evidence digest, and workflow conclusion.

It alerts immediately and publishes a signed deletion proposal after seven days.

The proposal proves no protected profile, Policy allowlist, source pin, active session, or seal references the version.

Only an authenticated owner can run scripts/ops/firecall-image-orphan-delete.sh for that exact proposal.

The script rereads all proofs and asks for the explicit image-version deletion approval.

No scheduled or routine identity receives Gallery-version delete authority for orphan cleanup.

This owner path also handles a workflow crash after version creation and before evidence publication.

The statement binds:

  • The exact gallery image-version resource ID.
  • The source repository and commit.
  • The caller and privileged workflow files, refs, commit digests, and run identifier.
  • The Packer configuration hash, required_plugins hash, cached-plugin evidence digest, and fixed-NSG document hash.
  • The Packer CLI version, binary digest, release-signature result, and verification-key fingerprint.
  • The exact Ubuntu marketplace version.
  • The dated package snapshot.
  • Every installed tool version and checksum.
  • The exact Gallery version resource ID, provisioning state, source properties, and ETag.
  • The digest of the complete Packer manifest and build log evidence.
  • The SBOM digest.
  • The scan-result digest.

After publication, the independent image-seal workflow:

  1. Verifies the complete signed VM evidence bundle offline.
  2. Reads the exact Gallery version, source properties, provisioning state, and ETag.
  3. Compares those values with the signed statement.
  4. Creates the deterministic ReadOnly lock on that exact version.
  5. Rereads the version and lock.
  6. Proves the builder can no longer update or delete the version.
  7. Produces a keyless-signed seal statement and Sigstore bundle.

Azure Compute Gallery does not expose a content digest for the resulting OS disk.

Do not claim or invent one.

The signed build evidence, exact version properties, ETag, and independent ReadOnly lock form the verifiable image identity.

The seal statement binds version ID, version ETag, lock ID, lock level, evidence digest, sealer workflow, source commit, and time.

Accept only these paired seal certificate identities:

https://github.com/OpenMined/infrastructure/.github/workflows/firecall-image-seal-privileged.yml@<approved-40-character-commit>
https://github.com/OpenMined/infrastructure/.github/workflows/firecall-image-seal-recovery-privileged.yml@<approved-40-character-commit>

Verify each caller build-config URI as .github/workflows/firecall-image-seal.yml@refs/heads/main.

Accept the normal identity only for workflow_run.

Accept the recovery identity only for workflow_dispatch and a new firecall-live approval.

The image-pin workflow verifies:

  • The signature and Sigstore bundle.
  • The exact certificate issuer and workflow identity.
  • The certificate repository, numeric repository, numeric owner, and main ref.
  • The exact build-signer and build-config URIs and their commit digests.
  • The container digest or gallery version in the statement subject.
  • The expected SBOM and provenance predicate types.
  • Every base-image, package-snapshot, tool, and action pin.
  • The scan time and blocking thresholds.
  • The seal signature, bundle, exact sealer identity, and build-config identity.
  • The exact current ReadOnly lock, version ETag, evidence digest, and seal statement.

The image-pin App first opens only the infrastructure pull request.

That pull request stores the exact continuation request and proposed release subjects.

The Policy apply workflow saves an exact handoff after the infrastructure change merges and applies.

The automatic continuation verifies that merge, the saved apply, two live Policy reads, the original seal, and current builder authority.

The continuation then opens only the control pull request.

Each stage resumes an existing exact proposal and rejects changed proposal bytes.

Each pin pull request contains the build and seal bundles with their evidence digests.

Infrastructure and control pull-request CI repeat offline verification.

The control runner repeats VM evidence verification before it creates a reservation.

It compares the signed gallery resource ID with the enabled profile pin.

The enabled profile vendors the signed VM bundle and evidence digest.

The runner verifies that bundle without a registry credential.

It reads only the exact pinned gallery version metadata through the control identity.

It also requires the deterministic lock at ReadOnly and the exact sealed version ETag.

A missing, changed, weakened, or mismatched lock blocks capacity reservation.

The image builder cannot modify or delete a sealed version.

Lock removal requires a reviewed infrastructure trust change and explicit deletion approval.

Remove every profile pin before any approved image retirement.

Use these blocking scan rules:

  • Any critical vulnerability fails publication.
  • Any secret or malware finding fails publication.
  • A high vulnerability fails without a reviewed exception.
  • A scan older than 24 hours fails publication.

Store exceptions under platform/firecall/security-exceptions/.

Each exception names the CVE, affected digest, owner, reason, compensating control, and expiry.

An exception expires within 30 days.

CODEOWNERS requires platform and security review for each exception.

Expired or mismatched exceptions fail verification.

Use a native Kubernetes ValidatingAdmissionPolicy in both Firecall namespaces.

The policy rejects every container, init-container, and ephemeral-container image without a digest.

Set failurePolicy: Fail and binding action Deny.

Match Pod create, Pod update, and the Pod ephemeral-containers subresource.

Bind through the reviewed namespace label firecall.openmined.org/digest-policy=enforced.

Create the policy and binding before either labeled namespace receives a workload.

Do not add a ServiceAccount or emergency bypass.

The native policy cannot verify a Sigstore signature.

The protected pin workflow and pull-request CI perform signature and provenance verification.

GitOps can deploy only reviewed digest pins that passed those checks.

Use this image delivery order:

  1. Merge runtime and image source without publication authority.
  2. Apply the standing gallery, temporary build group, builder identity, and sealer identity.
  3. Publish the first qualified but unsealed image.
  4. Seal the exact version and publish the signed seal statement.
  5. Let the image-pin App open infrastructure and control pin pull requests.
  6. Review and merge every exact digest, evidence digest, seal, and VM image version pin.
  7. Enable v2 session creation.

Each later image uses the same pull-request pin flow.

33.1 Recurring security rescan

Run firecall-rescan.yml every six hours and permit a separate manual recovery dispatch.

Its fixed event-specific jobs call only the commit-pinned firecall-rescan-schedule-privileged.yml or firecall-rescan-manual-privileged.yml reusable workflow.

Keep one reviewed platform/firecall/release-evidence/required-subjects.json closed-set manifest.

It lists every enabled Kubernetes container and init-container digest, the enabled Gallery version ID, and its evidence digest.

The list includes Firecall code, rotators, validator, health sidecars, and third-party images such as cloudflared.

Mirror its canonical digest into the protected control profile and runtime ConfigMap.

The rescan binds only the protected infrastructure manifest and source commit into its signed verdict.

The broker compares its mounted runtime digest with that signed verdict before request acceptance.

The control runner compares its protected control-profile digest with that signed verdict before reservation.

Neither comparison gives the rescan identity control-repository, Key Vault, or Kubernetes access.

Never discover subjects from a mutable registry tag.

For each enabled container, pull the exact manifest digest and verify its original manifest, SBOM, provenance, and build signatures offline.

Scan the pulled filesystem again with pinned vulnerability, secret, and malware scanner binaries.

For the enabled VM version, verify the signed evidence, Gallery properties, ETag, and ReadOnly seal.

Re-evaluate its signed SPDX SBOM with the current vulnerability database.

Also reverify its signed build-time secret and malware results.

Azure Compute Gallery does not expose the captured OS-disk content for an independent recurring filesystem scan.

The VM rescan therefore detects new package vulnerabilities but not a newly detectable file absent from its signed SBOM.

Record this residual in every VM verdict.

Use only scanner binaries and rule packs from the checked-in scanner lock.

Use Trivy DB schema version 2 as the vulnerability database format.

Its OCI layer type is application/vnd.aquasec.trivy.db.layer.v1.tar+gzip.

Do not let the rescan consume the unsigned upstream artifact directly.

Run a separate firecall-scanner-db-update-schedule-privileged.yml publisher every six hours.

Use firecall-scanner-db-update-manual-privileged.yml only for manual recovery.

That publisher validates the exact upstream OCI media types, schema, metadata, archive limits, and digest from two configured distribution endpoints.

It stores the exact OCI artifact bytes under firecall-scanner-data/trivy-db/v2/<oci_digest>/<blob_sha256>.tar.

It uses create-only authority for scanner-data and separate exact-pointer authority for scanner status.

It keyless-signs that exact OCI and Blob digest with this separate identity:

https://github.com/OpenMined/infrastructure/.github/workflows/firecall-scanner-db-update-schedule-privileged.yml@<approved-40-character-commit>
https://github.com/OpenMined/infrastructure/.github/workflows/firecall-scanner-db-update-manual-privileged.yml@<approved-40-character-commit>

The rescan accepts only that identity, GitHub OIDC issuer, protected source commit, complete Sigstore bundle, and offline transparency proof.

The database publisher has one short-lived Azure identity for only those create-only scanner-data paths.

It cannot write a Firecall verdict, security status, session object, registry, Gallery version, or Terraform state.

After immutable publication, it acquires the lease on firecall-scanner-status/trivy-db-v2-current.json.

It validates the current generation, ETag, prior digest, schema version, and selected database version.

It conditionally advances the pointer by one generation through If-Match while the lease remains valid.

The pointer contains only generation, prior digest, immutable statement path, OCI digest, Blob digest, database version, publication time, and signer identity.

It rereads the pointer and immutable target before lease release.

The rescan reads that exact pointer without listing and verifies the target signature, bundle, digests, version, and freshness.

A missing target, stale pointer, generation rollback, version rollback, bad prior digest, ETag race, or replay blocks scanning.

The publisher cannot change firecall-security-status/current.json or a rescan verdict.

Treat that publisher as a separate vulnerability-data trust holder and require owner acceptance before activation.

Use ClamAV main.cvd, daily.cvd, and bytecode.cvd as the malware database formats.

Accept only full CVD containers whose Cisco Talos digital signatures verify with the pinned ClamAV engine and embedded publisher key.

Do not accept an unsigned rule file, local YARA file, CLD file, failed CVD signature, or unknown builder.

The same scanner-database publisher retrieves all three CVD files from two configured official distribution endpoints.

It requires byte equality by digest, valid CVD headers, valid embedded signatures, increasing versions, and bounded file sizes.

Store each exact file create-once under firecall-scanner-data/clamav/<name>/<version>/<sha256>.cvd.

Create one keyless-signed bundle statement under firecall-scanner-data/clamav/bundles/<generation>/<statement_sha256>.json.

The statement binds all three names, versions, build times, byte lengths, digests, signature results, endpoints, source commit, and signer.

Use the same separate scanner-publisher certificate identity, but a distinct statement type and path.

After immutable reread, lease firecall-scanner-status/clamav-current.json and advance it by one generation through If-Match.

That pointer binds prior digest, exact bundle path, statement digest, three CVD paths and versions, publication time, and signer identity.

Reread the pointer and all four immutable objects before lease release.

The rescan reads both scanner pointers without listing and verifies both complete chains before artifact scanning.

A missing CVD, partial bundle, stale daily build, version rollback, prior-digest mismatch, pointer race, or signer mismatch blocks scanning.

Keep secret-detection rules in the reviewed scanner lock and fetch no external secret-rule pack.

Record every database digest, schema or CVD version, builder, signer identity, verification chain, publication time, and retrieval time.

Require Trivy DB publication within 12 hours and daily.cvd build time within 48 hours.

Require every database version to meet checked-in minimums and never regress from the prior signed current verdict.

Reject an unsigned, wrong-signer, replayed, stale, malformed, or checksum-mismatched database before scanning.

Run scanners without executing artifact content, with no Docker socket, bounded files, bounded archives, and fixed memory and time limits.

Apply the publication thresholds from this section to each rescan.

A critical vulnerability, secret, or malware finding always blocks.

A high vulnerability blocks unless one exact unexpired reviewed exception matches the artifact digest and finding identifier.

An exception cannot cover a critical, secret, malware, unknown artifact, or scanner failure.

The rescan produces one canonical statement with:

  • Schema version and monotonic generation.
  • Prior current-record digest.
  • Protected source commit and workflow run identifiers.
  • Exact scanner and database digests.
  • Required-subject-set digest.
  • One result and finding digest for every required subject.
  • Exact accepted exception identifiers and expiries.
  • Overall pass or blocked verdict.
  • Azure Storage response time as evaluated_at.
  • valid_until no later than 24 hours after evaluated_at.

Sign the canonical statement with the exact keyless rescan identity and include its complete offline Sigstore bundle.

Create it at firecall-security-evidence/rescans/<scan_epoch>/<statement_sha256>.json with If-None-Match: *.

Reread the immutable object and verify its digest and signature before changing current status.

Acquire the lease on the pre-created firecall-security-status/current.json object.

Require its current ETag, generation, and prior digest before writing generation plus one.

The current record contains only the generation, prior digest, evidence path, statement digest, verdict, subject-set digest, and validity times.

Write it through If-Match while the lease remains valid, then reread and release the lease.

Append one immutable safe audit record under security-rescan/.

Do not put finding descriptions, package paths, secrets, or complete scanner output in GitHub logs or Firecall audit.

The broker checks current status before it accepts an open or retry request.

It reads the exact evidence path and verifies the statement, Sigstore bundle, subject set, verdict, and time offline.

It uses the Azure Storage response Date header for the freshness comparison.

The control runner repeats that complete check after session locking and before capacity reservation or Azure login.

A missing, malformed, unsigned, stale, incomplete, or blocked verdict denies new open and retry operations.

The gate does not block close, sweep, break glass, or zero-resource proof.

Existing sessions continue by default, and the current verdict lists every affected active profile subject.

For an urgent finding, an owner disables creation and uses reduction-only close for affected sessions.

The rescan identity cannot close a session or change protected configuration.

A blocked subject enters logical quarantine.

Image pin, profile validation, and reservation reject a quarantined subject.

Quarantine does not delete an image, change Policy, or mutate a running workload.

Recover with a new digest, a new Gallery version, or one reviewed high-severity exception.

Run the manual rescan through its event-specific workflow and runtime guard after the protected change merges.

Recovery publishes a new immutable statement and advances current status.

Never rewrite or delete the blocked evidence.

If a scheduled scan fails before publication, keep the prior current record.

That record becomes stale after its signed validity and then blocks new sessions.

If scanning completes with an input, scanner, or database error, publish a signed blocked verdict.

34. Failure behavior

Failure Required behavior
Cloudflare JWT invalid Deny request
Cloudflare JWKS unavailable with no valid cache Deny request
CSRF or Origin check fails Deny state change
GitHub App unavailable Reject new control request; existing sessions continue
Key Vault request signing unavailable Block open; revoke authorized close access, then retry and alert; sweep and break glass remain available
Workflow dispatch fails Create no reservation
Request receipt storage unreadable Block create and pending-session views
Rate-limit storage unreadable Block create; allow authorized close through a safe reduction path
Envelope invalid Fail before Azure login
Status storage unreadable Block normal create, list, close, and join; permit reduction-only break glass
Close-revocation read unreadable Deny new data access and close every session stream within five seconds
Collaborator invitation read unreadable or absent Deny new collaborator access and close that invitation's streams within five seconds
Capacity storage unreadable Block create
Fourth session Return capacity-full message
Release signature, identity, or provenance invalid Block the pin pull request and deployment
VM evidence invalid or mismatched Fail before capacity reservation
Gallery seal missing, weak, changed, or mismatched Fail before capacity reservation
Publication scan threshold fails or exception expires Block image publication
Current rescan is missing, stale, incomplete, or blocked Block broker create, retry, and runner reservation; keep close and sweep available
Mutable Firecall image reaches admission Reject the Pod
Terraform open fails Run cleanup and verify zero resources
Readiness fails Run cleanup and verify zero resources
Readiness worker unavailable Remain verifying; never publish ready
Failed-open cleanup succeeds Publish failed and release capacity
Authorized early-close cleanup succeeds Publish closed and release capacity
Expiry cleanup succeeds Publish expired and release capacity
Cleanup fails or is unproved Publish rollback_failed and retain capacity
Proxy loses status read Close existing streams within 15 seconds
Close-revocation write fails Dispatch close; deny access when closing appears; report delayed denial
Close dispatch fails after revocation Keep access denied, retry three times, and alert for cleanup
Audit sink fails during open Reject the open before capacity reservation
Audit sink fails during close Continue reduction; retain a retryable workflow record
Proxy cannot reach box Show reconnect state; do not bypass mTLS
Controlled egress unavailable Block affected public control calls; never permit direct public fallback
Box certificate changes Refuse connection
Certificate rotation fails Keep current valid leaf, alert before expiry, and block new sessions before unsafe expiry
Close fails Retain rollback_failed and alert after proof
Schedule runs late Proxy and box still enforce wall-clock expiry
Browser network drops Preserve tmux; offer reconnect
Known session is not ready Serve only the trusted lifecycle page

User messages must be safe and actionable.

Show Retry this request only for a confirmed pre-reservation terminal failure.

Show Retry as a new session only after zero-resource cleanup proof.

Examples:

Condition Message Next action
Capacity full All 3 dev sessions are in use. Try again after one closes. Return to dashboard
Status read failure Firecall cannot read session state. It blocked this request. Retry status read
Network loss The terminal connection stopped. Your shell continues on the box. Reconnect
Expired This session expired. Its browser access is closed. Return to dashboard
Dispatch failed Firecall could not start control. No slot was reserved. Retry this request
Control failed Control failed before capacity. No slot was reserved. Retry this request
Control canceled Control stopped before capacity. No slot was reserved. Retry this request
Request rejected Firecall rejected this request safely. Review the reason. Create a new request
Build failed The debug box failed readiness. Firecall removed it. Retry as a new session
Rollback failed Firecall could not prove cleanup. Platform support must inspect it. Copy support reference
Forbidden namespace This profile does not permit namespace <name>. Select an allowed namespace

35. Accessibility and responsive behavior

Target WCAG 2.2 AA.

Provide:

  • Full keyboard navigation.
  • Visible focus.
  • Light, dark, and high-contrast themes.
  • Terminal font sizes from 12 to 24 pixels.
  • Reduced-motion support.
  • Screen-reader lifecycle announcements.
  • A non-virtualized accessible logs mode.
  • A terminal shortcut that returns keyboard focus to the toolbar.
  • Touch targets of at least 44 pixels.
  • Mobile dashboard, status, logs, events, and close flows.
  • An accessory key bar for Ctrl, Esc, Tab, and arrows.

Show Desktop recommended for k9s on small screens.

Never hide expiry or close controls on mobile.

Enable xterm.js screenReaderMode through an explicit terminal setting.

Give every terminal tab and toolbar control an accessible name.

Ctrl+Backquote moves focus from xterm to the terminal toolbar.

The next Tab key moves through normal workspace controls.

After a dialog closes, return focus to the control that opened it.

Do not send terminal output to a global live region.

Test screen-reader mode with NVDA and Firefox on Windows.

Test it with VoiceOver and Safari on macOS.

Support the terminal on mobile widths of 360 pixels or more.

Provide the Ctrl, Esc, Tab, arrow, and keyboard-dismiss accessory controls.

Keep the Desktop recommended warning for full-screen k9s.

36. Tests

36.1 Unit and contract tests

  • JWT claim validation.
  • Access aud accepts only one bounded ASCII string inside an array of length one.
  • Scalar, empty, duplicate, extra, nested, object, null, non-string, and cross-host audiences fail.
  • Exact email-domain validation.
  • ASCII-only email validation.
  • Extra-@ and Unicode lookalike email rejection.
  • Identity display escaping in every HTML view.
  • Service-token rejection.
  • Request signature verification.
  • Compact request JWS accepts only exact ES256, known kid, and firecall-request+jws;v=1 protected headers.
  • JSON serialization, unprotected headers, multiple signatures, none, crit, b64=false, and remote-key headers fail.
  • Padded, noncanonical, duplicate-key, oversized, extra-header, and extra-segment request envelopes fail.
  • An envelope has exactly 20 minutes of life, at most 60 seconds to dispatch, and at most 15 minutes in GitHub's queue.
  • A delayed workflow start, future issue time, excessive skew, or expired old-key envelope fails before Azure login.
  • A valid open envelope cannot authorize close, and a valid close envelope cannot authorize open.
  • Schema unknown-field rejection.
  • Profile digest validation.
  • An empty write-service subset resolves every selected service as read.
  • A write-service name must be selected, valid, and unique.
  • Service access and effective namespace access change scope_sha256.
  • Write wins when read and write services share one namespace.
  • Session profiles accept only the fixed Cluster User, Reader, and Writer role IDs.
  • Status transition validation.
  • Terminal status rejects missing proof fields or cleanup_verified=false.
  • One terminal write contains state, reason, proof digest, proof time, and cleanup verification.
  • ETag conflict handling.
  • Session ID validation.
  • Idempotency derivation.
  • Namespace validation.
  • SigNoz period-form validation.
  • Header stripping.
  • Certificate pin checks.
  • Log escaping.
  • Preview target validation.
  • Current and previous protocol compatibility.
  • Close-envelope actor and target validation.
  • Capacity counts occupied ledger slots only.
  • A capacity-ledger write without its active lease identifier fails at Azure Storage.
  • The normal control identity cannot delete the capacity-ledger Blob or container.
  • A stale capacity-ledger ETag or operation epoch fails.
  • An ambiguous ledger update is reread before any retry can select another slot.
  • Every reserved-state failure transition.
  • Every audit event variant rejects missing required fields and unknown fields.
  • A pre-authentication denial serializes with nullable identity and session fields.
  • An unverified token claim never enters an identity audit field.
  • RFC 8785 canonicalization produces the expected audit path digest.
  • Session-bootstrap serialization produces one canonical payload, bounded envelope, and stable digest across Go and Terraform fixtures.
  • Duplicate, unknown, noncanonical, oversized, invalid-UTF-8, padded-base64url, and trailing envelope data fail before planning.

36.2 Session lifecycle tests

  • A verified human creates a session without local tools.
  • The verified email cannot be changed.
  • A duplicate form submission creates one occupied ledger slot.
  • Duplicate or ambiguous dispatches still create one occupied slot and one session.
  • A delayed duplicate after ready exits without Azure or capacity changes.
  • A delayed duplicate after close and slot release cannot recreate the session.
  • Deleting retained status still leaves the permanent tombstone replay denial.
  • A mismatched replay tombstone fails closed and alerts.
  • A stale accepted receipt recovers through the lease-backed dispatcher.
  • A refresh before runner startup preserves the authorized pending page.
  • A dispatch failure creates no reservation and uses bounded lease-backed transport retries.
  • Pre-reservation rejection, failure, and cancellation never remain dispatched.
  • A pre-reservation retry keeps one request and session identifier.
  • Retry fails when a reservation, runner status, or close revocation exists.
  • Runner status takes precedence exactly when reserved appears.
  • After Classic open stops, three concurrent v2 opens can build in parallel.
  • After the v2 limit becomes three, a fourth concurrent open fails.
  • Two simultaneous opens for one subject through different broker replicas create exactly one occupied slot.
  • Two simultaneous opens for two different subjects can each create one occupied slot.
  • Lease loss, ETag conflict, and an ambiguous response cannot create a second slot for the same subject.
  • A state-list error blocks creation.
  • A capacity-ledger read, schema, slot-count, or ETag error blocks creation.
  • A capacity-limit or protected configuration-digest mismatch blocks creation.
  • ready appears only after all probes.
  • The identity phase creates no VM resource, and the compute phase accepts only the exact signed bootstrap-envelope digest.
  • Revocation between the two phases removes the identity and creates no VM.
  • closed appears only after zero-resource proof.
  • expired appears only after zero-resource proof.
  • failed appears only for failed-open cleanup with zero-resource proof.
  • The root-owned expiry timer stops access and powers off the box on time.
  • The unprivileged user cannot change the expiry timer.
  • rollback_failed retains capacity.
  • A normal user lists only their sessions.
  • A platform owner lists all sessions.
  • A collaborator cannot list or close another session.
  • A collaborator exchanges the fragment capability before any scope appears.
  • Capability exchange alone cannot use any data-plane API or stream.
  • A requester or platform owner gets scope through ownership without a capability.
  • Every requester, owner, and collaborator confirms full scope before terminal attachment.
  • An unacknowledged requester or platform owner cannot use a data-plane API or stream.
  • An unacknowledged collaborator cannot use any data-plane API or stream.
  • A join acknowledgement fails for a different subject key, session, profile digest, invitation, or expiry.
  • A collaborator with only a session identifier cannot view scope or join.
  • A join capability enters HTTP only in the join-exchange POST body after fragment extraction.
  • It never enters an HTTP request target, Referrer header, storage object, or log.
  • A revoked invitation closes collaborator streams within five seconds.
  • Revoking one invitation leaves a different active invitation connected.
  • Removing one platform owner or changing the owner-list revision closes that owner's active terminal within five seconds.
  • The requester and an unrelated current platform owner remain connected after that owner removal.
  • Every workspace HTTP route denies a stale requester, owner, or invitation authorization before agent contact.
  • An unreadable or absent invitation closes only its collaborator streams within five seconds.
  • A ninth active or thirty-third total invitation fails without a Blob write.
  • Concurrent mints across both replicas cannot exceed the active or total invitation cap.
  • Invitation mint and exchange subject-key and session rate limits return bounded 429 responses.
  • An invitation expires after two hours or session expiry, whichever occurs first.
  • Terminal status denies all invitations immediately and removes the index within 30 minutes.
  • Every workspace JSON route rejects its exact limit plus one byte for declared and chunked bodies.
  • Unknown, duplicate, trailing, malformed, and invalid-UTF-8 JSON fields fail without side effects.
  • An unreadable close-revocation read closes requester, owner, and collaborator streams within five seconds.
  • A cached invitation or close-marker absence never authorizes beyond five seconds.
  • Two subject keys racing one unbound capability produce one subject-key binding and one denial.
  • The same bound subject key can exchange its invitation on a new browser until revocation.
  • A capability hash from one session cannot authorize another session.
  • Close requested appears after revocation creation.
  • Close revocation denies access before an ambiguous or delayed dispatch finishes.
  • Duplicate close workflows remain idempotent.
  • Exact expiry derives Access expired · cleanup pending.
  • Building, failed, closing, closed, and expired URLs show trusted lifecycle pages.

36.3 Terminal tests

  • The terminal opens with Kubernetes access ready.
  • No connect command is necessary.
  • Refresh reconnects to the same tmux state.
  • Network interruption preserves shell state.
  • Closing a browser tab does not kill the shell.
  • Ending one terminal kills only one window.
  • Tabs survive refresh.
  • k9s starts in the selected namespace.
  • Open pod shell creates a local shell and preloads one fixed kubectl exec command without a newline.
  • No Pod shell command runs until the user presses Enter.
  • Each terminal keeps its original namespace after top-selector changes.
  • Bare kubectl get pods uses the tab namespace in two different tabs.
  • No generated command contains -A.
  • Access-token expiry closes the WebSocket.
  • Session expiry closes the WebSocket.
  • Session close closes every WebSocket.
  • Malformed frames do not crash services.
  • Zero-length frames do not crash services.
  • Oversized frames fail safely.
  • Slow clients cannot exhaust proxy memory.
  • Optional SSH details are visible only to the requester and platform owner.
  • The current-address shortcut disables safely for an IPv6-only request.
  • One workspace page creates exactly one presence connection.
  • One actor can open two presence connections, and its third fails without affecting another actor.
  • Eight session presence connections pass; the ninth fails; the final-slot rule preserves one new actor.
  • Presence slots release on clean close and after the 30-second heartbeat timeout.
  • Presence saturation does not consume another actor's terminal, log, or preview budget.
  • Terminal and log streams do not change the presence count.
  • Connections through both proxy replicas cannot exceed the box-agent session limits.
  • A shell process storm reaches TasksMax without stopping the agent or expiry timer.
  • Shell memory exhaustion reaches the user-slice limit without stopping the agent or expiry timer.
  • User disk exhaustion reaches its home filesystem bound without filling the system reserve.
  • Agent and expiry services survive the tested interactive CPU and I/O load.
  • firecall-sessiond launches every child with the firecall UID, GID, and no capability.
  • Sessiond rejects a wrong peer UID, unknown operation, unlisted namespace, and every caller-supplied command, path, argument, environment, or descriptor.
  • Fuzzed sessiond frames cannot create a process or crash the supervisor.
  • The agent retains NoNewPrivileges and cannot use sudo, setuid, Polkit, or systemd to elevate.

36.4 Kubernetes view tests

  • Every request names one allowed namespace.
  • An outside namespace returns 403.
  • Workloads show health and restart counts.
  • Secret data never appears in structured responses.
  • Non-Secret workload credentials do not appear in structured responses.
  • Arbitrary YAML, commands, arguments, annotations, and environment values never appear.
  • Current logs work.
  • Previous logs work.
  • Workload log and shell actions require an explicit Pod and container.
  • The Pod shell renderer rejects malformed, noncanonical, option-like, or control-byte names.
  • The rendered Pod shell command contains only the selected namespace, Pod, container, and fixed /bin/sh target.
  • No action silently selects a Deployment replica.
  • Follow mode stops on close.
  • Follow mode stops on expiry.
  • Log limits bound memory.
  • List pagination, cursor binding, response limits, and truncation markers work.
  • A slow log consumer receives slow_consumer before buffer growth.
  • The slow-consumer view retains its received tail and offers a fresh Resume action.
  • Warning event filters work.
  • Every generated command has the exact namespace.
  • Allowed exec, attach, logs, and port-forward operations work.
  • An unlisted namespace fails.
  • Cluster-scope writes fail.

36.5 Trust-boundary tests

  • Forged Cloudflare headers fail on direct origin access.
  • A wrong portal audience fails.
  • A wrong workspace audience fails.
  • A missing subject fails.
  • A non-OpenMined email fails.
  • A service identity fails.
  • A wrong proxy client certificate fails.
  • A valid CA certificate with the wrong URI identity fails.
  • A valid CA certificate with the wrong SPKI fingerprint fails.
  • A root with wrong subject, curve, constraints, usage, validity, version ID, or digest fails publication.
  • A CA-root workflow token from another workflow, repository, owner, ref, or pull request fails.
  • The CA-root identity cannot create a fifth key name, export a key, read a secret, or delete a root key.
  • The CA-root identity cannot list, overwrite, or delete public-root objects.
  • Correct root publication always sends If-None-Match: *.
  • The legal-held root container rejects a reused path even when a compromised publisher omits that header.
  • Root publication rereads and verifies each immutable object before source proposal.
  • The control runner rejects an unsigned, wrongly signed, stale, or digest-mismatched root lock.
  • A client-root switch waits for an image with both roots and zero old-root sessions.
  • An origin-root switch waits for both connectors to report the new bundle digest.
  • Origin-root addition and removal use a new immutable digest-named ConfigMap.
  • Argo applies that ConfigMap before the Deployment and keeps the old object through rollout.
  • Old connector Pods stay ready while new Pods mount and prove the new root bundle.
  • A wrong mounted bundle digest, broker TLS failure, proxy TLS failure, or tunnel disconnect makes each connector Pod unready.
  • The origin verifier sees two current ready Pods and cannot read Secrets, logs, exec, attach, port-forward, or patch.
  • The CA-root Azure identity cannot get credentials for another cluster.
  • Its Kubernetes identity cannot read another namespace or change a Firecall object.
  • Switching an origin leaf before connector trust fails its qualification test.
  • Removing a root while one allowed image or serving leaf needs it fails validation.
  • Current and previous proxy leaves work during rotation.
  • Every certificate consumer hot-reloads a verified inactive slot without restart or outage.
  • Argo self-heal does not revert certificate data, active epochs, or the tunnel rollout epoch.
  • Argo excludes and does not own the three cluster-owner-seeded coordination Leases.
  • Argo self-heals one changed unlisted field while every allowed runtime field remains unchanged.
  • Each ignoreDifferences rule matches one exact object and only its listed JSON pointers.
  • One tunnel-token ConfigMap and epoch-annotation rotation stays Synced through an Argo self-heal cycle.
  • The rotator cannot create a Secret, patch a Deployment, or change an unlisted Secret or ConfigMap.
  • A failed new-leaf probe leaves the prior certificate epoch active.
  • The runner rejects a public proxy leaf with the wrong CA, URI, digest, epoch, or remaining validity.
  • A concurrent leaf rotation gives the runner one complete old or new trust record, never a mixed record.
  • The proxy selects the private leaf that matches each session's recorded SPKI.
  • An expired proxy leaf fails.
  • A wrong box certificate fingerprint fails.
  • A status private address outside 10.240.0.0/27 fails.
  • Direct Internet access to box 8443 fails.
  • Same-subnet east-west access to box 8443 fails.
  • Another AKS workload with a node-subnet source still fails without mutual TLS.
  • Proxy-to-box traffic uses the peering and never the session public address.
  • Drift in either peering, subnet, CNI, or route fact blocks a new session.
  • Browser and Cloudflare credentials never reach the box.
  • Broker cannot read state or reach a box.
  • Broker can list only lifecycle status under sessions/ and can read only the exact capacity ledger.
  • Broker status writes, capacity writes, leases, sibling lists, and root lists fail.
  • Proxy cannot dispatch workflows.
  • Proxy cannot write lifecycle status or Terraform state.
  • Offboarding with no browser stream terminates an SSH-only session and detached tmux processes through the internal revoke route.
  • Broker and proxy restart after the marker write resume the same idempotent delivery operation.
  • Duplicate delivery returns the same signed receipt; a different nonce or marker binding fails.
  • A forged proxy delivery receipt demonstrates the documented proxy-compromise residual and fails with the wrong chain, SPKI, or request binding.
  • Missing marker, wrong stable subject key, wrong session, or unproved agent receipt prevents delivery success.
  • A stale or forged readiness nonce fails.
  • A worker-written result without the approved proxy signature fails.
  • A valid proxy signature with mismatched route facts fails.
  • Only the probe-worker mTLS identity reaches the internal probe Service.
  • Current and previous probe-worker leaves reach the internal probe Service.
  • A proxy leaf or wrong probe-worker SPKI fails on the internal probe Service.
  • After E1 trust reduction, pull-request code receives no Azure credential.
  • A main-branch token from the wrong workflow cannot use control, break-glass, image, policy, or Cloudflare identities.
  • A correct workflow token from a pull-request ref cannot use those identities.
  • A token without job_workflow_ref cannot use any privileged identity.
  • A token with the wrong numeric repository or owner identifier cannot use any privileged identity.
  • The control identity accepts the default name-based sub only with both exact numeric ID claims.
  • A renamed or transferred repository fails the combined name and numeric claim checks.
  • The infrastructure identities reject the control-repository subject and numeric IDs.
  • Each normal control credential accepts only its exact reusable privileged workflow.
  • The break-glass and image credentials reject every other reusable workflow.
  • The checked-in flexible-credential JSON exactly matches every live Microsoft Graph object.
  • An unexpected, changed, duplicate, or unreadable flexible credential blocks activation.
  • A box server private key never appears in Terraform state.
  • Terraform state and Azure custom data contain only the declared public bootstrap fields and offline signature bundle.
  • A token, email, subject key, capability, SSH key, command, environment value, or private material fails the bootstrap schema.
  • A wrong Sigstore issuer, repository, owner ID, workflow, ref, commit, source, transparency time, or payload digest stops every interactive service.
  • A changed session, profile, namespace, expiry, identity, assignment, AKS contract, CA, or image value fails local bootstrap.
  • A missing or changed proxy SPKI, client-root bundle digest, root-lock digest, or root-lock generation fails local bootstrap.
  • The box agent rejects a proxy leaf that chains to a different root or has a different SPKI despite valid time and URI identity.
  • The finalizer accepts the signed session envelope and Azure provisioning key only from their fixed paths.
  • The finalizer reads immutable verification input only from the image-pinned bootstrap-material path.
  • The finalizer writes only its fixed root-owned documents, trust files, generated private material, optional SSH key, kubeconfigs, and finalization marker.
  • Successful finalization removes the cloud-init source and reports only safe digests.
  • The interactive shell user cannot read or replace agent keys, proxy pins, or expiry configuration.
  • The interactive shell user cannot authenticate to the local agent endpoint.
  • The interactive shell user cannot open the sessiond control socket.
  • The interactive shell user cannot reach Azure Instance Metadata Service.
  • The interactive kubeconfig cannot authenticate to AKS outside its source box.
  • The kube proxy never returns an upstream Azure token in responses, files, environments, errors, or logs.
  • The kube proxy supports Kubernetes exec, attach, and port-forward upgrades through loopback.
  • The expiry timer stops the kube proxy before terminal shutdown.
  • A compromised agent cannot ask sessiond to create a root or arbitrary process.
  • The agent cannot write outside its two approved runtime paths.
  • Shared-key and anonymous Blob access fail.
  • A namespace outside firecall and firecall-edge cannot use the Firecall External Secrets store.
  • Firecall application ServiceAccounts cannot list or read Kubernetes Secrets.
  • The External Secrets identity can read Firecall secret values but cannot export Key Vault private keys.
  • An identity without an exact Firecall Key Vault role cannot read, sign, or publish a secret.
  • A plan that removes the Firecall storage account, listed container, or Key Vault fails prevent_destroy.
  • Fixed-key inventory drift or any attempted operational-key delete fails before a live mutation.
  • A live storage-account or Key Vault delete fails its exact CanNotDelete lock.
  • The standing identity cannot delete either parent, either lock, or a durable container.
  • A Blob data-plane delete still depends on exact roles and immutability, not the ARM lock.
  • Every signing identity fails decrypt, unwrap, export, delete, backup, restore, policy, and role-management operations.
  • Primary-to-secondary secret rotation completes without exposing a value in environment variables.
  • Broker cannot write lifecycle status.
  • Proxy cannot write probes or lifecycle status.
  • Probe worker cannot write lifecycle status.
  • Broker, proxy, probe worker, and rotator direct public HTTPS bypass fails.
  • Every required approved public hostname works through the controlled egress gateway.
  • Each component can reach only its assigned gateway listener and destination set.
  • Every credentialed component can query only its exact internal DNS-name set and query types.
  • Arbitrary labels, TXT, TCP-policy bypass, search suffixes, EDNS option data, and fragmented DNS requests fail.
  • Real Go resolver requests with an empty EDNS(0) record work over UDP and TCP; upstream queries contain no additional record.
  • The DNS proxy creates a new upstream packet and never preserves caller identifiers or extra sections.
  • cloudflared resolves only the reviewed origin and tunnel-discovery names through its separate listener.
  • Only the credentialless egress gateway and DNS policy proxy can query the cluster resolver directly.
  • A protected egress-client label on the wrong ServiceAccount, namespace, or image is rejected at admission.
  • A Pod without the protected label cannot reach any gateway listener.
  • An unapproved hostname, IP literal, non-443 port, Unicode ambiguity, and DNS result in any excluded range fail.
  • Gateway DNS, dial, ClientHello, ServerHello, idle, lifetime, byte, buffer, source, listener, and replica limit-plus-one tests fail safely.
  • Every gateway counter releases after denial, close, timeout, cancellation, and replica shutdown.
  • DNS query bytes, name length, question count, response bytes, TTL, cache, rate, source concurrency, listener concurrency, and timeout limits pass at limit and fail at limit plus one.
  • Agent pre-authentication handshake, source rate, global rate, byte, deadline, idle, authenticated, and proxy-replica limits release every counter.
  • Kube-proxy request rate, nonstreaming concurrency, upgrade totals, per-upgrade kind, buffer, body, idle, and lifetime limits pass at limit and fail at limit plus one.
  • Correct callers validate the destination certificate end to end.
  • A same-edge encrypted-authority test demonstrates the documented application-layer routing residual.
  • An approved endpoint can receive encrypted content and timing, and the threat test records that covert-channel residual.
  • The egress gateway cannot reach the Kubernetes API, box subnet, Pod CIDR, Service CIDR, VNet, or metadata service.
  • The egress gateway has no usable ServiceAccount, Azure, storage, GitHub, Cloudflare, or Key Vault credential.
  • cloudflared reaches only current reviewed Cloudflare Tunnel address ranges and required ports.
  • Routine tunnel-token rotation keeps one connector ready and leaves two refreshed connectors healthy.
  • A new Cloudflare API token passes the exact permission map and permitted and forbidden canaries before selection.
  • The protected edge apply succeeds with the selected new API-token slot before old-token revocation.
  • The old Cloudflare API token fails verification immediately after owner revocation.
  • Emergency API-token rotation revokes the suspected token first and never restores it.
  • An old tunnel token cannot start a connection after rotation.
  • Tunnel rotation leaves a zero-change Terraform plan and matching Key Vault version.
  • A rotation crash after the API call recovers state without a second token refresh.
  • A concurrent edge apply or rotation fails on the shared operation lease.
  • Normal developers cannot read edge state. Only the protected edge workflow and an exact bounded owner operation receive that development authority.
  • The active JIT role reaches only the exact state key and keyenv: list prefix for one hour.
  • Expiry, role revocation, and Azure-token expiry end JIT state access.
  • Emergency connection deletion permits only refreshed Pods to reconnect.
  • A tunnel Secret change alone does not claim or start a rollout.
  • A newer successful ExternalSecret refresh plus the fixed epoch ConfigMap starts one rollout.
  • The edge identity cannot read Secrets, change another ConfigMap, or change a Deployment field except the exact epoch annotation.
  • Admission rejects an edge patch to an image, command, environment, volume, identity, label, replica count, or second annotation.
  • Disallowed AKS roles and namespaces fail role assignment.
  • The disposable no-credential identity accepts only Cluster User at the exact cluster and Reader or Writer at each reviewed namespace.
  • An all-read request creates only Reader assignments.
  • A write-selected service creates Writer only for its effective write namespaces.
  • Write wins when one read service and one write service share a namespace.
  • Every other role, role-definition ID, parent scope, sibling namespace, and subscription scope fails that canary.
  • Removal appears in both Activity Log and immutable Firecall audit before the disposable identity is deleted.
  • A token issued before assignment removal remains usable only until its signed expiry and never beyond 60 minutes.
  • The assignment auditor matches every active session to its exact principal, role, scope, and deterministic GUID.
  • An unrelated principal, wrong role, wrong GUID, stale assignment, or missing assignment alerts within 15 minutes.
  • The assignment-auditor identity cannot write an assignment, status object, capacity value, or Azure resource.
  • Azure Policy denies wrong VM SKU, location, disk, subnet, public-IP SKU, exact image ID, resource type, or tags.
  • Policy accepts the exact session disk and managed identity names and types.
  • Policy denies a wrong disk SKU, size, name, identity name, or extra resource type.
  • Azure Policy denies a NIC without its exact deterministic NSG attachment.
  • The session role has only the four exact RG-scoped NSG, public-address, NIC, and identity linked actions.
  • Effective-permission canaries find those actions at the session resource group and nowhere else.
  • Policy requires one deterministic NIC and one deterministic user-assigned identity on each VM.
  • Policy requires one deterministic public address, one deterministic NSG, and one exact subnet on each NIC.
  • Every relationship Policy alias matches the pinned live resource type and path before assignment.
  • Attach, detach, add, replace, cross-session, cross-group, and wrong-type relationship changes fail.
  • Azure Policy denies a public 8443, broad SSH source, missing deny, changed priority, or extra NSG rule.
  • The session NSG denies Azure Load Balancer inbound traffic through priority 300, despite Azure's default priority-65001 allow rule.
  • Both UDP and TCP DNS succeed, and a missing TCP DNS rule fails policy qualification.
  • Azure WireServer guest-agent checks succeed through host nftables only on TCP 80 and 32526.
  • A missing or broader guest-firewall WireServer rule fails image qualification.
  • The NSG contract still contains both documentary WireServer entries and no broader custom entry.
  • Azure Policy denies inline and child-rule mutations after VM creation.
  • A sealed Gallery version outside the exact live allowlist is denied.
  • A control profile pin outside the exact live policy allowlist fails before capacity.
  • The policy-apply identity cannot change a resource, role, exemption, remediation, policy set, or non-Firecall assignment.
  • The Firecall node pool and its VM scale-set NICs use only the exact live aks-subnet resource ID.
  • A Firecall node address outside 10.224.0.0/16 blocks activation.
  • A Cloudflare drift plan detects a changed route, Access policy, audience, DNS record, or tunnel.
  • A compromised runner's remaining multi-VM spend risk is recorded and budget-alert delivery passes.
  • Activation fails when the BS-family quota cannot hold current use and the phase session limit.
  • Activation fails when the DDSv5-family quota cannot hold the fixed node and one surge node.
  • The unrelated-principal role-assignment limitation is an explicit accepted test result.
  • The normal control identity can use Run Command only on Firecall session VMs.
  • Run Command against every non-Firecall VM scope fails.
  • The accepted root-on-session and box-identity compromise boundary appears in canary evidence.
  • Normal identity removal takes effect no later than Access-token expiry.
  • Record the last managed-identity token iat and exp without recording its value.
  • Prove an issued token has no lifetime above the documented 24-hour ceiling.
  • Prove a deleted identity cannot mint another token.
  • Emergency Access revocation closes a new connection.
  • Every path-limited Blob assignment uses condition version 2.0 and the expected condition digest.
  • Exact Blob access succeeds while a sibling, adjacent suffix, encoded path, and container-root list fail.
  • A no-list identity fails Blob.List even when it supplies an allowed-looking prefix.
  • A list-capable identity succeeds only with its exact slash-terminated request prefix.
  • The generic Terraform workflows never discover, plan, or apply a Firecall v2 root.
  • A generic plan or apply identity cannot read any dedicated Firecall state key.
  • No generic identity receives inherited Blob data access from the state account or a parent scope.
  • Removing a dedicated state container from tfstate-bootstrap/prod fails prevent_destroy.
  • Two writers on one Firecall state key prove backend lease exclusion.
  • Every dedicated workflow completes real backend init and plan through its exact keyenv: list prefix.
  • A workspace-state write and a list with another prefix fail.
  • Standing, trust, and credential OIDC tokens fail from every wrong reusable workflow.
  • The trust bootstrap creates only the exact trust FIC and fixed definitions.
  • The trust workflow cannot change a role definition, policy definition, or unlisted assignment scope.
  • The standing workflow cannot change an existing AKS pool or unrelated VNet peering.
  • The standing identity reads and joins only the exact live AKS subnet during a real pool plan.
  • The sibling subnet, VNet parent, route table, NSG, peering, subnet write, and subnet delete canaries fail.
  • The same subnet read fails against its parent VNet and every sibling subnet.
  • Deleting the Firecall pool or AKS-side peering fails closed and requires owner bootstrap.
  • The policy identity cannot write the fixed definition or another assignment.
  • Builder, policy, control, and image-pin identities each read only their required exact Policy definition.
  • Each exact definition reader fails against the sibling definition and subscription-level definition listing.
  • Assignment reads succeed only at the required exact assignment resource; assignment listing fails.
  • A tenant Pod with the Firecall toleration, selector, affinity, or direct nodeName fails admission.
  • A tenant-created pods/binding request fails while the exact scheduler Binding succeeds.
  • A tenant-created core bindings request also fails.
  • Both Binding APIs require exact scheduler, Pod namespace, name, live UID, target.kind=Node, target.apiVersion=v1, and Firecall node.
  • A stale Pod UID, wrong target kind, wrong API version, or other-node tuple substitution fails.
  • An unavailable webhook, expired server certificate, invalid CA, or two-second timeout blocks only a Firecall-prefix Binding.
  • Other-node scheduling continues while the Firecall-prefix webhook fails closed.
  • The exact Konnectivity namespace, ServiceAccount, and labels reach the validator, while every other source receives denial.
  • kube-apiserver, host, remote-node, another kube-system Pod, every tenant, and every runtime Pod receive denial.
  • A selected Pod with literal sensitive environment data produces no validator response, log, metric, trace, or error containing that data.
  • Every approved kube-system DaemonSet tuple schedules, and a changed tuple fails admission.
  • The first Firecall agent-pool PUT fails until both admission bindings and negative probes pass live.
  • Each approved system Pod binds its reviewed spec image reference to its resolved imageID digest.
  • A missing, mutable-only, or changed resolved imageID blocks activation evidence.
  • Node capacity includes every approved system DaemonSet request.
  • External Secrets Operator recreates a deleted target Secret with creationPolicy: Owner.
  • Argo owns the ExternalSecret but never creates, deletes, or restores its target Secret data.
  • A stale certificate replica behind a healthy candidate Service blocks epoch selection.
  • Candidate proof maps every EndpointSlice target UID to one matching health response UID.
  • A foreign or stale EndpointSlice and an extra or missing Pod fail certificate rotation.
  • Root retirement blocks new leaves before bounded zero-use proof and later bundle removal.
  • A concurrent rotator cannot issue from a retiring_no_new_leaves slot.
  • The root workflow can read its App key but no other App key or Firecall secret.
  • The root-publication App cannot merge, approve, write control, or access another repository.
  • The Classic-observer App cannot write contents, dispatch, cancel, or access another repository.
  • Capacity reconcile rejects the wrong Classic installation ID, workflow ID, source digest, state-key set, or proof signature.
  • Capacity reconcile proves all three Classic states are retained, unlocked, and zero-resource.
  • Capacity reconcile checks deterministic Classic role assignments at cluster and namespace scopes.
  • A credential Job with a changed image, command, ServiceAccount, volume, or target fails admission.
  • Credential rotation reaches only its egress listener, Kubernetes API, and exact 9544 and 9545 credential-health ports.
  • Certificate rotation reaches only the exact 9444 and 9445 certificate-health ports.
  • Each rotator receives denial from the other rotator's health ports and routes.
  • Direct credential-rotator Internet, tenant, box, or unrelated Service access fails.
  • CONNECT to an allowed host fails when ClientHello SNI is absent, hidden, duplicate, malformed, or different.
  • CONNECT fails when the server selects TLS 1.2 or earlier.
  • A different visible SNI on an allowed multi-tenant IP fails.
  • A test proves that encrypted HTTP authority can differ after allowed SNI and records this accepted residual.
  • Exact CONNECT host, visible SNI, TLS 1.3 selection, and caller certificate validation succeed together.
  • Argo self-heal preserves only the listed HMAC and broker-key ConfigMap fields.
  • The first GitOps sync keeps all Firecall workloads at zero until every lower wave is live.
  • A missing default-deny policy, admission binding, RBAC object, or preflight signature blocks replica activation.
  • The replica activation diff changes no policy, network, RBAC, image, command, or trust object.
  • Normal HMAC rotation proves every proxy UID before active-key selection.
  • Each session token accepts only its fixed type, issuer, audience, purpose, version, key, and route schema.
  • Join, acknowledgement, cursor, preview-handoff, and preview-ack tokens fail every cross-purpose substitution.
  • Padded, duplicate-key, oversized, extra-segment, wrong-algorithm, and unknown-field tokens fail before authorization.
  • Emergency HMAC denial closes old-key streams and survives Argo self-heal.
  • Request-key phase one releases both leases before pull-request review.
  • Request-key phase two rejects a stale candidate, wrong control commit, or changed public-key digest.
  • Request-key reads, version listing, creation, update, sign, and verify fail against every other Key Vault key.
  • The rotator can enable or disable only versions of the exact CSRF and request keys.
  • One pending index entry blocks a second request-key candidate.
  • A crash before the index write recovers the tagged version through exact-key version listing.
  • An unindexed version without a valid candidate record becomes orphaned and never becomes active.
  • Blob root or candidate-prefix listing remains denied during activation and stale cleanup.
  • The legal-held trust-evidence container rejects candidate or status overwrite and delete attempts.
  • Every broker reloads one complete request-key selector resource version before phase two succeeds.
  • An Argo self-heal preserves only the exact request-key selector JSON pointers.
  • A selector absent from the attested control commit fails without changing the prior active key.
  • Stale request-key cleanup disables the unused version without selecting it.
  • The edge workflow sees the exact Deployment generation and all three replica counters at two.
  • Edge proof requires two current-revision, Ready, non-terminating Pod UIDs.
  • Exactly two current Ready Pods use the expected revision, epoch annotation, bundle digest, and passing local sidecar probe.
  • The complete Cloudflare inventory contains exactly two distinct healthy post-rollout connectors and no stale connector.
  • Qualification records that Pod UIDs and connector IDs have no cryptographic one-to-one mapping.
  • The connector Pod has no ServiceAccount token and cannot call the Kubernetes API.
  • The health sidecar gates Pod readiness through a local kubelet probe and cannot patch Pod status.
  • One old-revision Ready Pod cannot satisfy a new token rollout.
  • The edge workflow cannot read a Secret or watch another Deployment.
  • Direct child NSG rule write and delete fail Azure RBAC.
  • Parent NSG updates still enforce all eleven or twelve reviewed rules through policy.
  • Close removes optional SSH first and proves port 22 denial before deallocation.
  • The fixed revoke helper stops the kube proxy and agent before terminal cleanup.
  • Image-build Policy fixes one Standard_D2ds_v5 VM, NIC, and disk name and denies another name or SKU.
  • Two compliant fcimgip names can pass Policy, and the threat test records that spend residual.
  • The owner-only builder assignment has Default enforcement, no exclusion, and the protected source digest.
  • A missing, changed, or disabled builder assignment blocks the image workflow before Packer starts.
  • Another deployment wrapper can exist, but every nested wrong name, SKU, subnet, address, or rule fails Policy.
  • Azure Packer plugin v2.6.3 generates only reviewed fcimgdp<random5> and fcimgip<random5> live names in this configuration.
  • A successful Packer artifact records its actual deployment, address, and suffix before cleanup.
  • Failure cleanup finds deterministic and tagged temporary resources without a successful Packer artifact.
  • Idle cleanup deletes every ARM deployment wrapper in the dedicated image-build group after lease exclusion.
  • A pinned-source contract test and disposable build prove v2.6.3 uses the existing subnet and creates no temporary NSG.
  • Standard public-IP SSH succeeds only while the fixed subnet NSG contains the proved runner /32.
  • A broad source, extra rule, changed outbound rule, child-rule write, or fixed-NSG delete fails.
  • Cleanup restores the idle 192.0.2.1/32 rule before temporary resource deletion.
  • Standing and image workflows exclude each other through the exact image-build operation lease.
  • A failed workflow leaves an alert, and the scheduled reconciler restores the idle NSG after lease expiry.
  • A standing plan preserves only the runtime-owned inline rules and detects every other NSG drift.
  • Deployment validation and operation reads succeed only in the image-build group.
  • Deployment validation, operation reads, and every write fail in another resource group.
  • DDSv5 quota read succeeds while quota writes and sibling Compute writes fail.
  • The builder cannot reach AKS, Firecall, private, link-local, or metadata address ranges.
  • Builder cleanup leaves only the empty standing VNet, subnet, idle fixed NSG, Policy assignment, and operation lock.
  • The temporary marketplace builder records its pre-provisioning WireServer residual and has no managed identity or Firecall secret.
  • A VM from the captured image starts with the final UID-scoped IMDS and WireServer firewall rules already active.

36.6 Cleanup and failure tests

  • Force cloud-init failure.
  • Force managed-identity delay.
  • Force readiness failure.
  • Force Terraform partial apply.
  • Delay one accepted Azure operation until after the first empty inventory read.
  • Prove the delayed resource restarts quiescence and prevents tombstone, terminal status, and capacity release.
  • Delay one AKS role assignment until after the first absent GET and prove the 15-minute assignment interval restarts.
  • Return an ambiguous role-assignment response and prove direct GET and exact-scope listing resolve it before readiness.
  • A missing or nonterminal Azure operation journal entry forces rollback_failed after the two-hour cap.
  • Force Terraform destroy failure.
  • Force status ETag conflict.
  • Crash after tombstone creation and prove sweep publishes one complete terminal status.
  • Crash before the terminal ETag write and prove no partial terminal record exists.
  • Crash after terminal status but before ledger release and prove sweep releases only after exact proof validation.
  • A terminal ETag write never exposes state without cleanup proof fields.
  • Force capacity lease timeout.
  • Fail the first reserved status write after ledger commit and prove the slot stays occupied.
  • Race four creates and prove the leased ledger contains no duplicate or fourth occupied slot.
  • Bootstrap creates the missing ledger once and refuses to replace an existing ledger.
  • Bootstrap creates one immutable legal-held marker bound to the first ledger digest and ETag.
  • Bootstrap reads back the exact firecallbootstrap legal-hold tag before success.
  • The marker legal hold does not block a leased conditional ledger update.
  • The bootstrap identity can overwrite the exact ledger before revocation, and the threat test records that residual.
  • The same write fails after principal disablement, assignment removal, and token expiry.
  • Marker update and deletion fail while normal capacity writes succeed.
  • The one-time bootstrap identity cannot authenticate after its credential and roles are removed.
  • A missing ledger with an existing marker fails closed and never creates an empty ledger.
  • Recover a deleted ledger version and prove occupancy survives marker validation.
  • Change occupancy, reapply standing infrastructure, and prove ledger content and ETag do not change.
  • Terraform state contains the capacity container but no capacity-ledger Blob resource.
  • creation_enabled=false rejects opens with zero, one, two, and three occupied slots.
  • Portal, broker, and runner each fail closed on a config-to-ledger digest mismatch.
  • Capacity reconcile refuses limit three without exact Classic cutover evidence.
  • Capacity reconcile refuses limit three without the exact merged Classic open rejection.
  • An already-empty set of Classic states can publish the same signed proof through protected zero-proof mode.
  • zero-proof refuses an old rejection commit, any nonempty state, live role, live resource, or active lease on any Classic state.
  • The combined Classic workflow stays enabled and proves close and sweep still work.
  • Capacity reconcile refuses limit three for any queued, waiting, pending, requested, or in-progress Classic open run.
  • A pre-rejection Classic open run is canceled or drains before zero-resource proof.
  • Capacity reconcile repeats live Classic run and resource checks immediately before its ledger write.
  • Capacity reconcile changes gate, limit, digest, commit, and epoch in one leased write.
  • The capacity identity cannot change a session, status, role, network, policy, Key Vault, or tombstone.
  • Fail a close ledger update after zero-resource proof and prove the slot remains occupied until reconciliation.
  • Pause longer than 60 seconds and prove lease-loss abort and capacity recount.
  • Lose the session lease during apply and prove the stale epoch cannot publish status.
  • Force GitHub Actions delay.
  • Force proxy restart during a terminal.
  • Force cloudflared restart during a terminal.
  • Force status read loss during a terminal.
  • Verify sweep continues after one failed session.
  • Force close while open holds the session lease.
  • Close a queued open before reservation and prove that no resource reaches creation.
  • Close during Terraform and prove the watchdog interrupts open and cleanup wins.
  • Prove the VM deallocates before close waits for that lease.
  • Run break glass while lifecycle status is unavailable.
  • Delete the session identity first and prove deterministic role-assignment cleanup still succeeds.
  • Lose state and status, then derive and remove every assignment by deterministic UUID.
  • Prove break glass cannot create or expand resources.
  • Prove break glass denies an unlisted actor, self-approval, stale approval, and reused nonce.
  • Prove break glass rejects another issue, repository, run, commit, workflow, session, or reason digest.
  • Prove break glass rejects edited, deleted, duplicate, malformed, and post-expiry comments.
  • Race two consumers and prove exactly one immutable nonce object succeeds.
  • Prove a consumed nonce cannot authorize another run after a failed cleanup.
  • Prove its Azure identity denies Run Command, network join, and role-assignment write.
  • Prove audit objects cannot be overwritten or deleted by producers.
  • Fail open-audit append and prove no workflow dispatch or capacity reservation occurs.
  • Fail capacity-audit append and prove no VM starts before failed-open cleanup.
  • Fail close-audit append and prove cleanup continues with a durable pending marker.
  • Recover audit storage and prove broker and sweep commit their owned pending items idempotently.
  • One logical event with two payload digests alerts through the read-only verifier.
  • Pre-request identity denial passes the event-specific audit schema.
  • Prove Firecall runtime identities cannot read or list audit objects.
  • Prove the audit-verifier identity cannot create, overwrite, delete, or change any lifecycle fact.
  • Prove the audit-reader group cannot create, overwrite, or delete audit objects.
  • Prove retention never deletes an active or rollback_failed record.
  • Prove retention and lifecycle rules never delete a replay tombstone.
  • Prove audit lifecycle eligibility and overdue alerting in a disposable locked container without claiming a bounded deletion time.
  • Keep referenced scanner, security, node, access, tool-cache, release, and exception evidence beyond its minimum age.
  • Delete eligible unreferenced evidence only after two identical reference-graph reads 60 seconds apart.
  • A changed ETag, new reference, unreadable source, young object, active receipt, or unexpired WORM period stops the retention batch.
  • The retention identity cannot delete a current pointer, audit object, tombstone, bootstrap marker, status, state, request, trust object, or offboarding marker.
  • The event-specific retention workflow guards reject the wrong event, every rerun, and every caller override of container, path, age, or reference state.
  • A failed retention batch resumes only from its immutable operation ID and writes distinct intent and completion audit objects.
  • Prove tombstone legal-hold removal needs the explicit deletion authority.
  • Read back the exact firecallreplay tag and reject a hyphenated or otherwise invalid tag.
  • Restore a state and lifecycle object through Blob versioning and soft delete.
  • Deliver each required alert through the real notification path.
  • Notification verification pins the exact Action Group resource ID, SigNoz organization, contact point, policy, owner, ETags, and destination digest.
  • A missing, stale, wrong-target, failed, or unreceipted delivery blocks activation.
  • Firecall can read only the exact Action Group metadata and holds no SigNoz API key.
  • The notification verifier cannot change a receiver, contact point, policy, organization setting, dashboard, or alert rule.

36.7 Preview tests

  • Only allowed Services and Pods can create previews.
  • Arbitrary hosts and IP addresses fail.
  • ExternalName, selectorless, unmanaged, privileged, host-network, and host-path targets fail.
  • A stale Service, Pod, or EndpointSlice UID or resource version fails.
  • A selector mismatch or wrong EndpointSlice manager fails.
  • One mixed, external, unresolved, or unowned endpoint rejects the complete Service preview.
  • Safe unready endpoints remain unavailable; zero ready endpoints reject creation.
  • The user selects one exact ready backing Pod before a Service preview starts.
  • A changed Pod or EndpointSlice ends the forward and requires a new selection.
  • Creation and restore revalidate the complete target after kubectl reports readiness.
  • HTTP and HTTPS upstream protocol selection works.
  • Service HTTPS uses the exact Service DNS name for Host and TLS ServerName.
  • Direct Pod HTTPS fails.
  • A preview uses the separate domain.
  • The pinned Public Suffix List proves portal and preview effective registrable domains differ.
  • Two host strings under the same registrable domain fail configuration and runtime startup.
  • The proxy rejects a workspace-audience token on the preview domain.
  • Cookies do not cross the preview boundary.
  • Cloudflare credentials never reach preview content.
  • Application cookies and Set-Cookie never reach either side of the preview target.
  • JavaScript parent-domain cookie poisoning cannot affect sibling preview targets or Firecall acknowledgement cookies.
  • Application Authorization reaches only the selected preview target and never logs.
  • Close stops every preview.
  • Expiry stops every preview.
  • Idle previews stop.
  • Raw TCP forwarding is unavailable.
  • The trusted preview interstitial appears before application content.
  • A subject-key-bound handoff sets a preview-host acknowledgement cookie.
  • A preview without a valid handoff cannot reach application content.
  • The preview acknowledgement endpoint validates JSON, exact Origin, audience, subject key, target, and expiry.
  • The trusted interstitial removes the fragment through history.replaceState before reload.
  • Preview handoff values never appear in HTTP logs or Referrer headers.
  • The create form warns that cookies, login flows, and service workers are unsupported.
  • Application Authorization stays stripped until the preview-specific confirmation enables it.
  • Preview cards show target, state, idle countdown, safe errors, and working Open, Copy, and Stop actions.
  • A clean shutdown writes restart permission only after every preview process stops.
  • Restore consumes that one-use permission before it starts any preview process.
  • An agent restart revalidates and restores a safe preview or reports a recoverable error.
  • Oversized bodies, headers, WebSocket messages, and connection counts fail safely.
  • Per-actor, per-preview, and per-session HTTP rate and active-request limits hold across both proxy replicas.
  • The agent rejects a thirty-third active request for the session without opening another port-forward request.
  • A full bounded queue rejects its next request within one second.
  • Slow request bodies, slow response headers, stalled response bodies, and parallel requests release every counter after timeout.
  • One preview cannot reuse another preview's upstream connection pool.
  • CONNECT, TRACE, unsafe redirects, and client forwarding headers fail safely.
  • Application HTML receives no injected Firecall banner.
  • worker-src 'none' blocks service-worker registration.

36.8 Browser and accessibility tests

  • Chromium, Firefox, and WebKit pass core flows.
  • Portal, workspace, trusted interstitial, and untrusted preview responses match their exact header contracts.
  • An iframe, inline script, eval, external script, external connection, and service-worker registration fail under each applicable CSP.
  • Workspace and preview WebSockets succeed only to their exact current host.
  • A CSP violation reaches the bounded report endpoint without a blocked URL, source sample, identity, or secret.
  • Keyboard-only users complete create, join, terminal, logs, and close.
  • Join scope confirmation works with keyboard and screen reader.
  • The first-session quick-start card is readable and dismissible.
  • Automated accessibility checks report no serious issues.
  • Screen readers announce lifecycle changes.
  • Screen-reader mode does not announce every log line.
  • The interface works at 200 percent zoom.
  • Light, dark, and high-contrast themes meet contrast requirements.
  • Terminal font sizes 12 and 24 pixels keep controls usable.
  • Every interactive touch target measures at least 44 by 44 CSS pixels.
  • Non-terminal views work at 320 pixels.
  • The terminal and accessory controls work at 360 pixels in portrait and landscape.
  • Ctrl+Backquote moves focus out of xterm on every tested platform.
  • No Firecall shortcut replaces a documented browser or operating-system shortcut.
  • NVDA with Firefox and VoiceOver with Safari pass the terminal flow.
  • Reduced-motion mode removes status animation.

36.9 Load and soak tests

  • Qualification start fails outside the protected 12-hour owner-only epoch.
  • Drain start fails with a nonowner session, stale node verdict, existing load Pod, or wrong Pod contract.
  • Drain uses exactly 1800m CPU and a maximum eight-hour active deadline.
  • Sustain replaces the absent drain Pod and uses exactly 700m CPU with a maximum three-hour active deadline.
  • The workflow never creates both load modes and cannot update a mode in place.
  • Session CPU qualification uses only the fixed unprivileged command and stops by its signed three-hour deadline.
  • The one-hour suite starts only after all four authenticated credit balances stay below 15 for 15 minutes.
  • Preflight, rotation, certificate maintenance, and connector rollout fail while the node-load Pod exists.
  • Stop and cleanup remain available after lease loss, failed credit reads, deadline, cancellation, or partial session failure.
  • Cleanup deletes the load Pod, closes three sessions, clears all slots, and publishes zero-resource proof.
  • Any failed qualification target leaves normal creation disabled until a reviewed recovery merge.
  • Run three sessions with the maximum supported terminal and log streams.
  • Hold one workspace for the full eight-hour limit.
  • Restart each broker, proxy, readiness, and tunnel replica during active use.
  • Restart each egress-gateway replica and deny direct fallback during the rollout.
  • Prove reconnect works across proxy replicas.
  • Prove slow clients stay within memory limits.
  • Prove one noisy session cannot exhaust all proxy workers.
  • Prove the fourth reservation still fails during peak load.
  • Record CPU, memory, connection, and ready-latency results.

36.10 Supply-chain tests

  • The control bootstrap refuses a nonempty or wrong-ID repository.
  • The one direct seed contains no product code or credentialed workflow.
  • The first product change requires the configured pull-request protections.
  • The live settings check proves CODEOWNERS, review, status, conversation, force-push, deletion, and administrator controls.
  • A valid container signature, SBOM, provenance statement, and scan result pass offline verification.
  • A wrong issuer, repository, workflow, ref, source commit, or subject digest fails.
  • A missing or invalid Sigstore bundle fails.
  • An unpinned Terraform or Packer CLI, provider, plugin, checksum, signature, or verification key fails.
  • A missing, changed, or wrong-platform Terraform lock file fails.
  • A changed Packer required_plugins constraint, cached archive, checksum manifest, source, version, or isolated plugin inventory fails.
  • Packer starts with an empty mode-0700 plugin directory and lists only the cached Azure plugin at the exact version and checksum.
  • A packet capture proves Packer makes no plugin, package, tool, or update download during the image build.
  • Each tool-cache path equals its content digest and accepts only one create-once artifact and signed acquisition statement.
  • A moving URL, redirect change, wrong size, digest, signature, signer, key, platform, filename, media type, or lock digest fails acquisition.
  • The acquisition union fails for an omitted action, executable, OCI layer, package, provider, plugin, module, pnpm tarball, browser, font, rule, or scanner database.
  • Every acquisition batch digest and member resolves through the one canonical acquisition digest.
  • Pull-request acquisition gets no Azure credential, and privileged publication re-verifies bytes before Azure login.
  • Rootless BuildKit accepts only cached OCI objects, a local context, and network-none steps.
  • Go uses only the verified vendor tree, and pnpm uses only the verified offline store.
  • Packer APT uses only the signed local file: repository with --no-download.
  • Terraform uses only the local provider mirror, and Playwright uses only the verified browser tree.
  • Image, container, test, and scanner builds perform no DNS, registry, package, module, browser, rule, database, or update download.
  • execve and network traces fail on an unlisted executable, wrong phase, connection, or uncached input.
  • A cache overwrite, sibling write, list, delete, missing WORM policy, or wrong FIC fails.
  • An old cache object remains while any lock, image, release, scanner, or rollback record references it.
  • An old unreferenced cache object deletes only after WORM expiry and two identical reference-graph reads.
  • terraform init -lockfile=readonly succeeds without a repository change.
  • A mismatched gallery resource ID or VM evidence digest fails before reservation.
  • An unsealed Gallery version cannot enter a profile or reserve capacity.
  • The image builder cannot update or delete a sealed Gallery version or manage its lock.
  • A session VM can still deploy successfully from the sealed ReadOnly version.
  • The image sealer cannot create, update, or delete a Gallery version or delete its lock.
  • A missing, weakened, renamed, changed-ETag, or mismatched seal lock fails before reservation.
  • A retired Gallery version name can never be reused.
  • Correct image-pin workflow order verifies build and seal evidence before it reads the statically authorized App key.
  • A compromised image-pin job can read that App key before verification; the threat test records this static-RBAC residual.
  • A critical vulnerability, secret, or malware finding blocks publication.
  • A high vulnerability without a current matching exception blocks publication.
  • An expired, overlong, or wrong-digest exception fails.
  • Pull-request verification succeeds without Azure or registry credentials.
  • Correct image-pin workflow order opens a pull request only after verification succeeds.
  • A compromised image-pin App can open an arbitrary pull request; independent required CI and branch protection prevent merge.
  • Firecall namespaces reject mutable container, init-container, and ephemeral-container images.
  • Reviewed digest pins pass the live admission policy.

36.11 Recurring trust and security tests

  • Generate one golden FIC fixture for every closed table row and reject a missing or extra live credential.
  • Require exactly sub, repository_id, repository_owner_id, and job_workflow_ref in each claims expression.
  • Reject a wrong repository ID, owner ID, subject, workflow commit, issuer, audience, language version, non-null subject, wildcard, duplicate expression, or unsupported claim lookup.
  • Require each event-specific reusable workflow to reject the wrong event, rerun, caller workflow, caller commit, reusable path, or reusable commit before Azure login.
  • Decode every signing certificate and require the exact approved 40-character privileged-workflow commit.
  • Prove Entra rejects the wrong reusable path and old commit.
  • A request-key switch accepts an old envelope only inside its 20-minute age and 22-minute verifier window.
  • A workflow that starts after its 15-minute queue allowance fails before Azure login.
  • Old-key retirement waits for two zero-outstanding GitHub and receipt reads 60 seconds apart.
  • A signing and rotation race produces one signed switch epoch and no envelope issued after that epoch with the old key.
  • An unreadable run, receipt, selector, or authenticated time source delays old-key disablement.
  • Close, sweep, break glass, and zero-resource proof remain available when every creation verdict is stale or blocked.
  • The trust bootstrap creates the observer application without its FIC.
  • Observer consent grants only the one exact Graph permission in section 24.1.5.
  • The second trust apply refuses the observer FIC without exact consent evidence.
  • Scheduled and manual trust runs accept only their separate exact FICs.
  • The settings-observer App reaches only infrastructure and its four declared read permissions.
  • Settings-observer source, Actions, environment, protection, and rules reads pass while every write fails.
  • A wrong firecall-live environment ID, reviewer, self-review setting, branch rule, or administrator bypass blocks mutation.
  • Platform-owner configuration accepts one increasing revision, sorted unique subject keys, and its canonical digest.
  • Every Firecall token, cookie, envelope, status, Blob, audit event, log, metric, trace, workflow input, and Git file rejects a raw Cloudflare sub value.
  • The canonical-email domain-separated subject-key fixture produces the same lowercase digest in Go, TypeScript, shell tooling, and policy tests.
  • Changing Cloudflare sub changes access_subject_key but keeps the durable email-derived subject_key and offboarding denial.
  • A missing, mixed-case, malformed, or non-openmined.org email fails every existing authorization.
  • A short-lived browser artifact fails when its current Access sub binding changes, even when its durable key remains equal.
  • A missing, rollback, partial, mismatched, or non-atomic owner-list reload blocks owner access.
  • Removing one owner closes only that owner's streams within five seconds.
  • Control-monitor bootstrap refuses an existing, malformed, or wrong-path checkpoint.
  • The first poll and every later poll bind sequence, ETag, prior digest, run numbers, and immutable audit.
  • A disabled workflow, missing expected run, deleted artifact, run-number gap, rollback, or stale checkpoint alerts.
  • Control close, sweep, and break-glass jobs can read exact Actions facts and cannot mutate another run.
  • Image-idle can read exact image runs and cannot cancel, rerun, delete, or mutate them.
  • The access-canary batch requires a platform owner, exact Origin, CSRF, fetch-site, and three bound audiences.
  • Unauthorized, duplicate, cross-subject-key, cross-audience, stale, replayed, and flooded challenge requests fail.
  • Broker challenge create succeeds while read, list, overwrite, delete, sibling, and root-list operations fail.
  • Observer bounded challenge list and exact reads pass while sibling and root lists fail.
  • Consumption create and identical replay pass idempotently; conflicting replay, overwrite, and delete fail.
  • Each fresh canary uses an isolated browser profile and produces a distinct Cloudflare authentication event.
  • Each challenge email subject key matches the durable subject key without storing the email.
  • Missing amr, custom.tid, and custom.oid claims do not block development authentication or canary evidence.
  • A missing, malformed, reused, or wrong identity-nonce key blocks canary correlation.
  • The temporary identity canary proves final signed Access JWT claims through its unique audience and no-origin host.
  • That canary has zero connectors, no matching ingress, no accepted steady audience, and no application data path.
  • Its policy, application, and DNS record are absent in two reads 60 seconds apart before steady activation continues.
  • Its Chromium profile exists only on swap-disabled private tmpfs and its audience is never reused.
  • After Chromium exits, only one locked controller buffer contains its raw token.
  • The helper proves zero browser processes, mounts, profile entries, downloads, crash files, and persistent work-root entries.
  • Each canary token exists only in locked memory and clears after validation.
  • Expiry, closed activation state, a wrong audience, or missing origin cannot satisfy the Access identity canary.
  • A wrong one-time PIN provider, email-domain policy, application, audience, signer, or source blocks the verdict.
  • Evidence reread must pass digest, signature, schema, and WORM checks before pointer advance.
  • A missing, stale, unsigned, malformed, rollback, or blocked access verdict denies new requests and reservations.
  • The node monitor binds node name, UID, pool, subnet, InternalIP, exact label, exact taint, allocatable facts, and system tuples.
  • Removing or changing the Firecall label or taint publishes blocked, alerts, and denies new sessions.
  • A wrong node, UID, CIDR, pool, subnet, DaemonSet, Pod, request, image digest, signer, source, generation, or prior digest fails.
  • Missing, stale, or inaccessible CPU-credit metrics block the node verdict.
  • Sustained low credits block creation while close, sweep, and break glass remain available.
  • The node observer cannot read Secrets, logs, events, exec, attach, port-forward, Binding, or workload writes.
  • Scanner database schedule and manual modes use separate FICs and event-specific workflow guards.
  • Reject a database with a wrong signer, source, media type, schema, digest, CVD signature, freshness, or archive bound.
  • Reject a malformed, replayed, rollback, missing, or stale scanner current pointer.
  • Reject a missing, partial, stale, rollback, raced, or wrong-signer ClamAV bundle and current pointer.
  • All three exact CVD signature, digest, version, endpoint, size, and freshness checks must pass together.
  • Two publishers prove lease and ETag exclusion; an ambiguous pointer write resolves through exact reread.
  • WORM storage rejects scanner object overwrite, delete, path reuse, and signature replacement.
  • Rescan rejects incomplete enabled-subject coverage, a wrong subject-set digest, and a wrong Gallery seal.
  • Rescan rejects missing, stale, forged, malformed, rollback, or quarantined scanner input.
  • A critical, secret, malware, unexcepted high, or expired exception publishes blocked.
  • Recovery creates a higher generation with prior-digest continuity and never rewrites prior evidence.
  • Broker and runner fail closed on a missing, stale, unsigned, incomplete, or blocked security verdict.
  • Close and sweep continue during scanner outage, quarantine, pointer race, and recovery.
  • Zone-policy evidence rejects a wrong signer, zone, phase, ruleset, rule order, digest, source, or live state.
  • Edge consumption rejects a wrong run, attempt, workflow, repository, head SHA, artifact name, artifact digest, Blob path, or stale proof.
  • Artifact substitution, duplicate artifact, latest-run lookup, expired artifact, and extra archive content fail.
  • Cache-bypass probes prove no cache reuse between two subjects on every Firecall host class.
  • Classic cutover trust evidence proves the historical FIC absent and the two reduction FICs exact.
  • The no-Azure cutover workflow reruns and cancels only fixed Classic open runs and receives no OIDC credential.
  • Capacity three rejects wrong trust evidence, historical run, denial run, artifact, step result, or non-terminal Classic open.

36.12 Access, browser, SSH, and containment tests

  • Use a raw-byte HTTP/1.1 harness and a TLS HTTP/2 frame harness for every parser boundary.
  • Prove malformed host, path, framing, header, slow-read, and body-limit requests stop before JWT, storage, proxy, or agent calls.
  • HTTP/1.1 rejects duplicate Host, a port, trailing dot, mixed case, IDNA, percent encoding, and unknown session host.
  • HTTP/2 rejects missing authority, duplicate authority, Host mismatch, cross-route host, and wrong audience.
  • Reject encoded slash, encoded backslash, encoded dot, dot segments, duplicate slashes, NUL, invalid UTF-8, and semicolon paths.
  • Reject raw and decoded path mismatch before authorization on portal, workspace, preview acknowledgement, and WebSocket routes.
  • Preview applications can never receive a path under the reserved /.firecall/ prefix.
  • Reject an oversized request line, header bytes, header count, one header, Access JWT, or duplicate security header.
  • Slow headers, slow bodies, idle pre-auth connections, excess source connections, and excess HTTP/2 streams release resources.
  • Reject conflicting content lengths, transfer encoding with content length, unsupported coding, and HTTP/2 connection headers.
  • Reject folded headers, request trailers, absolute-form targets, authority-form targets, and conflicting HTTP/2 pseudo-headers.
  • Reject an Access JWT with lifetime above 3,600 seconds or invalid iat, nbf, exp, or skew relation.
  • Each Firecall application selects only the managed one-time PIN provider.
  • Each allow policy includes only openmined.org and requires that provider.
  • Three Firecall host canaries pass, and the owner records the tested impact on unrelated Access applications.
  • Wrong team auth domain, issuer, JWKS URL, organization setting, or application duration blocks readiness.
  • Startup without a fresh JWKS stays unready, and no cached key remains usable after 15 minutes.
  • Unknown-kid floods cause at most one single-flight refresh per minute per replica and never authorize an unknown key.
  • Every stream closes at the lesser of JWT expiry and one hour after issuance.
  • Per-user Cloudflare revocation with both global and application tokens denies all three hosts within the documented window.
  • The offboarding view exposes only subject fingerprint, role, count, and a signed five-minute selection token.
  • An email rename creates a different development subject key. Offboard the old key before authorizing the new key.
  • A collaborator bound in two invitation indexes causes both sessions to receive revocation before any close dispatch.
  • A platform-owner target selects every active session and requires the separate protected owner-list removal.
  • Missing or malformed invitation state conservatively revokes every active session.
  • Both proxy replicas enforce the global identity marker within five seconds and fail closed on storage errors.
  • The fixed agent revoke operation kills tmux, shells, and sshd; arbitrary agent or sessiond revoke input fails.
  • A partial close keeps access denied, capacity occupied, and cleanup visible.
  • Normal close, sweep, broker, and control identities cannot delete an identity marker.
  • Reinstatement fails before old Access-token expiry, mailbox restoration, different-owner approval, and exact audit proof.
  • Source IPv4 for SSH always equals the validated requester Cloudflare IPv4 and never accepts user input.
  • SSH key proof rejects a wrong key, signature namespace, subject key, request, address, expiry, or replay.
  • The provider accepts only one runner-generated Ed25519 provisioning public key for fixed user firecall-provision.
  • Its private half never leaves locked runner memory and is absent from state, custom data, disk, environment, arguments, logs, and artifacts.
  • Finalization rejects a missing, additional, changed, option-bearing, or wrong-fingerprint provisioning key.
  • Readiness requires every Azure or cloud-init default key file absent and the provisioning account locked with nologin.
  • A real requester PTY login succeeds with from=...,restrict,pty and the pinned host key.
  • Root, default Ubuntu user, stale key, password, keyboard-interactive, extra authorized key, and wrong source fail.
  • Agent, local, remote, dynamic, X11, tunnel, and device forwarding fail.
  • ~/.ssh/rc, user RC files, host-key substitution, trust-on-first-use, and host-key update fail.
  • Image qualification finds no builder, cloud-init, root, or default-user authorized key.
  • One noisy collaborator cannot exceed four terminal, four log, six combined streams, or actor byte limits.
  • A second actor keeps usable terminal and log capacity while the noisy actor is limited.
  • Per-actor counts remain exact across both proxy replicas and release after disconnect or timeout.
  • Preview IDs decode to exactly 128 random bits and canonical 26-character lowercase Base32.
  • User-supplied, colliding, mixed-case, noncanonical, extra-separator, IDNA, encoded, and mismatched preview hosts fail.
  • Host, authority, preview record, session, target, and Cloudflare route must match exactly.
  • Process storms, descriptor storms, inode exhaustion, byte floods, and disk exhaustion preserve agent, expiry, and control queues.
  • Swap, shell-history defaults, core dumps, process accounting, TTY audit, PTY journaling, backup, snapshot, and restore-point checks remain disabled.
  • Tmux history stays at the declared bound, and zero-resource proof includes disks, snapshots, restore points, and backup items.
  • A rollback_failed disk appears as retained sensitive session data and never as completed deletion.
  • Legacy ServiceAccount-token Secrets fail admission.
  • TokenRequest accepts only Pod-bound tokens with UID and 600 through 3,600 seconds.
  • A copied Secret or workload token remains an explicit residual and never becomes a false revocation claim.

36.13 Infrastructure and activation tests

  • The operation-lock bootstrap creates generation zero once and applies exact legal hold firecalloperationlock.
  • Data bootstrap receives exactly five one-current-object roles and one operation-lock role, then removes all six.
  • Ordinary account Blob versioning remains enabled while immutableStorageWithVersioning stays false on every Firecall container.
  • Every protected container rejects existing-path Put Blob, Put Block List, and Copy Blob operations.
  • The operation-lock container permits only its declared lease acquire, renew, release, and break operations against the fixed Blob.
  • Standing, builder, and idle identities can acquire, renew, and release the exact Blob lease.
  • The same identities cannot change content, create a sibling, delete, list, tag, or remove the hold.
  • A changed lock digest, missing hold, lease loss, or concurrent operation fails before Azure mutation.
  • Every Firecall backend uses Azure AD data-plane authorization and OIDC in protected workflows.
  • Owner bootstrap uses verified Azure CLI with Azure AD storage authorization and no shared key.
  • Every Firecall principal fails account-key, account-SAS, service-SAS, and user-delegation-key creation.
  • Every Firecall AzureRM provider disables resource-provider registration.
  • An unregistered required provider blocks the saved plan before mutation.
  • Cloudflare 5.23.0, AzureRM 4.81.0, AzureAD 3.9.0, and all lock hashes match source.
  • Protected AzureAD uses OIDC only; owner bootstrap uses verified Azure CLI only; secret and certificate fallbacks fail.
  • The bounded local exception rejects an unlisted root, script, state key, plan, commit, role, auto-approve, or destroy.
  • The exact state-bootstrap and first Policy-assignment saved applies pass only after separate owner approval.
  • Storage-account and Key Vault name-availability checks pass before their first saved plan.
  • A name conflict fails without a random suffix.
  • The storage account is StorageV2, Standard, LRS, hot, and single-region.
  • Hierarchical namespace, NFS, SFTP, cross-tenant replication, infrastructure encryption, nested public access, and shared-key authorization stay disabled.
  • The owner accepts LRS regional-loss and delayed-lifecycle-deletion residuals before activation.
  • The Key Vault is Standard and every fixed signing key uses software-backed P-256.
  • Each Firecall apex equals its complete full-setup Cloudflare zone apex and has a different registrable domain.
  • Universal SSL has active exact SAN coverage, a valid public chain, TLS 1.3 SNI, and at least 30 days remaining.
  • A nested apex, missing SAN, incomplete chain, short expiry, disabled certificate, or TLS downgrade blocks every public route.
  • The Cloudflare observer reads only both exact zones' SSL and certificate metadata.
  • The edge mutation token cannot edit SSL, certificate packs, Total TLS, or Advanced certificates.
  • Packer uses Azure CLI after exact OIDC login and rejects secret, certificate, managed-identity, device, and environment fallback auth.
  • Packer generated names, fixed NSG, absent generated NSG, plugin checksum, and resource inventory match evidence.
  • Image qualification uses only /dev/ptp_hyperv as its chrony time source and denies network NTP.
  • Five samples on Standard_D2ds_v5 and Standard_B2s stay within 100 milliseconds of authenticated time.
  • A missing PTP device, another selected source, UDP 123, or changed chrony.conf digest blocks image qualification.
  • BS-family, DDSv5-family, Total Regional vCPU, and Standard public-IP checks each fail independently at insufficient headroom.
  • The 2026-05-02-preview AKS VM-SKU read returns one unrestricted Standard_D2ds_v5 entry before the pool plan.
  • A changed API schema, missing SKU, duplicate, other location, restriction, or unreadable SKU response blocks the pool plan.
  • Reservation and image build count pending public-address demand before Azure exposes a new resource.
  • Three depleted B2s session VMs and one loaded D2ds v5 node meet the fixed one-hour acceptance targets.
  • If one target fails, activation blocks until SKU, Policy, quota, cost, and evidence change together.
  • The baseline has runtime Deployments at zero, two ready validators, and no CronJob, ExternalSecret, one-shot Job, or runtime PDB.
  • Both VNet peerings report Connected and Succeeded with synchronized address spaces and exact reviewed flags.
  • Either peering denies gateway transit and forwarded traffic while it permits only virtual-network access.
  • Ten fixed certificate Secrets exist before a consumer Deployment mounts them.
  • The certificate-bootstrap Job reports only fixed names, UIDs, resource versions, owners, field-presence booleans, and one sanitized digest.
  • The activation verifier reads that bounded Job result and never receives Kubernetes Secret read authority.
  • The broker receives broker-origin and broker-revoker slots and no proxy or readiness private key.
  • The proxy receives proxy-origin and proxy-client slots and no broker-revoker private key.
  • The readiness worker receives only readiness-client slots.
  • The active selector contains the broker-revoker slot and SPKI, and every replica reports that exact selection.
  • Rotation cannot select an epoch until the broker-revoker handshake and signed request canary pass.
  • A new client-root leaf fails until every cluster verifier and allowed VM image trusts the new bundle.
  • Old client-root ConfigMaps remain until current Pods, Jobs, CronJob templates, ReplicaSets, images, sessions, and leaves report zero references.
  • Argo remains Synced and Healthy through permitted runtime Secret, selector, and epoch changes.
  • The excluded owner-seeded Leases remain outside every Argo resource tree.
  • Each jq ignore applies only to the exact validator webhook CA or managed selector expression.
  • Deleting an unrelated ApplicationSet resource fails the source-preservation test.
  • External Secrets recreates one deleted target Secret without Argo owning its data.
  • External Secrets incident recovery revokes identity, rotates every exportable value, proves nonexportable keys, and restores canaries.
  • Certificate-authority compromise revokes both App slots, replaces roots and leaves, rejects old signatures, and restores no old key.
  • Every temporary owner, bootstrap, consent, state, vault, and Blob assignment disappears before routine automation.
  • Static network qualification never claims that a session endpoint exists.
  • Only the first owner canary can use the one-time network-path bootstrap for the current profile generation.
  • That canary requires zero sessions, limit two, the exact profile, static network proof, and no existing generation marker.
  • A failed mTLS path creates no network marker and forces complete canary cleanup.
  • A successful first path creates one legal-held statement with exact peerings, subnets, source CIDR, endpoint, image, and generation.
  • Every later open requires that exact current marker and repeats all live network facts.
  • Dark preflight creates no Azure session resource and produces no positive terminal, Kubernetes, log, preview, close, expiry, or cleanup claim.
  • Initial control, scanner, node, access, cache, credential, capacity, and legal-hold proofs all pass before creation becomes true.
  • Activation stops at the first missing, stale, failed, mismatched, or unsigned gate and leaves close and reduction paths available.

37. Capacity, cost, and retention

37.1 Capacity

Use these initial limits:

  • Three v2 sessions after cutover.
  • Two v2 sessions during Classic coexistence.
  • Eight terminal windows per session.
  • Three previews per session.
  • Eight-hour maximum session time.
  • Four-hour default session time.
  • Thirty-minute terminal idle timeout.
  • Fifteen-minute preview idle timeout.

37.2 New spend categories

Live activation adds:

  • Up to three Standard_B2s VMs.
  • OS disks.
  • One Standard static public IP for each active session.
  • Same-region VNet peering data transfer.
  • Azure Compute Gallery image storage.
  • One StorageV2 Standard LRS storage account with Blob operations and retained versions.
  • One Standard Key Vault with key, secret, and signing operations.
  • One fixed Standard_D2ds_v5 Firecall AKS node with a 64-GiB managed disk.
  • Cloudflare zone or domain costs.
  • One temporary Standard_D2ds_v5 image-builder VM, disk, and public address per build.

This design adds no Advanced Container Networking Services charge.

The Azure Retail Prices API reported these East US Linux rates on 2026-08-24:

  • Standard_B2s: USD 0.0416 per hour.
  • Standard_D2ds_v5: USD 0.113 per hour.
  • Standard static IPv4: USD 0.005 per hour.
  • E4 32-GiB Standard SSD LRS: USD 2.40 per month.
  • One Firecall Standard_D2ds_v5 AKS node: USD 0.113 per hour.
  • Its P6 64-GiB managed disk: USD 10.207 per month.
  • Standard_D2ds_v5 image builder: USD 0.113 per hour.

Using 730 hours per month, the eight-hour base estimate is USD 0.3991.

Round the displayed maximum base estimate to USD 0.40 per session.

Three full eight-hour sessions have a USD 1.20 base estimate.

The four-hour default has a USD 0.20 base estimate.

The dedicated Firecall node has a USD 92.70 monthly compute-and-disk estimate.

Three continuously occupied session slots have a USD 109.25 monthly base estimate.

The dedicated node plus continuous session slots have a USD 201.95 monthly base estimate.

Propose a combined USD 235 monthly Azure Firecall budget for the initial release.

The saved standing plan records current Standard LRS capacity, operation, retention, and Standard Key Vault estimates.

The owner accepts the LRS single-region durability residual and those variable operation costs with the budget.

Use an owner-managed terraform/stacks/azure/firecall-budget/prod root.

Its USD 130 budget filters the exact Firecall-owned control, session, and image-build resource groups.

That resource-group filter includes implicit VM OS disks that cannot carry Firecall tags.

Its separate USD 105 budget filters the exact managed AKS Firecall pool cost identity and reviewed tags.

Add a separate alert when a managed AKS resource does not preserve that identity or tag.

Alert each budget at 50, 75, 90, and 100 percent.

The two thresholds form the combined USD 235 operating limit.

The budget root owns only those exact cost alerts and the merged Action Group resource ID.

It rejects another notification target, a missing Action Group ETag, or a contract-to-live digest mismatch.

An authenticated owner first applies the USD 130 resource-group budget before any image build.

The owner adds the USD 105 exact-pool budget after the Firecall pool exists and before application replicas start.

No routine OIDC identity receives Microsoft.Consumption/budgets/write or cost-alert write authority.

The owner runs a reviewed plan monthly and after a cost-scope change.

The budget state and live readback record both filters, thresholds, contacts, ETags, and action references.

The budget includes normal image-build usage but not an unbounded build loop.

The budget is a detection control and never replaces the three-session reservation.

An AKS upgrade can add one temporary D2ds v5 surge node at the same hourly compute rate.

These are public retail estimates, not the subscription invoice rate.

They exclude data transfer, standing storage, existing AKS capacity, and failed-cleanup retention.

Refresh and review the pinned rates before the trust apply.

The owner must approve this spend.

37.3 Retention proposal

Propose:

  • Browser, terminal, preview, and box-local Kubernetes access end at close or expiry.
  • Azure cleanup follows, with the documented root-compromise token residual.
  • Zero-resource Terraform state remains current for 30 days and has a 61-day recovery target.
  • Request receipts remain current for 90 days and have a 121-day recovery target.
  • Lifecycle status remains current for 90 days and has a 121-day recovery target.
  • Replay tombstones remain for the complete Firecall service lifetime.
  • The capacity-bootstrap marker remains for the complete Firecall service lifetime.
  • Dedicated Terraform state follows the operator-approved state-retention schedule.
  • Public root objects and locks remain through every dependent image and audit retention.
  • Request-key candidate and status evidence remains under legal hold for the Firecall service lifetime.
  • Immutable session-level audit events remain current for at least 91 days and have a 122-day recovery target.
  • Readiness probes are deleted after verification.
  • Close revocations are deleted after a terminal lifecycle state.
  • Identity offboarding revocations remain for the Firecall service lifetime unless an owner approves reinstatement.
  • Immediate-revocation delivery receipts remain current for 91 days and have a 122-day recovery target.
  • Invitation records are deleted after a terminal lifecycle state.
  • Every historical public client-leaf version and immutable leaf-evidence object remains for the Firecall service lifetime.
  • Rate-limit windows are deleted after 24 hours.
  • Firecall does not copy terminal bytes into central audit, logs, traces, status, or workflow artifacts.
  • Tmux scrollback, user files, and process memory can retain session content until the box and OS disk are deleted.
  • A rollback_failed disk retains that content and remains an incident exposure until reduction completes.

Enable Blob versioning and 30-day soft delete for state, request, status, capacity, revocation, invitation, and public trust data.

These targets include the 30-day soft-delete window and one daily reconciliation interval.

Azure lifecycle management has no bounded completion service-level agreement.

Do not describe these targets as guaranteed maximum retention.

The scheduled control sweep owns retention deletion.

It acquires the session lease and repeats zero-resource proof immediately before deletion.

It can delete a session record only when all conditions pass:

  • State is closed, expired, failed, or safely rejected.
  • cleanup_verified=true when resource creation could have started.
  • No capacity-ledger entry remains.
  • Zero-resource proof still passes.
  • The terminal record has one immutable, schema-valid terminal_at value.
  • Azure Storage's authenticated response time is not earlier than terminal_at.
  • Zero-resource state age is at least 30 complete days before state deletion.
  • Request and lifecycle-status age is at least 90 complete days before their current-base deletion.

Use the Azure Storage service Date response, not the runner clock, for each age comparison.

A missing, changed, future, or regressed terminal timestamp blocks deletion and alerts.

rollback_failed has no eligible terminal timestamp and never enters retention deletion.

Never age-delete a current accepted, dispatched, reserved, building, verifying, ready, closing, or rollback_failed object.

Use container-and-prefix lifecycle filters to delete only prior versions of PII-bearing records.

Delete prior request, status, revocation, invitation, and capacity-ledger versions 90 days after each version's creation.

Delete prior zero-resource Terraform-state versions 30 days after each version's creation.

Do not apply these version actions to the current base Blob.

The scheduled sweep alone deletes a current base record after its state and cleanup checks pass.

Never delete the current capacity-ledger Blob.

Prior capacity-ledger versions use the same 90-day version rule and 30-day soft-delete recovery window.

The permanent bootstrap marker contains only the initial empty-ledger digest and no human or session identifier.

Never age-delete or lifecycle-delete a replay tombstone.

Apply the indefinite firecallreplay legal hold before the first canary session.

Read back and verify that exact tag before any tombstone writer receives authority.

Removing that legal hold or a tombstone needs explicit deletion approval.

Use one current-Blob lifecycle filter only for firecall-audit after 91 complete days.

The locked 90-day WORM policy must expire before that lifecycle deletion can succeed.

No human, workload, or workflow identity receives current-audit delete authority.

Add one exact prior-version audit filter after 92 complete days.

Azure lifecycle does not act on data that is already soft-deleted.

The account's 30-day soft-delete service expiry removes that deleted form.

Together, those controls target a 122-day recovery window from original audit creation.

After day 122, the daily verifier checks each eligible form and alerts until Azure reports it absent.

Prove lifecycle eligibility and alerting in a disposable locked container without claiming a bounded deletion time.

Keep mutable scanner, security, node-tuple, and access-policy current pointers for the Firecall service lifetime.

Delete their prior pointer versions after 91 complete days, with at most 30 additional recovery days.

Retain each immutable scanner database, scanner bundle, rescan, node-tuple check, access-policy challenge, consumption, and check for at least 91 days.

Retain an older immutable object while any retained current pointer, session, audit event, image record, release bundle, exception, or immutable evidence references it.

An object can remain longer than 122 days only because one of those retained references still needs it.

After its last retained reference ends, attempt deletion of an already old eligible object at the next daily run.

Alert when it remains after 24 hours, but do not claim that the storage service guarantees this bound.

Soft delete permits recovery for at most 30 more days.

Add protected firecall-retention.yml, firecall-retention-schedule-privileged.yml, and firecall-retention-recovery-privileged.yml workflows on infrastructure main.

Run the retention workflow once each day and permit one owner-approved recovery dispatch.

Its dedicated OIDC identity can list and read only the exact reference and evidence prefixes.

It can delete only eligible current objects and versions under firecall-scanner-data, firecall-security-evidence, firecall-node-tuple-evidence, and firecall-access-policy-evidence.

It can also delete only terminal delivery receipts under firecall-revocations/deliveries/ after 91 complete days.

It cannot delete an audit object, current pointer, identity offboarding marker, tombstone, trust object, release pin, status, session record, request, capacity object, or active delivery receipt.

Before deletion, it builds one canonical reference graph from exact ETags, version IDs, paths, and content digests.

It requires two identical graph reads 60 seconds apart and authenticated Azure Storage time.

It writes one immutable retention intent before deletion and one immutable completion after verified absence.

An unreadable container, unknown schema, missing digest, reference race, changed ETag, WORM rejection, or audit failure stops the batch.

A compromised retention identity can delete expired unreferenced evidence and deny historical analysis.

It cannot delete evidence before WORM expiry or remove the authoritative audit intent.

Owner approval of this residual and every exact deletion scope is required before its first schedule starts.

Use other storage lifecycle rules only for expired rate-limit bases, the exact prior-version filters above, and approved deleted-version scopes.

Exclude the current capacity-ledger base Blob from every current-Blob lifecycle rule.

Exclude firecall-trust and firecall-trust-evidence completely from age-based lifecycle deletion.

Never age-delete a public leaf slot version, immutable leaf evidence, or the active rotation record.

Do not use an age-only current-base lifecycle rule for session state, requests, status, revocations, or invitations.

Test the lifecycle policy against every version and soft-deleted copy in disposable containers.

Prove the daily verifier alerts on every PII-bearing version that remains readable after its deletion target.

Prove an active or rollback_failed current object remains readable beyond that age.

Test the audit policy in a disposable container.

Then lock the production 90-day policy before any audit writer receives authority.

The operator runbook includes version and soft-delete recovery.

Storage deletion policies need owner approval before activation.

38. Classic coexistence and migration

38.1 No shared ownership

Classic keeps:

  • Resource groups named rg-firecall-dev-<session_id>-eastus.
  • The possible pre-concurrency group rg-firecall-dev-eastus until zero proof confirms its absence.
  • State key azure/firecall/dev.tfstate.
  • State key azure/firecall/dev-slot2.tfstate.
  • State key azure/firecall/dev-slot3.tfstate.
  • Its current workflow and sweeper.

V2 uses:

  • Resource group rg-firecall-sessions-dev-eastus.
  • One independent state per session.
  • Its control-repository workflows and sweeper.

The systems never manage the same resource.

38.2 Canary limits

Classic can use three sessions before coexistence admission starts.

Before v2 creation becomes enabled, merge a protected Classic change that reduces new-open admission from three sessions to one. Close and sweep must continue to enumerate all three Classic state keys.

Two repeated live reads must prove the protected admission limit and no more than one active Classic session.

V2 uses a temporary maximum of two sessions.

This keeps the combined ceiling at three.

38.3 Cutover

  1. Complete the section 21.7 dark activation with creation disabled and limit two.
  2. Run the complete negative identity, policy, network, secret, and admission suite.
  3. Merge the protected profile enable after image, Policy, rescan, node, access, and certificate proofs pass.
  4. Verify the live enabled profile while creation_enabled=false remains unchanged.
  5. Merge the protected Classic change that reduces new-open admission from three to one.
  6. Keep Classic close and sweep coverage for all three state keys.
  7. Require two live reads to prove the admission limit and at most one active Classic session.
  8. Merge creation_enabled=true while capacity_limit: 2 remains unchanged.
  9. Reconcile and verify the protected config, bootstrap marker, marker digest, enabled profile, and leased ledger.
  10. Open one owner canary session.
  11. Verify terminal, k9s, logs, events, SigNoz links, and one selected web preview.
  12. Join from a second verified identity and test owner and invitation revocation.
  13. Verify close and expiry.
  14. Open two v2 sessions and verify that a third v2 session is denied.
  15. Run failed-open, failed-destroy, signature, digest-admission, and rollback tests.
  16. Merge a Classic workflow change that rejects new open requests and shows the portal URL.
  17. Activate the new SHA-bound Classic reduction identity for close, sweep, and zero-proof only.
  18. Remove the historical combined-workflow federated credential and wait for its issued tokens to expire.
  19. Keep the current combined caller and new reduction workflow enabled for reduction only.
  20. Prove that a new Classic open request is rejected.
  21. Rerun one completed pre-rejection Classic open and require Azure authentication denial.
  22. Let the owner-approved no-Azure cutover job cancel every bounded non-terminal Classic open run.
  23. Wait until two live API reads, five minutes apart, show no non-terminal Classic open run.
  24. Keep Classic close and sweep actions available while every existing Classic session drains.
  25. Prove zero Classic managed resources, zero bound roles, no state lease on any Classic key, and only null session outputs.
  26. Dispatch protected Classic zero-proof when no final close produced the artifact.
  27. Bind the signed artifact to the merged rejection commit and repeat the workflow-run query.
  28. Record both workflow IDs, credentials, rejection commit, run IDs, query times, and zero-proof digest.
  29. Merge the protected config change and let capacity-reconcile.yml raise v2 from two to three.
  30. Make capacity reconcile repeat the live source, run, credential, and resource checks before its ledger write.
  31. Verify the leased ledger and live protected configuration both report three.
  32. Merge qualification_mode=true with only current platform-owner subject keys, a 12-hour epoch, and three-hour session limits.
  33. Start the exact node-load Pod in drain mode before any qualification session exists.
  34. Wait for authenticated node credits below 15 for 15 minutes, with an eight-hour drain deadline.
  35. Replace the drain Pod with the exact sustain contract and verify node credits remain below 15.
  36. Open three normal owner sessions and verify that a fourth session is denied.
  37. Start fixed three-hour CPU qualification on all three boxes.
  38. Require all three VM credit balances below 15 for 15 minutes before the timed suite begins.
  39. Hold all four balances below 15 and run the complete one-hour load suite.
  40. Test terminal, Kubernetes proxy, preview, close, expiry, local revoke, latency, and cleanup during that hour.
  41. Delete the load Pod, close every qualification session, and require zero-resource proof and zero occupied slots.
  42. Merge qualification_mode=false; if any target failed, also merge creation_enabled=false.
  43. Publish signed node, VM, latency, expiry, cleanup, and credit evidence.
  44. Observe for seven days.
  45. Retire Classic only after explicit deletion approval.

Keep all three valid zero-resource Classic state Blobs through approved audit and rollback retention.

Do not delete them as part of capacity cutover.

38.4 Rollback

Rollback first merges creation_enabled=false and capacity_limit: 2 together.

The protected capacity reconcile workflow writes both values to the leased ledger atomically.

The portal disables Create, the broker rejects requests, and the runner rejects any delayed open.

The gate blocks new opens with zero, one, two, or three occupied slots.

If three v2 sessions exist, keep Classic open disabled until one selected v2 session closes.

Then prove at most two v2 sessions remain.

Do not restore the historical Classic federated credential.

A Classic-open rollback needs a new owner-approved credential bound to current protected source and its exact reusable-workflow SHA.

Restore Classic open only after that new credential passes positive and historical-rerun denial tests.

While any v2 session remains, restore Classic with an admission limit of one. Restore its admission limit to three only after signed proof shows zero v2 sessions.

Keep v2 creation disabled until a reviewed rollback-exit config merge.

V2 close and sweep remain enabled for existing v2 sessions.

Do not point Classic at v2 state.

Do not delete v2 status during rollback.

39. Implementation groups

The suspended program workflow does not apply.

Use six coherent landing groups.

The owner reviews each group before merge.

Use this dependency order:

Group A: architecture and contracts
        ├── Group B: control engine
        ├── Group C: runtime and image source
        └── Group D: portal and workspace
Groups B + C + D
        └── Group E: trust, image publication, GitOps, and edge activation
Group E
        └── Group F: live qualification and Classic cutover

Groups B, C, and D can proceed in parallel after Group A.

No delivery-program package or milestone record gates these groups.

39.1 Agent and worktree coordination

Use one isolated worktree and branch for each open pull request.

Before work starts, record the branch, assigned agent, exact file prefixes, dependencies, and expected merge order.

Do not let two agents edit the same file in parallel.

Group A owns shared schemas, contracts, ADR text, and the cross-repository source inventory.

Group B owns the control repository and its prerequisite infrastructure bootstrap support after Group A merges.

Group C owns infrastructure runtime, image, supply-chain, and protocol implementation.

Group D owns portal and workspace presentation and consumes Group A protocol fixtures.

Any B, C, or D contract-shape change first lands as a small Group A amendment.

An activation record that uses the unchanged contract can land with its owning implementation group.

One Group E integrator owns each ordered E1 through E5 pull request and all shared Terraform or GitOps files.

Group F starts only from the exact merged Group E evidence digest.

Each agent rebases on current origin/main, preserves unrelated work, and publishes a changed-file list before review.

Each pull request contains one coherent group, its tests, documentation, rollback, and evidence schema.

The owner reviews and merges each group before the dependent group starts.

Group A — architecture and contracts

Infrastructure pull request:

  • Land this plan.
  • Amend and accept ADR-0012 for v2.
  • Amend CLAUDE.md with the exact bounded Firecall owner-bootstrap exception in section 24.1.1.
  • Supersede stale PR #389.
  • Add exact request, profile, status, and agent contracts.
  • Add release-evidence and vulnerability-exception contracts.
  • Add threat model and migration contract.
  • Add platform/firecall/README.md and the complete v1-to-v2 handover.
  • List every planned source path, owner, merge group, activation gate, and rollback owner.
  • Exclude every Firecall v2 Terraform root from the generic workflow.
  • Remove unused pull-request OIDC from the platform image workflow.
  • Split Talks pull-request validation from its protected-main OIDC publish workflow.
  • Add credentialless validation routing for each dedicated root.

No live cloud resource changes occur. This group removes pull-request entry into the generic Terraform Azure OIDC workflow. It also removes every current mixed pull-request and OIDC workflow. The legacy PR-trusted identity remains live until E1 removes it.

Group B — isolated control engine

Bootstrap-source infrastructure pull request:

  • Add scripts/ops/firecall-owner-operation.sh as the shared enforcement wrapper.
  • Add scripts/ops/firecall-control-bootstrap.sh and its offline tests.
  • Add the mode-aware, digest-bound dark control-repository seed.
  • Grant no live authority.

The owner reviews and merges this source change first.

Small Group A owner-activation pull request:

  • Activate one exact repository_bootstrap owner-operation row.
  • Amend ADR-0012 and CLAUDE.md in the same protected change.
  • Update both hidden authority markers to bind the active manifest.
  • Set source_commit to the verified prior bootstrap-source merge commit.
  • Bind the row to the verified control repository ID and exact GitHub scope.
  • Grant no Azure, Terraform, secret, deletion, or spend authority.

The owner reviews and merges this activation before the bootstrap runs.

One-time control-repository bootstrap:

  • An authenticated OpenMined repository administrator runs the pinned control bootstrap only through scripts/ops/firecall-owner-operation.sh.
  • The script requires repository ID 1345147942 and an empty or exact self-generated retry state.
  • It pushes one direct dark main seed before protection exists.
  • The seed contains the control contracts, disabled profiles, dormant Terraform, credentialless validation, and fail-closed workflow entry points.
  • A sorted manifest binds all seed paths, file modes, and SHA-256 digests.
  • Every profile, request key, service profile, and creation gate remains disabled.
  • It sets main as the default branch.
  • It requires the final commit's GitHub Actions validate check to succeed.
  • It then applies repository protection through the GitHub API.
  • Protection requires pull requests, CODEOWNERS, one approval, stale-review dismissal, resolved conversations, and the validation check.
  • Protection also blocks force pushes, branch deletion, and administrator bypass.
  • The script rereads every setting and writes a sanitized evidence file.
  • It never prints or stores the administrator token.

The repository-bootstrap readback requires this exact check set:

  • admin-bypass-blocked.
  • branch-deletion-blocked.
  • codeowners-required.
  • default-branch-main.
  • Exactly one of bootstrap-entry-empty, bootstrap-entry-initializer, or bootstrap-entry-final.
  • force-push-blocked.
  • repository-id.
  • required-pull-request-review.
  • required-status-check.
  • resolved-conversations.
  • seed-files.
  • stale-review-dismissal.

Small Group A authority-consumption pull request:

  • Add bounded sanitized readback and completion records under their content-addressed owner-bootstrap paths.
  • Bind their exact-byte digests to the completed operation ID and prior active authorization generation.
  • Set the owner-operation row to consumed.
  • Remove the active operation text from ADR-0012 and CLAUDE.md.
  • Update both hidden authority markers to bind the consumed manifest.
  • Keep the generic closed owner-bootstrap mechanism unchanged.

The owner reviews and merges this consumption change before product work starts in the control repository. These two authority pull requests use Group A ownership because they change the shared manifest, ADR-0012, and CLAUDE.md.

This bootstrap changes OpenMined repository settings.

It uses the named repository-administrator authority and needs explicit owner approval.

Repository settings cannot land through a source pull request.

First protected control-repository product pull request:

  • Add profiles and schemas.
  • Keep the production profile catalog disabled until the first image pin merges.
  • Add session Terraform.
  • Commit the reviewed Terraform CLI, provider constraints, and linux_amd64 lock file.
  • Add request verification.
  • Add the single leased capacity ledger.
  • Add the one-time ledger-and-marker bootstrap workflow.
  • Add protected capacity configuration and reconcile workflows.
  • Add the scheduled read-only audit verifier workflow.
  • Add the scheduled read-only AKS assignment auditor workflow.
  • Add the scheduled break-glass audit reconciler workflow.
  • Add open, close, sweep, and break-glass workflows.
  • Add exact break-glass request, approval, consumption, and replay tests.
  • Add readiness and zero-resource verification.
  • Add unit and integration tests.

Tests use local emulators and mocks.

This group needs no live Azure credential.

The direct seed exists only because an empty repository cannot receive a pull request.

All product code lands through the first protected pull request.

Group C — runtime and immutable image

Infrastructure pull request:

  • Add broker.
  • Add terminal proxy.
  • Add the credentialless controlled egress gateway.
  • Add the credentialless DNS policy proxy.
  • Add the node-binding validator and callback tuple checks.
  • Add readiness probe worker.
  • Add the dedicated certificate-rotator command and container build.
  • Add the dedicated credential-rotator command and digest-pinned container build.
  • Add the credentialless cloudflared origin-trust health sidecar.
  • Add a reproducible cloudflared source build and verify its digest through the same evidence format.
  • Add the cloudflared health-sidecar command and digest-pinned container build.
  • Add fixed-slot reload logic to every certificate consumer.
  • Add box agent.
  • Add the root-owned loopback Kubernetes API proxy and IMDS isolation rules.
  • Add the narrow root firecall-sessiond supervisor.
  • Add fc helper.
  • Add Packer image build.
  • Pin official Azure plugin v2.6.3, its checksum, and generated-name contract.
  • Add fixture tests for fixed names, fcimgdp<random5>, fcimgip<random5>, and absent fcimgsg<random5>.
  • Add container builds.
  • Add one credentialless runtime validation workflow with logic in scripts/ci/.
  • Add reproducible build logic that the Group E image workflows will call.
  • Add SBOM, provenance, and scan generation.
  • Add keyless-signing logic, offline evidence verification, and signed fixtures. Group E executes live signing and publication.
  • Add Firecall digest-admission policy fixtures and render tests. Group E owns the Kubernetes policy manifests.
  • Add credential-rotator and connector-health unit, admission, and network tests.
  • Add protocol and integration tests.

Use typed hostname variables and exact non-secret logical object names.

Do not merge a placeholder digest, package version, tool pin, provider pin, image pin, or evidence digest.

This group does not activate live trust.

Group D — polished portal and workspace

Infrastructure pull request:

  • Add portal screens.
  • Add build timeline.
  • Add workspace shell and tabs.
  • Add k9s launcher.
  • Add command palette.
  • Add workloads, logs, and events.
  • Add period-form SigNoz links.
  • Add co-debug and reconnect behavior.
  • Add join scope confirmation and active-connection presence.
  • Add separate-domain web previews.
  • Add mobile and accessibility behavior.
  • Add Playwright and axe tests.

Group E — GitOps, trust, and edge activation

Infrastructure and control pull requests:

Use five ordered pull requests inside Group E.

E1 adds standing resources, bootstrap source, and complete trust declarations:

  • Amend the operating contract through Group A before any owner bootstrap.
  • Remove the generic sp-infra-ci-plan FIC, application, principal, Azure and state roles, repository variable, outputs, and input wiring.
  • Keep its drift entry until live readback passes. Remove it in the protected evidence change.
  • Wait for old plan tokens to expire and prove a pull-request token cannot exchange before any Firecall identity activates.
  • Narrow generic Blob roles and add every dedicated backend container with prevent_destroy.
  • Add the exact storage account, all section 15 operational containers, locks, holds, and retention policies.
  • Add the Key Vault, Gallery, image definition, VNets, subnets, fixed image NSG, and registry paths.
  • Add every key, data, checkpoint, legal-hold, edge-state, and manual-secret bootstrap manifest and script.
  • Add every GitHub App: broker, image pin, root publication, Classic observer, control observer, and settings observer.
  • Add primary and secondary slots for every App key and edge, observer, and zone-policy Cloudflare token.
  • Add every identity, FIC, custom role, assignment, and residual in the complete sections 23 and 24 inventory.
  • Add trust-root, trust-observer, control-monitor, image-idle, image-inventory, Cloudflare-drift, and node-tuple workflows.
  • Add security-rescan and scanner-database publisher identities, workflows, roles, and signed pointer contracts.
  • Add tool-cache publisher and retention-reconciler applications, exact FICs, storage roles, workflows, and signed evidence contracts.
  • Create and lock the public tool-cache container before either identity receives data authority.
  • Add control-monitor, security, scanner, node-tuple, and access-policy current objects and immutable evidence prefixes.
  • Add the Cloudflare email one-time PIN contract, final-JWT canary, and first access verdict. Do not add a development Conditional Access or PIM gate.
  • Add the Cloudflare organization-read observer permission and every positive and negative credential canary.
  • Add the owner-only Policy-definition, image-build Policy, session Policy, and budget roots.

E2 publishes and proves immutable artifacts:

  • Create the four fixed CA roots, root lock, public bundle, and legal-held evidence.
  • Acquire, verify, sign, and publish every merged tool-lock artifact into the create-once cache.
  • Require exact cache bytes and acquisition evidence before any container or VM image build starts.
  • Build, scan, sign, and offline-verify every runtime container, including rebuilt cloudflared.
  • Build the Packer image through OIDC Azure CLI authentication and seal its exact Gallery version.
  • Publish and sign the first Trivy and ClamAV database generation.
  • Run the manual recurring-rescan path against the complete pinned subject set.
  • Merge image manifests, Policy allowlists, and the disabled control profile pin only after those proofs pass.

E3 adds the inactive GitOps and AKS baseline:

  • Activate dedicated firecall-apps.yaml, exclude both entries from platform-apps.yaml, and preserve every unrelated ApplicationSet resource.
  • Require that ApplicationSet to generate exactly the firecall and firecall-edge Applications.
  • Keep every runtime Deployment at zero replicas.
  • Run exactly two binding-validator replicas with its PodDisruptionBudget.
  • Omit every CronJob, ExternalSecret, one-shot Job, and runtime PodDisruptionBudget.
  • Add namespaces, quotas, fixed leaf Secrets, ServiceAccounts, RBAC, network policy, and admission in section 21.7 waves.
  • Create exactly three inert Leases through the retry-safe dev cluster-owner bootstrap before activation admission.
  • Apply node-isolation and scheduler-Binding policy before the Firecall pool exists.
  • Load the live system-DaemonSet contract and complete positive and negative admission probes.
  • Add firecall-aks/dev, create only the tainted pool and AKS-side peering, and leave preview nodes unchanged.
  • Verify exact quota, subnet, VMSS, node label, taint, system tuples, allocatable capacity, and CPU-credit metrics.
  • Publish the first signed node-tuple pass result before runtime activation.

E4 adds internal runtime, secret delivery, and Cloudflare source:

  • Add firecall-azure-kv with namespace conditions for only firecall and firecall-edge.
  • Add ExternalSecrets only after exact nonfunctional bootstrap objects, roles, and consumer canaries pass.
  • Add DNS proxy and HTTPS egress first, then internal HMAC and application secret delivery.
  • Add broker, proxy, readiness, fixed certificate slots, candidate Services, and the one-shot bootstrap Job.
  • Add each unsuspended maintenance CronJob only after its dependency proof.
  • Add the edge root, tunnel, routes, DNS, three self-hosted Access applications, and exact ordered policies.
  • Add the separate zone-policy root, state, token, complete phase imports, cache rule, and signed evidence handoff.
  • Add edge, observer, zone-policy, credential-rotation, and drift workflows with exact independent identities.

E5 performs the section 21.7 dark activation in exact order:

  • Run both trust applies around directory-admin observer consent.
  • Require initial credential canaries, control-monitor poll, scanner verdict, node verdict, and cache evidence.
  • Activate internal runtime and prove certificate and credential bootstrap before connectors start.
  • Read exactly one existing Cloudflare email one-time PIN provider while edge applications remain closed.
  • Prove the edge plan changes no identity-provider or unrelated application.
  • Prove the final development JWT through a temporary portal policy, then remove that policy and prove closed state.
  • Merge the exact provider, application, policy, and canary evidence.
  • Prove Universal SSL before the saved public routes become active.
  • Add the saved email-domain policies and ingress routes in protected dark_ready state with creation disabled.
  • Start two connectors and prove the three distinct application audiences.
  • Require distinct live connector and origin proofs.
  • Complete three fresh email-PIN canaries and publish pass.
  • Run the complete signed dark preflight while creation remains disabled.
  • End with the profile disabled and creation_enabled=false.

E5 does not enable a profile, reconcile an enabled creation gate, or open a session.

This group changes trust, secrets, edge authentication, and spend.

It needs explicit owner approval before live apply.

Apply standing resources before the first image publication.

Publish the image before the production profile pull request.

Merge the reviewed image pin before session creation is enabled.

Group F — qualification and cutover

Control and Classic pull requests:

  • Use protected config merges for every v2 creation-gate or limit change.
  • Run the exact capacity-reconcile workflow after each merge.
  • Enable the reviewed profile only after its sealed image, Policy allowlist, signed rescan, node verdict, and access verdict pass.
  • Keep creation_enabled=false while the profile-enable merge and live profile readback complete.
  • Merge creation_enabled=true with limit two only after that profile proof and the complete dark preflight pass.
  • Reconcile capacity and verify the marker, configuration digest, leased ledger, and enabled profile before the owner canary.
  • Disable Classic open before the v2 limit-three merge.
  • Keep every close and sweep path available.

Evidence pull request:

  • Record every live acceptance test.
  • Add requester runbook.
  • Add operator runbook.
  • Add incident and rollback runbook.
  • Add credential rotation runbook.
  • Add image update runbook.
  • Add signature, provenance, and vulnerability-exception runbooks.
  • Add cost and retention evidence.
  • Record the Classic cutover.

Disable Classic open only after cutover steps 1 through 12 pass.

Raise v2 capacity only after cutover steps 13 through 25 pass.

Complete three-session, admission, and seven-day evidence before v2 acceptance.

40. Closed implementation and activation inputs

Groups A through D can start without a live secret, trust mutation, cloud apply, or new spend.

Typed variables are valid for values that live activation supplies later.

No release, activation, or live-apply merge can contain an unresolved artifact digest, version, identity, scope, policy, or evidence reference.

Only the explicitly inert preparation contracts can contain typed null fields before their generated-ID review.

Group C cannot merge until its Group C source union in section 33 passes.

Group E cannot publish until the complete activation union in section 33 passes.

Groups E and F require the owner approvals defined in CLAUDE.md before each live mutation.

The following list is the complete external-input inventory.

40.1 Repository and human inputs

  1. Verify OpenMined organization ID 30185530, infrastructure repository ID 510317576, and control repository ID 1345147942 live.
  2. Supply one OpenMined repository administrator for the one-time empty-control-repository seed and protection operation.
  3. Supply two control-repository maintainers who cannot approve their own protected environment deployment.
  4. Pin the exact firecall-live environment ID, reviewers, wait timer, branch rule, and disabled self-review and administrator bypass.
  5. Pin the exact required checks, CODEOWNERS teams, branch protections, workflow IDs, and reusable-workflow commits in both repositories.
  6. Pin the locked break-glass issue ID, permitted GitHub actor IDs, different-owner approver IDs, and retention settings.
  7. Pin reviewed platform-owner canonical @openmined.org emails and their derived development subject keys. Keep Azure audit-reader identities as a separate operational input.
  8. Verify all three Classic state keys, the workflow ID, workflow path, historical FIC, source commit, bounded resources, and current zero-proof contract.
  9. Supply the final v1 source and live inventory used by the migration and rollback tests.

40.2 GitHub application and OIDC inputs

  1. Create the broker App on OpenMined/firecall-control with only Actions read/write, Contents read, and Metadata read.
  2. Create the image-pin App on infrastructure and control with only Contents read/write, Pull requests read/write, and Metadata read.
  3. Create the root-publication App on infrastructure with only Contents read/write, Pull requests read/write, and Metadata read.
  4. Create the Classic-observer App on infrastructure with only Actions read, Contents read, and Metadata read.
  5. Create the control-observer App on control with only Actions read, Contents read, and Metadata read.
  6. Create the settings-observer App on infrastructure with only Metadata, Contents, Actions, and Administration read.
  7. Pin every App ID, installation ID, repository selection, permission response, webhook-disabled state, and owner.
  8. Supply and prove primary and secondary private-key slots for all six Apps.
  9. Verify flexible FIC support for exact sub, repository_id, repository_owner_id, and commit-pinned job_workflow_ref expressions. Also verify the exact issuer and one audience.
  10. Pin every live application, service-principal, FIC, role, scope, condition, and denial-canary result from sections 23 and 24.
  11. Keep pull-request jobs credentialless. Prove each reusable pre-login guard rejects wrong-event and historical-source-SHA runs. Prove each privileged FIC rejects pull-request and wrong-workflow tokens.

40.3 Cloudflare, domain, and browser-identity inputs

  1. Supply one platform-owned FIRECALL_APEX that is the apex of its complete Cloudflare zone.
  2. Supply a different registered FIRECALL_PREVIEW_APEX that is the apex of its complete Cloudflare zone.
  3. Require different effective registrable domains and no nested-zone or shared-cookie boundary.
  4. Pin the Cloudflare account ID, both zone IDs, zone names, nameservers, and account team domain.
  5. Prove active Universal SSL SAN coverage, public chain, TLS 1.3 SNI, and at least 30 days remaining for all host classes.
  6. Pin the existing email one-time PIN provider ID, type, test digest, issuer, JWKS URL, and auth domain.
  7. Prove the edge plan changes no identity provider or unrelated Access application.
  8. Record the exact live Cloudflare global-session duration in seconds and accept its account-wide effect.
  9. Pin each Firecall Access application ID, audience, one-hour duration, hostname, ordered policy, and fallback route.
  10. Supply primary and secondary slots for edge, read-only observer, and zone-policy API tokens.
  11. Give the selected edge token Access: Organizations, Identity Providers, and Groups Read for the existing one-time PIN provider lookup.
  12. Prove every token's exact account, zone, tunnel, Access, DNS, SSL, cache, ruleset, filter-list, and denial permissions.
  13. Supply the separate short-lived identity-canary token only when the normal protected edge workflow cannot perform the closed-state canary.
  14. Pin the two complete cache-phase imports and accept the zone-policy token's account Rulesets and Filter Lists authority.
  15. Accept that Cloudflare Access and Tunnel can observe authenticated terminal, log, preview, and application traffic.
  16. Accept the account-wide residuals of Access application mutation, tunnel mutation, organization reads, and the email-PIN provider.

40.4 Azure service identity inputs

Development human authentication needs no Entra, Conditional Access, PIM, license, directory administrator, or human approver input. The items below are only Azure workload-identity inputs.

  1. Pin the Entra tenant ID, Azure subscription ID, and every required service-principal object ID.
  2. Grant and prove only Application.ReadWrite.OwnedBy administrator consent to the trust-root identity.
  3. Grant and prove the trust observer's one exact Microsoft Graph read permission from section 24.1.5.
  4. Pin tenant service-principal IDs for Microsoft Graph and Microsoft Azure Management and verify their Microsoft publisher facts.
  5. Keep staging and production human-directory and PIM design outside the development activation input set.

40.5 Azure resource, cluster, and network inputs

  1. Pin East US, every resource-group ID, the existing ACR ID, and the existing Terraform-state storage-account ID.
  2. Supply globally available deterministic names for the Firecall StorageV2 account and Standard Key Vault.
  3. Accept Standard LRS single-region durability and Azure lifecycle deletion without a bounded completion SLA.
  4. Pin the Firecall storage account as Standard, LRS, hot, shared-key-disabled, and version-level-immutability-disabled.
  5. Pin the Key Vault as Standard with Azure RBAC, public endpoint, purge protection, 90-day soft delete, and software P-256 keys.
  6. Pin all container names, backend keys, legal holds, WORM policies, soft-delete settings, lifecycle filters, locks, and prevent_destroy controls.
  7. Approve replacement of inherited generic Blob roles with the section 15 container-and-path-scoped assignments.
  8. Supply time-limited owner access for the first standing state, key, data, legal-hold, edge-state, and trust adoption operations.
  9. Pin the AKS resource ID, tenant, server, port, CA bytes and digest, Kubernetes version, token audience, creation time, and contract generation.
  10. Pin the cluster UID, Azure CNI Overlay mode, Pod CIDR, Service CIDR, DNS Service IP, and reviewed Konnectivity tuple.
  11. Pin the AKS and Firecall VNet IDs, subnet IDs and prefixes, both peering IDs, and every peering flag.
  12. Pin the session subnet 10.240.0.0/27, source-path contract, private agent port 8443, and network-path generation.
  13. Pin the Firecall node pool name, Standard_D2ds_v5 SKU, one-node count, 64-GiB disk, exact label, taint, scheduler identity, and node Policy.
  14. Pin every required managed AKS DaemonSet tuple and accept its mutable managed image-tag residual.
  15. Prove the pinned AKS VM-SKU API returns unrestricted Standard_D2ds_v5 support for East US.
  16. Prove the 10-vCPU BS-family quota supports the current phase. Prove six free DDSv5-family vCPUs.
  17. Prove Total Regional vCPU and Standard public-IP headroom for node, image, and three-session phases.
  18. Pin every required Azure resource-provider registration and block automatic provider registration.
  19. Approve the fixed image-build and session Policy definitions, assignments, exemptions, and complete parameter digests.
  20. Approve registry-wide AcrPush for the one image builder and registry-wide AcrPull for sealer, pin, inventory, and rescan identities.
  21. Pin the External Secrets controller identity, exact Firecall-vault role, store name, and two permitted namespaces.
  22. Pin the exact SigNoz collector namespace, Pod labels, Service DNS name, TCP 4317, workload owners, and network policies.
  23. Pin the Azure Monitor metrics and scopes for node utilization and B2s session credit qualification.

40.6 Keys, exportable secrets, and trust inputs

  1. Create the six exact non-exportable software P-256 key objects from section 24.10.
  2. Supply the four client and origin root generations and their signed public-root bundle.
  3. Supply current and previous session-token HMAC slots and their initial selectors.
  4. Supply primary and secondary slots for all six GitHub App keys and all three stored Cloudflare API-token types.
  5. Supply the one versioned Cloudflare tunnel token and the three generated Access audience values.
  6. Pin all ten fixed Kubernetes certificate-leaf Secret names, selectors, issuers, consumers, and overlap periods.
  7. Pin request-key and CSRF-key version IDs, public digests, selectors, overlap windows, and protected control commits.
  8. Prove every credential through its permitted and forbidden consumer canaries before selection.
  9. Accept External Secrets Operator as a trust holder for every exportable Firecall secret.
  10. Keep every private value outside Git, Terraform inputs, state outputs, workflow artifacts, arguments, environments, and logs.

40.7 Supply-chain and image inputs

  1. Complete the section 33 acquisition union with no unresolved item.
  2. Pin every action SHA, executable digest, Terraform provider, Packer plugin, OCI object, Ubuntu package, Go module, pnpm tarball, browser, asset, font, scanner, rule, and database.
  3. Pin the Ubuntu marketplace publisher, offer, SKU, exact version, dated package snapshot, signed metadata, and all transitive packages.
  4. Pin every runtime base manifest, config, layer, signature, provenance statement, source, and license.
  5. Pin the GitHub runner image and record the host, Docker, and privileged-runner residuals.
  6. Pin the Sigstore TUF bootstrap, Fulcio roots, Rekor keys, sequential metadata, and every accepted signer identity.
  7. Pin the ACR repositories, Gallery, image definition, monotonically increasing image version, evidence digest, ETag, and ReadOnly lock.
  8. Publish current signed Trivy and ClamAV generations and one current complete security-rescan pass verdict.
  9. Supply reviewed, unexpired vulnerability exceptions for any accepted high finding.
  10. Prove all build and verification phases use only cached inputs and reject every network acquisition.

40.8 Notification, retention, and spend inputs

  1. Pin the exact Azure Action Group resource ID, subscription, resource group, name, ETag, owner group, and destination digest.
  2. Pin the SigNoz organization ID, contact-point UID, notification-policy UID, owner, ETags, and destination digest.
  3. Supply current sanitized readbacks and real receipt IDs for every required destination.
  4. Keep receiver addresses, webhook URLs, and notification credentials outside Git and Firecall runtime authority.
  5. Approve the locked 90-day WORM policies, legal holds, 30-day soft delete, reference graph, and deletion-target alerting.
  6. Accept that Azure lifecycle deletion has no bounded completion guarantee.
  7. Approve the USD 130 Firecall-owned-resource budget before image publication.
  8. Approve the additional USD 105 AKS-pool budget before pool activation.
  9. Approve the combined USD 235 monthly Firecall budget and documented exclusions.

40.9 Explicit residual acceptances

  1. Accept the public identity-authorized Storage and Key Vault endpoints required by GitHub-hosted runners.
  2. Accept registry-wide push and pull exposure because ACR cannot scope these identities to one repository.
  3. Accept default Reader, optional Writer, root TokenRequest use, and copied token residuals for each debug box.
  4. Accept the possible 24-hour cached managed-identity token after box-root compromise and role deletion.
  5. Accept Secret and ServiceAccount authority inside only effective write namespaces.
  6. Accept the optional SSH identity-bypass path, exact source /32, and box-local host-key lifecycle.
  7. Accept GitHub-hosted runner access to each short-lived credential used by its one privileged job.
  8. Accept the broker App's repository-wide Actions write authority in the control repository.
  9. Accept normal-control Activity Log and Azure Monitor metric reads at the exact Firecall resource scopes.
  10. Accept that an unrelated assignment at a permitted parent scope can be observed but never changed by the assignment auditor.
  11. Accept broad Cloudflare account metadata reads for tunnel, Access, organization, ruleset, and filter-list verification.
  12. Accept Standard LRS regional-loss risk, mutable managed-AKS system image tags, and no bounded lifecycle-deletion SLA.
  13. Accept that an exact bounded owner recovery operation can read the connector token during its temporary edge-state access.

40.10 Final activation evidence

  1. Every saved plan digest, source commit, workflow commit, role, scope, policy, and live readback must match protected source.
  2. Every positive and negative FIC, Graph, storage, Key Vault, ACR, AKS, Cloudflare, and GitHub App canary must pass.
  3. The notification, security, scanner, node-tuple, access-policy, control-monitor, and assignment-audit verdicts must be current.
  4. The first owner canary is the only operation that can publish the initial network-path generation marker.
  5. Later sessions require that marker and repeat the exact live network comparison.
  6. The three-session depleted-credit qualification, fourth-session denial, cleanup proof, and seven-day observation must pass.
  7. Classic open rejection, historical-token denial, zero-resource proof, and reduction-only identities must pass before capacity three.
  8. Do not delete Classic resources, state, evidence, or history without the separate explicit deletion approval.

41. Intended final result

After Group F, a developer opens one URL and creates a debug session.

The developer sees exact scope and an estimated maximum cost before creation.

The developer watches durable build stages.

The workspace opens only after Kubernetes and terminal probes pass.

The developer gets browser shell access without a kubeconfig.

The developer can use k9s, exact commands, logs, events, and workload details.

The developer can open prefilled SigNoz views.

The developer can share a co-debug link with another verified human.

The developer can preview selected HTTP workloads on an isolated domain.

The requester or platform owner can close the session.

Expiry stops browser and local terminal access even when the sweep is late.

Cleanup destroys the box, identity, roles, network, and disks.

Firecall reports success only after zero-resource proof.

42. References