Skip to content

ADR-0012: Firecall v2 access broker and browser workspace

Date: 2026-08-13 Amended: 2026-08-14, 2026-08-25, 2026-08-26, 2026-08-28, 2026-09-02, 2026-09-03 Status: accepted Decider: Platform owner (bitsofsteve) Supersedes: Unmerged Firecall v1 proposal in PR #389

Context

Firecall Classic creates up to three temporary Azure debug VMs through a serialized GitHub Actions dispatch. Each VM identity receives AKS Cluster User access and Writer access in the reviewed profile namespaces. SSH is the only interactive path.

Each Classic session uses one exact Terraform state key:

  • azure/firecall/dev.tfstate.
  • azure/firecall/dev-slot2.tfstate.
  • azure/firecall/dev-slot3.tfstate.

Each new Classic session uses a resource group named rg-firecall-dev-<session_id>-eastus.

A session created before the three-slot change can still use the static rg-firecall-dev-eastus group. Close, sweep, migration queries, and zero proof must include that group until live readback proves it absent.

The 2026-08-14 amendment proposed a portal, ttyd, browser VS Code, a dedicated control repository, and separate broker and proxy services. No v1 product code landed. The owner later selected a browser debug workspace like a cloud console. The workspace must include terminals, k9s, workloads, logs, events, SigNoz links, co-debugging, and bounded web previews. Browser VS Code is deferred.

This amendment accepts Firecall v2. It supersedes the unlanded v1 product design. The owner ordered implementation from the reviewed plan on 2026-08-25. Firecall Classic remains live until the v2 canary and cutover pass.

Decision

1. Keep the ephemeral box as the Kubernetes credential boundary

For each session, only its temporary box receives Kubernetes access.

The browser, broker, terminal proxy, readiness worker, Cloudflare connector, and controlled egress gateway receive no kubeconfig or Kubernetes token.

Each box receives one managed identity. That identity receives Cluster User on the exact dev AKS cluster. It receives Reader or Writer in each resolved namespace from the selected service profiles. The portal shows every namespace and effective access mode before creation.

Read is the default for each selected service. Reader can view most namespace objects, but it cannot view Secrets, roles, or role bindings.

The requester can select write for one or more selected services without human approval. Writer can read Secrets and run Pods as namespace ServiceAccounts. Firecall revokes its access path at close or expiry. It cannot revoke copied values or undo Kubernetes changes made during a session.

2. Split the control plane from the data plane

The control plane contains the portal broker, the isolated control repository, GitHub Actions, request and status storage, the leased capacity ledger, per-session Terraform state, and immutable audit records.

The data plane contains the terminal proxy, readiness worker, private session network, session box, box agent, root session supervisor, and dev AKS API.

The broker validates the human, profile, scope, request, and lifecycle action. It signs one strict request envelope and dispatches one fixed workflow. It cannot run Terraform, contact a box, or read Terraform state.

The terminal proxy validates the human and session on every route. It reads trusted lifecycle and revocation records. It connects to the box agent through mutual TLS on fixed private port 8443. It holds no GitHub or Kubernetes credential.

The box agent exposes only bounded terminal, workload, log, event, preview, and revocation operations. The root supervisor accepts a fixed local protocol. It does not accept arbitrary root commands from the proxy.

3. Isolate session authority in OpenMined/firecall-control

The private OpenMined/firecall-control repository owns profiles, capacity, session Terraform, request verification, open, close, sweep, and cleanup proof.

Its protected main workflows use Firecall-specific Azure OIDC identities. OIDC means OpenID Connect. The target architecture gives pull-request code no Azure credential. Group A does not provide that live guarantee. The infrastructure repository keeps a legacy generic PR plan identity until Group E removes it.

The repository contains no infrastructure-repository credential. The broker GitHub App is installed only on this control repository. The normal runner can manage only declared Firecall session resources, storage paths, and approved AKS role assignments.

Each session has one independent Terraform state and one resource identity. An atomic Blob lease protects the capacity ledger. Unreadable request, status, state, security, or capacity data blocks creation.

Normal close and sweep do not require a human approval. Reduction-only break glass remains available when normal control fails.

3.1 Development infrastructure identity

Decided by the platform owner on 2026-09-03. These points replace conflicting earlier text for the development environment:

  1. Cloudflare Access is the only developer authentication layer.
  2. No Entra application exists for developer login.
  3. No manually seeded sp-firecall-trust application is created or required.
  4. The existing main-branch Azure apply identity (ARM_CLIENT_ID_APPLY) provisions the development infrastructure. The earlier precondition that retires the generic plan identity before any Firecall identity activates is waived for development; it stays a staging and production requirement.
  5. Terraform creates the runtime identities: five user-assigned managed identities in the control root, each federated to one exact AKS ServiceAccount.
  6. Each managed identity receives only the Azure permissions its service needs: container-scoped storage roles on the first apply, and Key Vault secret- or key-scoped roles after the owner's Key Vault bootstrap.
  7. Managed identities are service credentials. They are not developer login applications.
  8. Development activation does not use the firecall-live GitHub environment. One saved-plan workflow with the plan-run, stack, commit, lock-digest, and plan-digest binding applies each development root.
  9. Approval gates stay for the future staging and production paths.
  10. Destructive actions stay disabled.

On 2026-09-14, the owner approved a separate dev session workflow identity, retirement of the legacy infrastructure PR trust and production-state read grant, and one create/delete test session. The control root creates the user-assigned identity id-firecall-session-workflow-dev. Its federation accepts only repo:OpenMined@30185530/firecall-control@1345147942:ref:refs/heads/main. It is not an Entra developer login or a replacement infrastructure apply identity. It receives only reviewed dev lifecycle permissions. No production workload change or general production permission follows from this approval. Creating the identity alone does not enable its lifecycle or portal.

The parked trust root (terraform/stacks/azure/firecall-trust/prod) remains in the repository as reference for the staging and production design.

4. Use Cloudflare email one-time PIN for development

Cloudflare Access authenticates development users through email one-time PIN. The Access policy permits only the exact openmined.org email domain. A one-time PIN proves current access to that corporate mailbox.

Development authentication does not use Entra, Conditional Access, PIM, or a human session approver. PIM means Microsoft Privileged Identity Management. PIM is not a development login or session gate.

The services validate the final signed Access JWT, exact issuer, exact one-item audience, time bounds, current key, non-empty subject, and exact normalized email domain. They do not trust forwarded identity headers by themselves. Development does not require Entra tenant, Entra object, or MFA claims.

Firecall derives the durable development human key from the lowercase canonical email address:

hex(SHA-256(UTF8("firecall-dev-email-subject-v1\0" || canonical_email)))

The local part permits only lowercase letters, digits, dots, underscores, and hyphens. Firecall rejects plus addresses and other routing forms. Developers must use their primary corporate address.

It uses the current Cloudflare subject only for short-lived browser bindings. Mailbox reassignment can transfer the durable development identity. This risk is accepted for development. An administrator-created alias or shared mailbox can also create a separate identity. Offboarding must close every known address, revoke its Cloudflare Access sessions, and disable the primary mailbox, aliases, and shared addresses.

The proxy rechecks requester, owner, collaborator, close, and offboarding state during active streams. It closes affected streams after revocation or an unreadable required record.

Staging and production are not part of this development release. Future staging and production Access applications require stronger corporate identity, MFA, and explicit environment reviewers. They must not reuse the development email-PIN policy.

4.1 Keep shared-account Cloudflare authority resource-scoped

The platform owner pre-creates the dedicated firecall-dev tunnel and the three development Access applications. Each application uses a six-hour session, the exact openmined.org email-domain Include rule, and the existing one-time PIN provider as a Require rule.

The edge stack receives the tunnel ID and the three Access audiences. It configures the existing tunnel and creates DNS records in only openmined.tools and openmined.app. It never writes a Key Vault value. It never creates, updates, imports, or deletes the tunnel or the three Access applications.

The platform owner stores the tunnel connector token, the scoped Cloudflare API token, and the three audiences in the dedicated Firecall Key Vault after the control root creates that vault and before edge activation. Terraform never reads or replaces the connector token. The development apply identity reads only the API token when it plans or applies the edge root.

Cloudflare permissions for the automation principal must select only the existing Firecall tunnel and Access resources. Zone permissions must select only the two Firecall zones. Do not grant an account-level Tunnel, Access, or Zero Trust role to that principal.

This amendment supersedes plan text that assigns account-wide tunnel or Access mutation to Firecall. It also supersedes the earlier one-hour Access duration and Terraform-created edge objects.

5. Separate trusted controls from untrusted preview content

Use these separate origin classes:

  • portal.openmined.tools for session control.
  • <session_id>.openmined.tools for one workspace.
  • <preview_id>--<session_id>.openmined.app for untrusted web content.

openmined.tools and openmined.app are different registered domains and separate complete Cloudflare zones. One dedicated tunnel uses static routes to in-cluster services. No Cloudflare credential reaches a session box.

The proxy uses strict host and path parsers, host-only cookies, Origin and CSRF checks, no-store responses, fixed security headers, and route-specific Content Security Policy. It removes Cloudflare and Firecall credentials before a box request. Preview routes also remove cookies and block Set-Cookie.

The preview proxy accepts only reviewed namespace targets and ports. It permits HTTP and WebSocket previews. It does not permit arbitrary TCP forwarding.

6. Use a private, policy-controlled runtime

Firecall uses a dedicated VNet, private subnet, peering, and fixed NSGs. Credentialed cluster services use controlled DNS and HTTPS gateways. Proxy-to-box traffic stays private.

Session boxes use Azure DNS and direct public TCP 443 for development tools. Normal tools use HTTPS, but the network controls do not enforce TLS or HTTP. NSGs and the host firewall deny private-network and other public ports. A compromised box can exfiltrate readable data or tunnel another protocol over public TCP 443. Each session accepts this residual for its resolved read and write scope.

Optional SSH is disabled by default. When enabled, it accepts one proved public key from one verified public IPv4 /32. The provider bootstrap key is per-VM, memory-only, removed before readiness, and never enters persistent state as private material.

The control runtime uses one tainted fixed-size Firecall AKS node pool. Admission, scheduling policy, digest policy, network policy, and exact node-tuple evidence isolate Firecall workloads from tenant and preview nodes.

7. Make lifecycle, cleanup, and audit fail closed

Creation uses strict request receipts, one signed envelope, generation and ETag checks, a leased reservation, and monotonic lifecycle transitions.

ready requires independent terminal, Kubernetes, identity, certificate, and route probes. A failed open starts cleanup. Capacity remains occupied until zero-resource proof passes. Unproved cleanup publishes rollback_failed.

Close denies browser access before Azure deletion begins. The proxy and box also enforce absolute session expiry. The scheduled sweep reconciles expiry and orphaned state.

Firecall records lifecycle, identity, join, denial, revocation, and cleanup events in create-once audit storage. It records no terminal input, terminal output, Kubernetes logs, token, cookie, private key, or secret value.

Co-debugging has session-level attribution only. It does not identify the human who typed one command.

8. Require immutable, verified release inputs

Every action, tool, provider, package, module, browser asset, container, and VM input uses an exact version and digest. Builds use the checked-in offline input set. Releases include SBOMs, provenance, signatures, malware scans, and vulnerability verdicts.

Azure Policy and Kubernetes admission accept only published immutable images. A missing, stale, blocked, unsigned, or mismatched security verdict blocks a new session. Close and sweep remain available.

9. Migrate without shared resource ownership

Classic and v2 never manage the same resource, identity, or state key.

Classic supports three sessions before coexistence. Before v2 creation starts with a limit of two, protected Classic source reduces new-open admission to one. Close and sweep continue to inspect all three Classic state keys. Repeated readback must prove at most one active Classic session before v2 creation starts. This keeps the combined coexistence ceiling at three.

New Classic opens stop only after the v2 acceptance suite passes. Classic reduction paths remain until zero-resource proof passes.

The v2 limit becomes three only after Classic trust and resources reach the specified zero state. A three-session depleted-credit qualification and a seven-day observation period follow. Classic resource deletion needs separate explicit approval.

The migration contract defines the ordered gates and rollback.

10. Permit one bounded owner bootstrap

The first Firecall trust workflow cannot create its own trust. Some backend, Policy, pool, secret, and directory objects also need an authenticated owner before routine OIDC identities exist.

Permit only an active exact row in platform/firecall/contracts/owner-bootstrap-operations.json. Group A shipped that manifest with no active operation and granted no live authority. Generation 2 consumes control-repository-bootstrap after its protected-main read failed before any repository mutation. The retained readback proves GitHub repository 1345147942 remained empty. Generation 4 consumes control-repository-bootstrap-retry after it wrote the exact seed commit. The required validation failed because three local-storage tests inherited the GitHub Actions marker. Its incident-bound readback proves the failed check, exact seed tree, absent branch protection, and absent team assignment. No owner operation remains active. Source commit 1baa8c328d5b4c32ba2e7a2eba466c5dc51ec8e7 remains the consumed source. The follow-up source repairs the GitHub Actions test environment and defines control-repository-bootstrap-finalize. That operation accepts only the exact failed seed commit or its exact corrected successor. It also requires the known failed GitHub Actions check before it replaces the failed seed. This replacement uses GitHub's atomic beforeOid condition. A concurrent main update cannot be overwritten. Generation 6 consumes this finalization after the corrected seed and required check succeeded. GitHub rejected the request that combined app-bound checks with an empty legacy contexts field. The retained readback binds the exact live seed, successful check, protection applied during diagnosis, and absent Engineering team assignment. No owner operation remains active. Protected source 1689df9f6d4b8fe3315c9cfc4a73a9fa70f82e3d remains the consumed source. The next protected source defines control-repository-bootstrap-complete. Its live entry accepts only the exact corrected seed. It can repair protection and assign Engineering without changing the protected seed. Generation 7 authorizes only this operation against repository 1345147942. Protected source e66c831776ebd3cb0715898e3edf39e61d2c85e6 contains the reviewed source. Generation 8 consumes the operation after GitHub applied exact protection and Engineering maintain access. GitHub's CODEOWNERS diagnostics endpoint returned 404 after the mutations. The retained readback binds that response, exact CODEOWNERS content, seed, required check, protection, and team access. No owner operation remains active. Each row binds one operation ID, exact script, subcommand, scopes, role set, mutation class, and one-time or recovery state. A Terraform row also binds a non-null root, state key, and saved-plan digest. A non-Terraform row requires those fields to be null.

Adding or activating a row requires the same protected change to amend this ADR and the Firecall bootstrap paragraph in CLAUDE.md. An incomplete, disabled, missing, or mismatched row denies the operation. Each Terraform operation uses remote state, a read-only lock file, and a reviewed saved plan bound to the protected script commit.

Both documents contain one hidden authority marker. The marker binds the raw manifest digest, generation, source commit, and sorted active operation IDs. Credentialless CI rejects a missing or stale marker. Activation and consumption therefore require both documents to change with the manifest.

The manifest's source_commit pins one prior protected commit and its complete repository tree. A later protected activation commit carries the rows and cannot reference its own commit SHA. The fixed source_execution_mode is sandboxed_source_commit_archive.

Every owner script runs only through scripts/ops/firecall-owner-operation.sh. The wrapper verifies the current active row and materializes source_commit through git archive. It rejects unsafe archive paths and any Git metadata. Each operation profile extracts only its fixed required paths. It executes the exact script inside a filesystem sandbox. The pinned wrapper maps each exact (script, subcommand, mutation_class) tuple to one closed sandbox profile. The caller cannot provide a path or mount request. Each profile fixes source paths, host executable paths, credential transport, file descriptor numbers, sandbox destinations, and API origins. An absent mapping fails. The wrapper rejects a missing, linked, or caller-writable executable. It rejects an undeclared input, inherited credential agent, or unlisted environment variable. The owner host operating system, kernel, system libraries, tools, DNS, and network remain explicit privileged trust boundaries. A runtime-generated host digest cannot remove that trust. The sandbox exposes the source archive read-only and one empty operation scratch directory. It exposes no repository checkout, .git, home directory, host temporary directory, unrelated credential cache, or container socket. The control-repository profile exposes only the wrapper, bootstrap script, and complete manifest-bound control seed from the pinned commit. The wrapper reads protected main through authenticated GitHub REST against numeric repository ID 510317576. It requires source_commit to be an ancestor of current protected main. CI requires the wrapper and each active script to exist and match that commit. It applies this rule while the current or prior protected manifest contains an active row. Consume the row in a separate protected change before either file changes.

A consumed row is a permanent tombstone. It binds the prior active generation, completion record, sanitized readback, and their exact-byte SHA-256 digests.

Each manifest change increments its generation by exactly one. Existing rows never disappear or change their operation bindings. A new row can start disabled or active. Existing rows can move only from disabled to active or active to consumed. An unused active operation uses revoked_without_execution with a matching zero-change readback.

The exception permits no generic root, discovery loop, unsaved apply, auto-approve, Terraform destroy, unapproved deletion, or unrelated resource change. Every temporary role is removed. The operation waits for issued tokens to expire before routine automation starts.

The repository operating contract still requires approval for trust, secrets, deletion, and new spend.

The three dev Kubernetes coordination Leases use the existing direct-cluster delivery exception under kubernetes/bootstrap/. They do not use this authenticated owner-operation mechanism. Argo CD 3.4 always prepends Lease to its compiled core resource exclusions, so no argocd-cm value can make an Application create or reconcile them.

Only kubernetes/bootstrap/firecall-state/bootstrap.sh can seed these objects. It requires the exact AKS resource, endpoint, CA digest, and kube-system UID. It also requires clean source at the exact commit used by both healthy Argo applications. The live Argo configuration and v3.4.5 controller must retain the default Lease exclusion. Both namespaces, all 25 activation data keys, and the complete absence of activation admission must match the closed contract. The script accepts each Lease only when absent or in its complete inert state. It renders the fixed Kustomization once into a private file and compares that render with a separate hard-coded object contract. It creates only missing objects and never updates an existing Lease. This supports partial retry without writing a current lock. It verifies all three objects afterward.

--verify requires every exact inert Lease and emits canonical non-secret evidence. That evidence binds the source commit, cluster UID, Lease UIDs, resource versions, and SHA-256 digests of sorted compact specs. Activation source must pin and verify this evidence before it registers create-denying admission.

The exact object set is firecall-certificate-rotator and firecall-credential-rotation in firecall, plus firecall-edge-operation in firecall-edge. The seed grants no identity, RBAC, secret, edge, workload, or spend authority. It does not create an active owner-operation row. The hidden authority markers continue to describe only the manifest-governed owner-operation mechanism above.

No dormant recovery switch exists. Lease loss after activation requires a new incident-bound protected source change. Operators first close Firecall and any edge exposure. Protected GitOps then removes the activation overlays and exact admission and returns the applications to the closed baseline. Recovery invalidates every plan and proof bound to the old UID. The exact bootstrap can then create only the missing Lease and emit its new identity. Admission can return only after new live evidence passes. Operators never bypass admission.

11. Select reviewed service profiles beneath one session ceiling

The portal lets a developer select one or more service profiles for each session. A service profile is one reviewed file that names a service, one base session profile, and its required Kubernetes namespaces.

The existing SessionProfile remains the authority ceiling. It fixes the cluster, VM, lifetime, roles, namespace ceiling, cost, and protocol versions. A ServiceProfile can reduce that scope. It cannot expand it.

The browser sends selected names in service_profiles. It sends the write subset in write_service_profiles. Read is the default. It cannot send a raw namespace, role, cluster, or VM value. The broker loads every selected service profile at the same exact control-repository commit as the base profile. It rejects an unknown, disabled, duplicate, mismatched, or out-of-ceiling selection. It also rejects a write selection that is not a selected service.

The broker sorts the selected names and creates the sorted unique namespace union. It calculates effective access for each namespace. Write wins when a read service and write service share a namespace. It binds the base profile revision and digest, each selected service profile digest and access mode, the namespace union, and effective namespace access into one canonical resolved scope. The broker signs that scope digest with the open request. The control runner recomputes the scope before Azure login.

The bootstrap payload, box-agent configuration, durable lifecycle status, and workspace status preserve the same service bindings, access modes, namespace union, effective namespace access, and scope_sha256. Each consumer recomputes the digest before it accepts the scope. Join invitations and acknowledgements bind scope_sha256, because two sessions can use the same base profile with different service or access selections.

The control runner is the lifecycle-status producer. It must copy the signed scope into every status write before activation. Activation must also prove that no v2 status, invitation, or join record lacks the scope fields. A legacy record fails closed. The operator must drain or migrate such a record before activation.

One declarative service profile file is the onboarding adapter for a new service. The broker, portal, runner, and workspace contain no hard-coded service list. A service profile has no hidden dependency expansion. A developer must select every service they need.

Kubernetes namespace RBAC cannot separate two services in the same namespace. The portal can present AIGW and AIGW UI as different choices, but both choices grant the complete reviewed sf-aigw namespace scope. If either service uses write, the effective sf-aigw access is write.

The first implementation keeps the existing status-driven creation flow. Email alerts and other durable notifications remain deferred until the priority create, use, and close path passes qualification.

Pre-build public bootstrap material (2026-09-10)

New VM images use firecall.bootstrap-material.v2. The previous format pins the image's final evidence digest inside the image itself. That creates a self-reference: image publication evidence cannot exist before the image.

V2 pins the planned gallery version, all public trust/source identities, and complete canonical profile templates. Only imageEvidenceSha256 is empty in each template. The final signed session payload supplies that digest after publication. The parser reconstructs the final profile with that one value and verifies its complete hash against the signed payload. It retains the fixed image version, namespace ceilings, AKS connection, roots, role IDs, resource limits, and protected workflow identity. No unsigned runtime file can leave the parser. Existing v1 material keeps its original strict checks.

This permits one image to contain several reviewed base profiles without rebuilding for each session. It does not permit runtime profile replacement. A changed namespace ceiling or other template field requires a new image. Final evidence authenticity remains the protected lifecycle workflow's responsibility; the offline parser verifies its signature-covered digest and planned version, not a live Azure resource or scanner report. The lifecycle workflow must validate actual image publication evidence before signing.

Options considered

Styled ttyd or browser VS Code

  • A styled ttyd page is small, but it does not provide the required workload, log, event, collaboration, and preview experience.
  • Browser VS Code adds editor, file, extension, and workspace trust surfaces.
  • A purpose-built xterm.js workspace is selected. Browser VS Code is deferred to v2.1.

Browser-held Kubernetes credentials

  • A downloaded kubeconfig supports local tools.
  • It also moves the Kubernetes credential outside the expiring box boundary.
  • V2 keeps every Kubernetes token on the ephemeral box.

Broker-held Azure authority

  • Direct provisioning reduces workflow latency.
  • It gives a long-lived web service Azure and state credentials.
  • V2 uses signed requests and protected GitHub OIDC workflows.

One service or browser origin

  • One service and origin reduce deployment count.
  • Compromised terminal or preview content could then reach session controls.
  • V2 separates broker, proxy, workspace origins, and the preview domain.

Public proxy-to-box traffic

  • Public 8443 avoids VNet peering.
  • It exposes every box gateway to the Internet and makes network evidence weak.
  • V2 uses private peering and fixed mutual TLS.

One shared Terraform state for all sessions

  • A shared state simplifies a single-session design.
  • It makes one state failure block unrelated session cleanup.
  • Classic and v2 use one state per session. V2 also uses a leased capacity ledger.

Consequences

  • Firecall v2 needs a control repository, Azure trust, standing storage, network resources, a dedicated AKS pool, two Cloudflare zones, and a sealed VM image.
  • Groups B, C, and D can start only after the Group A contracts merge.
  • Initial live trust, secret, edge, and spend changes need owner approval.
  • Normal development session creation, read access, write selection, use, close, and expiry need no human approval.
  • The complete release needs a separate preview domain and live qualification.
  • The detailed compromise ceilings and accepted risks remain normative in the threat model.
  • The implementation plan remains the detailed source for schemas, exact limits, permissions, tests, activation order, and external inputs.

Supersession and rollback

A later ADR can replace the portal, edge, control repository, proxy, or box model. It must preserve an available reduction path for every active session.

Reverting this document changes no live resource. Runtime rollback disables v2 creation, keeps v2 close and sweep available, and follows the migration contract. It does not restore the historical Classic credential.