ADR-0012: Firecall v2 access broker and browser workspace
Date: 2026-08-13 Amended: 2026-08-14, 2026-08-25, 2026-08-26, 2026-08-28, 2026-09-02, 2026-09-03 Status: accepted Decider: Platform owner (bitsofsteve) Supersedes: Unmerged Firecall v1 proposal in PR #389
Context
Firecall Classic creates up to three temporary Azure debug VMs through a serialized GitHub Actions dispatch. Each VM identity receives AKS Cluster User access and Writer access in the reviewed profile namespaces. SSH is the only interactive path.
Each Classic session uses one exact Terraform state key:
azure/firecall/dev.tfstate.azure/firecall/dev-slot2.tfstate.azure/firecall/dev-slot3.tfstate.
Each new Classic session uses a resource group named
rg-firecall-dev-<session_id>-eastus.
A session created before the three-slot change can still use the static
rg-firecall-dev-eastus group. Close, sweep, migration queries, and zero proof
must include that group until live readback proves it absent.
The 2026-08-14 amendment proposed a portal, ttyd, browser VS Code, a dedicated
control repository, and separate broker and proxy services. No v1 product code
landed. The owner later selected a browser debug workspace like a cloud console.
The workspace must include terminals, k9s, workloads, logs, events, SigNoz
links, co-debugging, and bounded web previews. Browser VS Code is deferred.
This amendment accepts Firecall v2. It supersedes the unlanded v1 product design. The owner ordered implementation from the reviewed plan on 2026-08-25. Firecall Classic remains live until the v2 canary and cutover pass.
Decision
1. Keep the ephemeral box as the Kubernetes credential boundary
For each session, only its temporary box receives Kubernetes access.
The browser, broker, terminal proxy, readiness worker, Cloudflare connector, and controlled egress gateway receive no kubeconfig or Kubernetes token.
Each box receives one managed identity. That identity receives Cluster User on the exact dev AKS cluster. It receives Reader or Writer in each resolved namespace from the selected service profiles. The portal shows every namespace and effective access mode before creation.
Read is the default for each selected service. Reader can view most namespace objects, but it cannot view Secrets, roles, or role bindings.
The requester can select write for one or more selected services without human approval. Writer can read Secrets and run Pods as namespace ServiceAccounts. Firecall revokes its access path at close or expiry. It cannot revoke copied values or undo Kubernetes changes made during a session.
2. Split the control plane from the data plane
The control plane contains the portal broker, the isolated control repository, GitHub Actions, request and status storage, the leased capacity ledger, per-session Terraform state, and immutable audit records.
The data plane contains the terminal proxy, readiness worker, private session network, session box, box agent, root session supervisor, and dev AKS API.
The broker validates the human, profile, scope, request, and lifecycle action. It signs one strict request envelope and dispatches one fixed workflow. It cannot run Terraform, contact a box, or read Terraform state.
The terminal proxy validates the human and session on every route. It reads
trusted lifecycle and revocation records. It connects to the box agent through
mutual TLS on fixed private port 8443. It holds no GitHub or Kubernetes
credential.
The box agent exposes only bounded terminal, workload, log, event, preview, and revocation operations. The root supervisor accepts a fixed local protocol. It does not accept arbitrary root commands from the proxy.
3. Isolate session authority in OpenMined/firecall-control
The private OpenMined/firecall-control repository owns profiles, capacity,
session Terraform, request verification, open, close, sweep, and cleanup proof.
Its protected main workflows use Firecall-specific Azure OIDC identities.
OIDC means OpenID Connect. The target architecture gives pull-request code no
Azure credential. Group A does not provide that live guarantee. The
infrastructure repository keeps a legacy generic PR plan identity until Group
E removes it.
The repository contains no infrastructure-repository credential. The broker GitHub App is installed only on this control repository. The normal runner can manage only declared Firecall session resources, storage paths, and approved AKS role assignments.
Each session has one independent Terraform state and one resource identity. An atomic Blob lease protects the capacity ledger. Unreadable request, status, state, security, or capacity data blocks creation.
Normal close and sweep do not require a human approval. Reduction-only break glass remains available when normal control fails.
3.1 Development infrastructure identity
Decided by the platform owner on 2026-09-03. These points replace conflicting earlier text for the development environment:
- Cloudflare Access is the only developer authentication layer.
- No Entra application exists for developer login.
- No manually seeded
sp-firecall-trustapplication is created or required. - The existing main-branch Azure apply identity (
ARM_CLIENT_ID_APPLY) provisions the development infrastructure. The earlier precondition that retires the generic plan identity before any Firecall identity activates is waived for development; it stays a staging and production requirement. - Terraform creates the runtime identities: five user-assigned managed identities in the control root, each federated to one exact AKS ServiceAccount.
- Each managed identity receives only the Azure permissions its service needs: container-scoped storage roles on the first apply, and Key Vault secret- or key-scoped roles after the owner's Key Vault bootstrap.
- Managed identities are service credentials. They are not developer login applications.
- Development activation does not use the
firecall-liveGitHub environment. One saved-plan workflow with the plan-run, stack, commit, lock-digest, and plan-digest binding applies each development root. - Approval gates stay for the future staging and production paths.
- Destructive actions stay disabled.
On 2026-09-14, the owner approved a separate dev session workflow identity,
retirement of the legacy infrastructure PR trust and production-state read
grant, and one create/delete test session. The control root creates the
user-assigned identity id-firecall-session-workflow-dev. Its federation
accepts only repo:OpenMined@30185530/firecall-control@1345147942:ref:refs/heads/main.
It is not an Entra developer login or a replacement infrastructure apply
identity. It receives only reviewed dev lifecycle permissions. No production
workload change or general production permission follows from this approval.
Creating the identity alone does not enable its lifecycle or portal.
The parked trust root (terraform/stacks/azure/firecall-trust/prod) remains
in the repository as reference for the staging and production design.
4. Use Cloudflare email one-time PIN for development
Cloudflare Access authenticates development users through email one-time PIN.
The Access policy permits only the exact openmined.org email domain. A
one-time PIN proves current access to that corporate mailbox.
Development authentication does not use Entra, Conditional Access, PIM, or a human session approver. PIM means Microsoft Privileged Identity Management. PIM is not a development login or session gate.
The services validate the final signed Access JWT, exact issuer, exact one-item audience, time bounds, current key, non-empty subject, and exact normalized email domain. They do not trust forwarded identity headers by themselves. Development does not require Entra tenant, Entra object, or MFA claims.
Firecall derives the durable development human key from the lowercase canonical email address:
hex(SHA-256(UTF8("firecall-dev-email-subject-v1\0" || canonical_email)))
The local part permits only lowercase letters, digits, dots, underscores, and hyphens. Firecall rejects plus addresses and other routing forms. Developers must use their primary corporate address.
It uses the current Cloudflare subject only for short-lived browser bindings. Mailbox reassignment can transfer the durable development identity. This risk is accepted for development. An administrator-created alias or shared mailbox can also create a separate identity. Offboarding must close every known address, revoke its Cloudflare Access sessions, and disable the primary mailbox, aliases, and shared addresses.
The proxy rechecks requester, owner, collaborator, close, and offboarding state during active streams. It closes affected streams after revocation or an unreadable required record.
Staging and production are not part of this development release. Future staging and production Access applications require stronger corporate identity, MFA, and explicit environment reviewers. They must not reuse the development email-PIN policy.
4.1 Keep shared-account Cloudflare authority resource-scoped
The platform owner pre-creates the dedicated firecall-dev tunnel and the
three development Access applications. Each application uses a six-hour
session, the exact openmined.org email-domain Include rule, and the existing
one-time PIN provider as a Require rule.
The edge stack receives the tunnel ID and the three Access audiences. It
configures the existing tunnel and creates DNS records in only
openmined.tools and openmined.app. It never writes a Key Vault value. It
never creates, updates, imports, or deletes the tunnel or the three Access
applications.
The platform owner stores the tunnel connector token, the scoped Cloudflare API token, and the three audiences in the dedicated Firecall Key Vault after the control root creates that vault and before edge activation. Terraform never reads or replaces the connector token. The development apply identity reads only the API token when it plans or applies the edge root.
Cloudflare permissions for the automation principal must select only the existing Firecall tunnel and Access resources. Zone permissions must select only the two Firecall zones. Do not grant an account-level Tunnel, Access, or Zero Trust role to that principal.
This amendment supersedes plan text that assigns account-wide tunnel or Access mutation to Firecall. It also supersedes the earlier one-hour Access duration and Terraform-created edge objects.
5. Separate trusted controls from untrusted preview content
Use these separate origin classes:
portal.openmined.toolsfor session control.<session_id>.openmined.toolsfor one workspace.<preview_id>--<session_id>.openmined.appfor untrusted web content.
openmined.tools and openmined.app are different registered domains
and separate complete Cloudflare zones. One dedicated tunnel uses static routes
to in-cluster services. No Cloudflare credential reaches a session box.
The proxy uses strict host and path parsers, host-only cookies, Origin and CSRF
checks, no-store responses, fixed security headers, and route-specific Content
Security Policy. It removes Cloudflare and Firecall credentials before a box
request. Preview routes also remove cookies and block Set-Cookie.
The preview proxy accepts only reviewed namespace targets and ports. It permits HTTP and WebSocket previews. It does not permit arbitrary TCP forwarding.
6. Use a private, policy-controlled runtime
Firecall uses a dedicated VNet, private subnet, peering, and fixed NSGs. Credentialed cluster services use controlled DNS and HTTPS gateways. Proxy-to-box traffic stays private.
Session boxes use Azure DNS and direct public TCP 443 for development tools.
Normal tools use HTTPS, but the network controls do not enforce TLS or HTTP.
NSGs and the host firewall deny private-network and other public ports. A
compromised box can exfiltrate readable data or tunnel another protocol over
public TCP 443. Each session accepts this residual for its resolved read and
write scope.
Optional SSH is disabled by default. When enabled, it accepts one proved public
key from one verified public IPv4 /32. The provider bootstrap key is per-VM,
memory-only, removed before readiness, and never enters persistent state as
private material.
The control runtime uses one tainted fixed-size Firecall AKS node pool. Admission, scheduling policy, digest policy, network policy, and exact node-tuple evidence isolate Firecall workloads from tenant and preview nodes.
7. Make lifecycle, cleanup, and audit fail closed
Creation uses strict request receipts, one signed envelope, generation and ETag checks, a leased reservation, and monotonic lifecycle transitions.
ready requires independent terminal, Kubernetes, identity, certificate, and
route probes. A failed open starts cleanup. Capacity remains occupied until
zero-resource proof passes. Unproved cleanup publishes rollback_failed.
Close denies browser access before Azure deletion begins. The proxy and box also enforce absolute session expiry. The scheduled sweep reconciles expiry and orphaned state.
Firecall records lifecycle, identity, join, denial, revocation, and cleanup events in create-once audit storage. It records no terminal input, terminal output, Kubernetes logs, token, cookie, private key, or secret value.
Co-debugging has session-level attribution only. It does not identify the human who typed one command.
8. Require immutable, verified release inputs
Every action, tool, provider, package, module, browser asset, container, and VM input uses an exact version and digest. Builds use the checked-in offline input set. Releases include SBOMs, provenance, signatures, malware scans, and vulnerability verdicts.
Azure Policy and Kubernetes admission accept only published immutable images. A missing, stale, blocked, unsigned, or mismatched security verdict blocks a new session. Close and sweep remain available.
9. Migrate without shared resource ownership
Classic and v2 never manage the same resource, identity, or state key.
Classic supports three sessions before coexistence. Before v2 creation starts with a limit of two, protected Classic source reduces new-open admission to one. Close and sweep continue to inspect all three Classic state keys. Repeated readback must prove at most one active Classic session before v2 creation starts. This keeps the combined coexistence ceiling at three.
New Classic opens stop only after the v2 acceptance suite passes. Classic reduction paths remain until zero-resource proof passes.
The v2 limit becomes three only after Classic trust and resources reach the specified zero state. A three-session depleted-credit qualification and a seven-day observation period follow. Classic resource deletion needs separate explicit approval.
The migration contract defines the ordered gates and rollback.
10. Permit one bounded owner bootstrap
The first Firecall trust workflow cannot create its own trust. Some backend, Policy, pool, secret, and directory objects also need an authenticated owner before routine OIDC identities exist.
Permit only an active exact row in
platform/firecall/contracts/owner-bootstrap-operations.json.
Group A shipped that manifest with no active operation and granted no live
authority. Generation 2 consumes control-repository-bootstrap after its
protected-main read failed before any repository mutation. The retained
readback proves GitHub repository 1345147942 remained empty. Generation 4
consumes control-repository-bootstrap-retry after it wrote the exact seed
commit. The required validation failed because three local-storage tests
inherited the GitHub Actions marker. Its incident-bound readback proves the
failed check, exact seed tree, absent branch protection, and absent team
assignment. No owner operation remains active. Source commit
1baa8c328d5b4c32ba2e7a2eba466c5dc51ec8e7 remains the consumed source.
The follow-up source repairs the GitHub Actions test environment and defines
control-repository-bootstrap-finalize. That operation accepts only the exact
failed seed commit or its exact corrected successor. It also requires the
known failed GitHub Actions check before it replaces the failed seed. This
replacement uses GitHub's atomic beforeOid condition. A concurrent main
update cannot be overwritten. Generation 6 consumes this finalization after
the corrected seed and required check succeeded. GitHub rejected the request
that combined app-bound checks with an empty legacy contexts field. The
retained readback binds the exact live seed, successful check, protection
applied during diagnosis, and absent Engineering team assignment. No owner
operation remains active. Protected source
1689df9f6d4b8fe3315c9cfc4a73a9fa70f82e3d remains the consumed source.
The next protected source defines control-repository-bootstrap-complete.
Its live entry accepts only the exact corrected seed. It can repair protection
and assign Engineering without changing the protected seed. Generation 7
authorizes only this operation against repository 1345147942. Protected
source e66c831776ebd3cb0715898e3edf39e61d2c85e6 contains the reviewed source.
Generation 8 consumes the operation after GitHub applied exact protection and
Engineering maintain access. GitHub's CODEOWNERS diagnostics endpoint returned
404 after the mutations. The retained readback binds that response, exact
CODEOWNERS content, seed, required check, protection, and team access. No owner
operation remains active.
Each row binds one operation ID, exact script, subcommand, scopes,
role set, mutation class, and one-time or recovery state. A Terraform row also
binds a non-null root, state key, and saved-plan digest. A non-Terraform row
requires those fields to be null.
Adding or activating a row requires the same protected change to amend this
ADR and the Firecall bootstrap paragraph in CLAUDE.md. An incomplete,
disabled, missing, or mismatched row denies the operation. Each Terraform
operation uses remote state, a read-only lock file, and a reviewed saved plan
bound to the protected script commit.
Both documents contain one hidden authority marker. The marker binds the raw manifest digest, generation, source commit, and sorted active operation IDs. Credentialless CI rejects a missing or stale marker. Activation and consumption therefore require both documents to change with the manifest.
The manifest's source_commit pins one prior protected commit and its complete
repository tree. A later protected activation commit carries the rows and
cannot reference its own commit SHA. The fixed source_execution_mode is
sandboxed_source_commit_archive.
Every owner script runs only through
scripts/ops/firecall-owner-operation.sh. The wrapper verifies the current
active row and materializes source_commit through git archive. It rejects
unsafe archive paths and any Git metadata. Each operation profile extracts only
its fixed required paths. It executes the exact script inside a filesystem
sandbox. The pinned wrapper maps each exact (script, subcommand,
mutation_class) tuple to one closed sandbox profile. The caller cannot provide
a path or mount request. Each profile fixes source paths, host executable paths,
credential transport, file descriptor numbers, sandbox destinations, and API
origins. An absent mapping fails. The wrapper rejects a missing, linked, or
caller-writable executable. It rejects an undeclared input, inherited
credential agent, or unlisted environment variable. The owner host operating
system, kernel, system libraries, tools, DNS, and network remain explicit
privileged trust boundaries. A runtime-generated host digest cannot remove
that trust. The sandbox exposes the source archive read-only and one empty
operation scratch directory. It exposes no repository checkout, .git, home
directory, host temporary directory, unrelated credential cache, or container
socket. The control-repository profile exposes only the wrapper, bootstrap
script, and complete manifest-bound control seed from the pinned commit. The wrapper reads protected main through
authenticated GitHub REST against numeric repository ID 510317576. It
requires source_commit to be an ancestor of current protected main. CI
requires the wrapper and each active script to exist and match that commit. It
applies this rule while the current or prior protected manifest contains an
active row. Consume the row in a separate protected change before either file
changes.
A consumed row is a permanent tombstone. It binds the prior active generation, completion record, sanitized readback, and their exact-byte SHA-256 digests.
Each manifest change increments its generation by exactly one. Existing rows
never disappear or change their operation bindings. A new row can start
disabled or active. Existing rows can move only from disabled to active
or active to consumed. An unused active operation uses
revoked_without_execution with a matching zero-change readback.
The exception permits no generic root, discovery loop, unsaved apply, auto-approve, Terraform destroy, unapproved deletion, or unrelated resource change. Every temporary role is removed. The operation waits for issued tokens to expire before routine automation starts.
The repository operating contract still requires approval for trust, secrets, deletion, and new spend.
The three dev Kubernetes coordination Leases use the existing direct-cluster
delivery exception under kubernetes/bootstrap/. They do not use this
authenticated owner-operation mechanism. Argo CD 3.4 always prepends Lease to
its compiled core resource exclusions, so no argocd-cm value can make an
Application create or reconcile them.
Only kubernetes/bootstrap/firecall-state/bootstrap.sh can seed these objects.
It requires the exact AKS resource, endpoint, CA digest, and kube-system UID.
It also requires clean source at the exact commit used by both healthy Argo
applications. The live Argo configuration and v3.4.5 controller must retain
the default Lease exclusion. Both namespaces, all 25 activation data keys, and
the complete absence of activation admission must match the closed contract.
The script accepts each Lease only when absent or in its complete inert state.
It renders the fixed Kustomization once into a private file and compares that
render with a separate hard-coded object contract. It creates only missing
objects and never updates an existing Lease. This supports partial retry
without writing a current lock. It verifies all three objects afterward.
--verify requires every exact inert Lease and emits canonical non-secret
evidence. That evidence binds the source commit, cluster UID, Lease UIDs,
resource versions, and SHA-256 digests of sorted compact specs. Activation
source must pin and verify this evidence before it registers create-denying
admission.
The exact object set is firecall-certificate-rotator and
firecall-credential-rotation in firecall, plus
firecall-edge-operation in firecall-edge. The seed grants no identity,
RBAC, secret, edge, workload, or spend authority. It does not create an active
owner-operation row. The hidden authority markers continue to describe only
the manifest-governed owner-operation mechanism above.
No dormant recovery switch exists. Lease loss after activation requires a new incident-bound protected source change. Operators first close Firecall and any edge exposure. Protected GitOps then removes the activation overlays and exact admission and returns the applications to the closed baseline. Recovery invalidates every plan and proof bound to the old UID. The exact bootstrap can then create only the missing Lease and emit its new identity. Admission can return only after new live evidence passes. Operators never bypass admission.
11. Select reviewed service profiles beneath one session ceiling
The portal lets a developer select one or more service profiles for each session. A service profile is one reviewed file that names a service, one base session profile, and its required Kubernetes namespaces.
The existing SessionProfile remains the authority ceiling. It fixes the
cluster, VM, lifetime, roles, namespace ceiling, cost, and protocol versions.
A ServiceProfile can reduce that scope. It cannot expand it.
The browser sends selected names in service_profiles. It sends the write
subset in write_service_profiles. Read is the default. It cannot send a raw
namespace, role, cluster, or VM value. The broker loads every selected service
profile at the same exact control-repository commit as the base profile. It rejects an
unknown, disabled, duplicate, mismatched, or out-of-ceiling selection. It also
rejects a write selection that is not a selected service.
The broker sorts the selected names and creates the sorted unique namespace union. It calculates effective access for each namespace. Write wins when a read service and write service share a namespace. It binds the base profile revision and digest, each selected service profile digest and access mode, the namespace union, and effective namespace access into one canonical resolved scope. The broker signs that scope digest with the open request. The control runner recomputes the scope before Azure login.
The bootstrap payload, box-agent configuration, durable lifecycle status, and
workspace status preserve the same service bindings, access modes, namespace
union, effective namespace access, and scope_sha256. Each consumer recomputes
the digest before it accepts the scope. Join invitations and acknowledgements
bind scope_sha256, because two sessions can use the same base profile with
different service or access selections.
The control runner is the lifecycle-status producer. It must copy the signed scope into every status write before activation. Activation must also prove that no v2 status, invitation, or join record lacks the scope fields. A legacy record fails closed. The operator must drain or migrate such a record before activation.
One declarative service profile file is the onboarding adapter for a new service. The broker, portal, runner, and workspace contain no hard-coded service list. A service profile has no hidden dependency expansion. A developer must select every service they need.
Kubernetes namespace RBAC cannot separate two services in the same namespace.
The portal can present AIGW and AIGW UI as different choices, but both choices
grant the complete reviewed sf-aigw namespace scope. If either service uses
write, the effective sf-aigw access is write.
The first implementation keeps the existing status-driven creation flow. Email alerts and other durable notifications remain deferred until the priority create, use, and close path passes qualification.
Pre-build public bootstrap material (2026-09-10)
New VM images use firecall.bootstrap-material.v2. The previous format pins
the image's final evidence digest inside the image itself. That creates a
self-reference: image publication evidence cannot exist before the image.
V2 pins the planned gallery version, all public trust/source identities, and
complete canonical profile templates. Only imageEvidenceSha256 is empty in
each template. The final signed session payload supplies that digest after
publication. The parser reconstructs the final profile with that one value
and verifies its complete hash against the signed payload. It retains the
fixed image version, namespace ceilings, AKS connection, roots, role IDs,
resource limits, and protected workflow identity. No unsigned runtime file
can leave the parser. Existing v1 material keeps its original strict checks.
This permits one image to contain several reviewed base profiles without rebuilding for each session. It does not permit runtime profile replacement. A changed namespace ceiling or other template field requires a new image. Final evidence authenticity remains the protected lifecycle workflow's responsibility; the offline parser verifies its signature-covered digest and planned version, not a live Azure resource or scanner report. The lifecycle workflow must validate actual image publication evidence before signing.
Options considered
Styled ttyd or browser VS Code
- A styled
ttydpage is small, but it does not provide the required workload, log, event, collaboration, and preview experience. - Browser VS Code adds editor, file, extension, and workspace trust surfaces.
- A purpose-built xterm.js workspace is selected. Browser VS Code is deferred to v2.1.
Browser-held Kubernetes credentials
- A downloaded kubeconfig supports local tools.
- It also moves the Kubernetes credential outside the expiring box boundary.
- V2 keeps every Kubernetes token on the ephemeral box.
Broker-held Azure authority
- Direct provisioning reduces workflow latency.
- It gives a long-lived web service Azure and state credentials.
- V2 uses signed requests and protected GitHub OIDC workflows.
One service or browser origin
- One service and origin reduce deployment count.
- Compromised terminal or preview content could then reach session controls.
- V2 separates broker, proxy, workspace origins, and the preview domain.
Public proxy-to-box traffic
- Public
8443avoids VNet peering. - It exposes every box gateway to the Internet and makes network evidence weak.
- V2 uses private peering and fixed mutual TLS.
One shared Terraform state for all sessions
- A shared state simplifies a single-session design.
- It makes one state failure block unrelated session cleanup.
- Classic and v2 use one state per session. V2 also uses a leased capacity ledger.
Consequences
- Firecall v2 needs a control repository, Azure trust, standing storage, network resources, a dedicated AKS pool, two Cloudflare zones, and a sealed VM image.
- Groups B, C, and D can start only after the Group A contracts merge.
- Initial live trust, secret, edge, and spend changes need owner approval.
- Normal development session creation, read access, write selection, use, close, and expiry need no human approval.
- The complete release needs a separate preview domain and live qualification.
- The detailed compromise ceilings and accepted risks remain normative in the threat model.
- The implementation plan remains the detailed source for schemas, exact limits, permissions, tests, activation order, and external inputs.
Supersession and rollback
A later ADR can replace the portal, edge, control repository, proxy, or box model. It must preserve an available reduction path for every active session.
Reverting this document changes no live resource. Runtime rollback disables v2 creation, keeps v2 close and sweep available, and follows the migration contract. It does not restore the historical Classic credential.