Skip to content

ADR-0014: Pull-request Previews use isolated host namespaces

Date: 2026-08-24 Status: accepted Decider: Platform owner (bitsofsteve) Supersedes: ADR-0011

Context

ADR-0011 selected one shared vCluster. That design did not reach an implementation. It added a second Kubernetes API, syncer, destination, and credential path before one Preview could run.

The owner stopped the PVW program process on 2026-08-24. The owner requested a direct implementation with developer Kubernetes access, SigNoz links, and automatic deletion.

The verified platform already has a tainted Preview node pool, separate registry, restricted build identity, protected Gateway, wildcard edge, and SigNoz collectors.

Decision

Each pull request uses one untrusted workload namespace in the dev AKS cluster:

sf-preview-pr-<number>

Each pull request also uses one trusted credential namespace:

sf-preview-access-pr-<number>

The access namespace prevents pull-request workloads from selecting the developer ServiceAccount or calling the kubeconfig Service directly.

Two Argo CD Applications represent one Preview:

  • A trusted platform Application reads the infrastructure main branch.
  • A restricted child Application reads the exact pull-request commit.

One pull-request ApplicationSet creates the platform Application. The trusted platform chart creates the child after the exact AppProject. Reverse sync-wave deletion removes the child before the AppProject and namespaces.

The ApplicationSet authenticates with a dedicated ScreamingFace GitHub App. The App has read-only Contents and Pull requests permissions. Its installation targets only ScreamingFace/screamingface. Azure Key Vault stores the private key. External Secrets Operator creates the Argo repository credential.

The platform Application creates an exact per-PR AppProject. That project can write only approved namespaced workload kinds into its one workload namespace. It cannot create RBAC, secrets, policies, routes, or cluster resources.

The tenant workflow separates the build graph from the runtime graph:

  • preview-component-* labels select the smallest runnable dependency closure.
  • preview-image-* labels select the pull-request images to build and deploy.
  • Unchanged dependencies use explicit reviewed-main image pins from the platform registry.
  • Chart-only changes select a runtime profile and build no application image.

For example, an Engine change builds Engine and benchmark images. Its runtime also contains reviewed AI Gateway, NATS, Garage, and AI Gateway PostgreSQL. It does not contain Scoreboard or the AI Gateway UI.

The parent Application installs exact per-PR admission policies before it creates the child workload Application. Cluster admission enforces these invariants for every Preview Pod:

  • The Pod selects openmined.org/pool=preview.
  • The Pod tolerates workload=preview:NoSchedule.
  • The Pod cannot set spec.nodeName at creation.
  • The Pod uses an approved ServiceAccount and exact selected image reference.
  • Init containers and ephemeral containers use the same approved image list.
  • Only the Engine control plane can automount its namespace-scoped API token.

The Engine token can manage Runner Jobs only in its Preview namespace. A second admission policy constrains each Runner Job's identity, token, volumes, image, and node placement.

The developer downloads a one-hour, pod-bound ServiceAccount kubeconfig through Cloudflare Access. The endpoint also verifies a GitHub token against the exact pull-request author from the trusted ApplicationSet generator. It verifies the Access JWT and forwarded email consistency. It separately requires the GitHub /user login to equal the pull-request author. A RoleBinding grants that ServiceAccount operational access only inside the matching workload namespace. It cannot change RBAC, policies, quota, or namespaces.

The protected Gateway gives each selected public component a separate hostname. Engine, AI Gateway, AI Gateway UI, Scoreboard, and kubeconfig routes do not share names. Envoy verifies Cloudflare Access before it forwards each request.

The pull-request CI workflow receives no Kubernetes credential. It receives only Preview-registry push access. Argo CD performs deployment.

Default-deny NetworkPolicies block cross-namespace and private-network traffic. There is no blanket same-namespace rule. Explicit rules permit only selected database, AI Gateway, NATS, Garage, Kubernetes API, and telemetry paths. Only the protected Gateway reaches public Preview backends. Public egress excludes private, carrier-grade NAT, loopback, and link-local ranges.

Consequences

  • Preview creation uses namespace speed and the existing Argo destination.
  • Developers use normal local kubectl and k9s commands.
  • Namespace deletion revokes access and removes all disposable resources.
  • A compromised Preview can affect its own namespace and Preview nodes.
  • It cannot schedule onto live dev nodes or use another Preview namespace.
  • Kubernetes audit identifies a shared per-Preview ServiceAccount, not the individual developer.
  • The host API remains shared. Namespace RBAC and admission replace vCluster API isolation.
  • Applications must use Preview-tier credentials. No dev or production secret is copied.

Acceptance

The Preview architecture guide defines the end-to-end tests, developer runbook, operator runbook, cleanup proof, and rollback.

Rollback

Remove all active preview labels. Confirm both namespaces disappear for every pull request. Revert the Preview ApplicationSet and chart. The registry, node pool, identity, and budgets can remain without running code.