ADR-0011: Preview isolation architecture — five walls on the shared dev cluster
Date: 2026-08-13 Status: superseded by ADR-0014 Decider: Platform owner (bitsofsteve)
Context
Preview environments run unreviewed pull-request code on the same cluster as the live tenant dev environment. Without designed walls, preview pods could reach dev services, the node metadata endpoint, and in-cluster control machinery.
Source facts that bind the design (verified 2026-08-13):
- The cluster is Azure CNI overlay with the Cilium network-policy engine. Every network wall rides Cilium enforcement; with the engine off, policies apply cleanly and enforce nothing. Live deny-probes are therefore part of the acceptance evidence, not optional.
- The AKS module has no autoscaler, no taints, and no third pool. Its header mandates growing flags deliberately. The preview pool is a named module extension.
- The admission guardrails bind through the platform's namespace label set. A namespace without those labels silently receives no guardrails.
- Kubernetes network-policy allows are additive. A carve-out in an allow rule is not a deny; only a deny-precedence rule survives future sloppy allows.
- Azure has no metadata-service session mechanism. The metadata token is
per-node: any pod that reaches the endpoint can mint tokens for every
identity on that node. Pod-security baseline forbids host namespaces,
which closes the
hostNetworkbypass at admission. - A dedicated preview cluster was rejected earlier in the program: multi- cluster delivery machinery does not exist, and a second Argo installation decision (STP-D12) belongs to the STP program and stays open.
This decision is routed by the preview-environments program (PVW-D03 through work package PVW-M0-W03; links below). The owner selected Option D on 2026-08-13 — the shared-vcluster variant proposed during the decision discussion.
Decision
Previews run on the shared dev cluster behind five walls.
The developer contract this serves
A developer with an open preview can: see their own workload, test a quick change, and feel like they are in their own cluster — terminal, k9s, and observability included. Developers are not expected to install CRDs or operators. Their credential never touches the host API.
Wall 1 — separate metal
A dedicated preview node pool with a taint, labels, and the cluster autoscaler (minimum one warm node). Preview workloads schedule only there; dev workloads never share a kernel with pull-request code. The pool arrives through a deliberate AKS-module extension (autoscaler, taints, labels, additional-pool support).
Wall 2 — separate API: one shared preview vcluster
One long-lived vcluster hosts all previews. Each preview is one namespace inside it. The developer receives a namespace-admin kubeconfig for the vcluster — the host API is unreachable by construction.
- The vcluster deploys as one platform app from the pinned upstream chart: rootless from first deploy, embedded datastore, syncer pinned to namespace-scoped RBAC, CRD sync off. These are creation-time decisions; changing them means redeploying the vcluster.
- The vcluster registers once as one stable Argo CD destination in the destination contract. Per-preview delivery is an Argo Application into a vcluster namespace — no dynamic destinations, which keeps the destination-validation model intact.
- Teardown of a preview is deletion of its vcluster namespace; the syncer garbage-collects the host objects.
- Escape hatch and upgrade path: a preview that needs exclusive CRDs or operators gets a dedicated per-PR vcluster on request (the Option A shape, recorded below). If shared-control-plane contention ever hurts, per-PR vclusters are the upgrade, not a redesign.
Wall 3 — network walls
- Default-deny ingress and egress on the preview host namespace, and per-preview policies inside the vcluster keyed on the synced namespace labels.
- In-cluster allows are selector-based only (the tight kube-dns pattern). Internet egress uses the standard private-range and link-local carve-outs.
- One cluster-wide Cilium egress-deny on the metadata endpoint (169.254.169.254/32), scoped to preview namespaces. Deny precedence means no future allow can re-open it. Implementation verifies whether the Azure host-agent endpoint (168.63.129.16) can join the deny without breaking platform dependencies.
Wall 4 — resource fences
Quota and limit-range on the host preview namespace bound the aggregate. Per-preview quotas are stamped inside the vcluster at namespace creation; developers hold namespace-admin and cannot remove them. Pod security: baseline enforced (parity with dev), restricted warn and audit.
Wall 5 — nothing valuable inside
Preview pods pull only from the preview registry (ADR-0010), use only preview-tier secrets, and have no path to the platform vault. The platform label set is stamped on preview namespaces so admission guardrails bind. The runner-job admission constraint learns the preview shapes in the guardrail implementation milestone.
Residual risks, named
- The host API server is shared with the vcluster syncer. Users never hold host credentials; the syncer's host Role stays namespace-scoped and reviewed.
- The shared vcluster control plane is a shared failure domain for preview debugging: a preview that hammers the API blips the preview API for all previews. Running workloads and dev are untouched. Mitigation: control plane resource limits and API fairness settings.
- Node compromise defeats pod-level controls. Accepted: the taint confines it to preview nodes, which carry nothing valuable.
- Kubelet identity on preview nodes can pull dev images. Pull, not push; images must carry no secrets.
Wall tests (acceptance evidence, each observable)
- An untolerated pod never schedules onto a preview node.
- A developer's vcluster kubeconfig cannot read any host object.
- The metadata probe times out from a preview pod — including from a pod that holds the internet-egress allow.
- A cross-namespace probe from a preview to a dev service is denied.
- A quota-busting pod is rejected inside its preview namespace.
- A
hostNetworkpod is refused at admission. - Preview teardown leaves zero cluster-scoped residue on the host.
Options considered
Option A — one vcluster per pull request
- Benefit: strongest isolation; developer holds full admin of a disposable cluster; exclusive CRDs.
- Cost or risk: control-plane boot on every preview (slower); one control plane per preview (costlier); per-PR Argo destinations would be dynamic, which the destination contract exists to forbid.
- Kept as the escape hatch and upgrade path.
Option B — plain host namespaces, no vcluster
- Benefit: least machinery, fastest to ship.
- Cost or risk: developer credentials land on the host API; k9s and self-service degrade; external identities on the platform cluster.
- Rejected: fails the developer contract and widens host exposure.
Option C — the managed IMDS-restriction feature as the metadata wall
- Benefit: node-level enforcement.
- Cost or risk: preview-labeled, requires node reimage, incompatible with add-on classes the platform may need.
- Rejected.
Option D — one shared vcluster, namespace per preview (selected)
- Benefit: warm control plane (previews create at namespace speed); one control plane's cost; teardown is namespace deletion; one stable Argo destination; full developer contract without host credentials.
- Cost or risk: shared preview control plane (mitigated above); cluster- scoped installs are shared (out of contract; escape hatch covers them).
Consequences
- PVW-M1 gains the AKS-module extension package (autoscaler, taints, labels, additional pool) and the preview-pool rollout.
- PVW-M2 builds: the shared vcluster platform app, the one-time destination registration, the per-preview namespace machinery with stamped fences and labels, the metadata deny policy, and the TTL reaper working inside the vcluster.
- Implementation checks recorded now: verify per-preview network isolation via synced namespace labels; verify rootless mode against the pinned chart tag; benchmark the shared control plane before setting fairness limits; verify the host-agent endpoint deny; decode the developer kubeconfig path end to end.
- A pre-pull mechanism for common base images on the preview pool is the remaining speed lever; decide during PVW-M2 if image pull dominates creation time.
- Tests and operating work: the seven wall tests above become the package acceptance evidence, re-run at every guardrail change.
- Remaining risks: the residual list above.
Supersession and rollback
A later accepted ADR can replace this architecture — the recorded upgrade path is per-PR vclusters (Option A) if shared-control-plane contention outgrows its mitigations. Deleting the preview pool, the vcluster app, and the preview namespaces removes the architecture; nothing outside previews references it.
Superseded architecture
The suspended PVW records remain as history and reference. ADR-0014 replaced this design with host namespaces on 2026-08-24. Its implementation and runbook live in the Preview architecture guide.