Skip to content

ADR-0002: Delivery architecture — two planes, three clusters, one engine

Date: 2026-07-14 Status: proposed Amended: 2026-08-12. ADR-0008 supersedes this ADR's human-review rule for routine tenant application promotion. Tenant release gates drive automatic Kargo promotion. Production infrastructure remains human-gated.

Amended: 2026-08-25. ADR-0012 removes pull-request entry from the generic Terraform Azure OIDC workflow. Its pull-request lane runs credentialless source validation. Protected main plans and applies generic non-production roots. Production uses the human-gated saved-plan dispatch. Other legacy pull-request OIDC paths and the generic plan identity remain ADR-0012 Group E removal blockers.

Context

The infra repo must service engineers in other repos: deploy their packages on merge/release, give them preview environments on PR open, and host shared platform services (identity via Keycloak, ingress, DNS, TLS, monitoring) — without the infra team becoming a ticket queue. ADR-0001 fixed where code lives; this ADR fixes how changes flow from any repo to running infrastructure.

This is the target shape, not a build order: docs/roadmap.md sequences the minimal slice of it at each step, and several pieces named here (Front Door/WAF, vcluster, OpenCost) are explicitly deferred until pulled by a real need.

Decision

1. Two planes with different tempos

Plane Tool Tempo "dev" means "prod" means
Infrastructure Terraform days–weeks credentialless PR checks; plan and auto-apply on merge manual saved-plan workflow_dispatch
Delivery Kargo and Argo CD minutes Kargo promotes; Argo syncs dev Kargo promotes releases after required checks; Argo syncs production

They meet at exactly one seam: Terraform creates a cluster and bootstraps Argo CD onto it (the only sanctioned kubectl apply, in kubernetes/bootstrap/). Terraform never manages what runs inside clusters; Argo never touches cloud resources.

2. One GitOps engine: Argo CD (not Argo + Flux)

Two reconciliation brains on the same clusters means two mental models for every serviced engineer and split-brain drift. Argo CD covers reconciliation, ApplicationSets, per-cluster enablement, and a UI serviced engineers can inspect. Kargo discovers release artifacts and creates the checked, immutable pin changes defined by ADR-0008.

3. Three clusters, each with a one-line purpose

Cluster Purpose Notes
aks-dev churn: PR previews, main-tracking deploys, vclusters dev data but prod service — other teams' workflow depends on it
aks-prod serviced workloads (APIs, products of onboarded repos) hostile-ish neighbor profile: code we didn't write arrives automatically
aks-identity Keycloak and NOTHING else tier-0; see §5

vclusters are workloads, not infrastructure: deployed by Argo on aks-dev for (a) previews needing their own CRDs/operators and (b) self-service tenant "clusters". Prod stays flat — namespaces, NetworkPolicies, node pools; no nested control planes in the critical path.

4. Shared platform services, consumed not installed

kubernetes/infrastructure/ (ingress-nginx, cert-manager, external-dns, monitoring) installs once per cluster via sync waves, before apps. Apps consume declaratively: an Ingress with a host + a cert-manager annotation yields DNS records (external-dns writes to Terraform-owned zones) and TLS (ClusterIssuer) automatically. Multiple domains (openmined.org, screamingface.ai, …) coexist on one controller via host routing; zones live wherever DNS is hosted, with --domain-filter scoping. Escape hatch if the shared data path ever worries us: a second ingress class with its own IP — pre-planned, not pre-built.

5. Identity is the exception: dedicated cluster, private by default

The IdP is tier-0 (compromise mints tokens for everything) and aks-prod hosts semi-trusted workloads — so Keycloak gets its own minimal cluster, separating control plane, CRDs, RBAC, nodes, and upgrade lifecycle at once. Exposure:

  • Internal ingress (private LB); internal realm and /admin + /realms/master are never internet-reachable (split-horizon DNS for auth.openmined.org).
  • The external-apps realm is exposed ONLY via Front Door/App Gateway + WAF with a path allowlist (/realms/external-apps/*), rate-limited. The public surface is itemized in one Terraform file.
  • Certs via DNS-01 (private endpoints can't answer HTTP-01).
  • Keycloak's Postgres: stacks/azure/keycloak-infra/prod, private endpoint, reachable only from the identity cluster subnet (lifecycle split per ADR-0001).

6. The serviced-repo contract (DevEx)

Onboarding repo X = image build to ACR in their CI + one app directory and one ApplicationSet entry here (to become the /onboard-repo skill). Then:

Event in repo X What happens Human involved?
PR opened ApplicationSet PR generator: ephemeral env on aks-dev (pr-<n> namespace or vcluster), live URL pr-<n>.<x>.dev.openmined.org, destroyed on close no
merge to main image main-<full-40-hex-sha> → ACR → Kargo render and checked pin change → Argo syncs aks-dev no
release Kargo promotes the accepted release through checked, immutable pin PRs no platform human for routine application promotion; ADR-0008 governs
needs auth OIDC client against Keycloak (dev realm on previews/dev; prod realm after promotion) onboarding only

Prod's deploy history is this repo's git log; rollback is git revert.

The invariant

Every layer leaves a repository diff and an immutable audit path.

Production infrastructure requires one deliberate human apply approval. Routine tenant application promotion follows ADR-0008 and uses tenant release gates plus checked Kargo automation. There is no path to production that does not leave a diff.

Consequences

  • aks-dev availability matters beyond "dev": previews and main-tracking are other teams' daily workflow. Monitor it like a product.
  • The generic Terraform pull-request lane runs credentialless source validation only. Protected main and production dispatches use the apply identity. Other legacy pull-request OIDC paths remain until Group E.
  • Kargo writes narrow pin pull requests through a scoped GitHub App. Required checks pass before Kargo merges them.
  • Running one engine means betting on Argo CD; Flux-only features (e.g. OCI artifact sources) are foregone until/unless Argo grows them.
  • Three clusters ≈ three × (control plane + system nodes) baseline cost; accepted for the tier-0 isolation argument in §5. Everything else stays consolidated.