ADR-0002: Delivery architecture — two planes, three clusters, one engine
Date: 2026-07-14 Status: proposed Amended: 2026-08-12. ADR-0008 supersedes this ADR's human-review rule for routine tenant application promotion. Tenant release gates drive automatic Kargo promotion. Production infrastructure remains human-gated.
Amended: 2026-08-25. ADR-0012 removes pull-request entry from the generic
Terraform Azure OIDC workflow. Its pull-request lane runs credentialless
source validation. Protected main plans and applies generic non-production roots. Production
uses the human-gated saved-plan dispatch. Other legacy pull-request OIDC paths
and the generic plan identity remain ADR-0012 Group E removal blockers.
Context
The infra repo must service engineers in other repos: deploy their packages on merge/release, give them preview environments on PR open, and host shared platform services (identity via Keycloak, ingress, DNS, TLS, monitoring) — without the infra team becoming a ticket queue. ADR-0001 fixed where code lives; this ADR fixes how changes flow from any repo to running infrastructure.
This is the target shape, not a build order: docs/roadmap.md sequences the minimal slice of it at each step, and several pieces named here (Front Door/WAF, vcluster, OpenCost) are explicitly deferred until pulled by a real need.
Decision
1. Two planes with different tempos
| Plane | Tool | Tempo | "dev" means | "prod" means |
|---|---|---|---|---|
| Infrastructure | Terraform | days–weeks | credentialless PR checks; plan and auto-apply on merge | manual saved-plan workflow_dispatch |
| Delivery | Kargo and Argo CD | minutes | Kargo promotes; Argo syncs dev | Kargo promotes releases after required checks; Argo syncs production |
They meet at exactly one seam: Terraform creates a cluster and bootstraps
Argo CD onto it (the only sanctioned kubectl apply, in kubernetes/bootstrap/).
Terraform never manages what runs inside clusters; Argo never touches cloud
resources.
2. One GitOps engine: Argo CD (not Argo + Flux)
Two reconciliation brains on the same clusters means two mental models for every serviced engineer and split-brain drift. Argo CD covers reconciliation, ApplicationSets, per-cluster enablement, and a UI serviced engineers can inspect. Kargo discovers release artifacts and creates the checked, immutable pin changes defined by ADR-0008.
3. Three clusters, each with a one-line purpose
| Cluster | Purpose | Notes |
|---|---|---|
aks-dev |
churn: PR previews, main-tracking deploys, vclusters | dev data but prod service — other teams' workflow depends on it |
aks-prod |
serviced workloads (APIs, products of onboarded repos) | hostile-ish neighbor profile: code we didn't write arrives automatically |
aks-identity |
Keycloak and NOTHING else | tier-0; see §5 |
vclusters are workloads, not infrastructure: deployed by Argo on aks-dev for (a) previews needing their own CRDs/operators and (b) self-service tenant "clusters". Prod stays flat — namespaces, NetworkPolicies, node pools; no nested control planes in the critical path.
4. Shared platform services, consumed not installed
kubernetes/infrastructure/ (ingress-nginx, cert-manager, external-dns,
monitoring) installs once per cluster via sync waves, before apps. Apps consume
declaratively: an Ingress with a host + a cert-manager annotation yields DNS
records (external-dns writes to Terraform-owned zones) and TLS (ClusterIssuer)
automatically. Multiple domains (openmined.org, screamingface.ai, …) coexist on
one controller via host routing; zones live wherever DNS is hosted, with
--domain-filter scoping. Escape hatch if the shared data path ever worries
us: a second ingress class with its own IP — pre-planned, not pre-built.
5. Identity is the exception: dedicated cluster, private by default
The IdP is tier-0 (compromise mints tokens for everything) and aks-prod hosts semi-trusted workloads — so Keycloak gets its own minimal cluster, separating control plane, CRDs, RBAC, nodes, and upgrade lifecycle at once. Exposure:
- Internal ingress (private LB); internal realm and
/admin+/realms/masterare never internet-reachable (split-horizon DNS for auth.openmined.org). - The
external-appsrealm is exposed ONLY via Front Door/App Gateway + WAF with a path allowlist (/realms/external-apps/*), rate-limited. The public surface is itemized in one Terraform file. - Certs via DNS-01 (private endpoints can't answer HTTP-01).
- Keycloak's Postgres:
stacks/azure/keycloak-infra/prod, private endpoint, reachable only from the identity cluster subnet (lifecycle split per ADR-0001).
6. The serviced-repo contract (DevEx)
Onboarding repo X = image build to ACR in their CI + one app directory and one
ApplicationSet entry here (to become the /onboard-repo skill). Then:
| Event in repo X | What happens | Human involved? |
|---|---|---|
| PR opened | ApplicationSet PR generator: ephemeral env on aks-dev (pr-<n> namespace or vcluster), live URL pr-<n>.<x>.dev.openmined.org, destroyed on close |
no |
| merge to main | image main-<full-40-hex-sha> → ACR → Kargo render and checked pin change → Argo syncs aks-dev |
no |
| release | Kargo promotes the accepted release through checked, immutable pin PRs | no platform human for routine application promotion; ADR-0008 governs |
| needs auth | OIDC client against Keycloak (dev realm on previews/dev; prod realm after promotion) | onboarding only |
Prod's deploy history is this repo's git log; rollback is git revert.
The invariant
Every layer leaves a repository diff and an immutable audit path.
Production infrastructure requires one deliberate human apply approval. Routine tenant application promotion follows ADR-0008 and uses tenant release gates plus checked Kargo automation. There is no path to production that does not leave a diff.
Consequences
- aks-dev availability matters beyond "dev": previews and main-tracking are other teams' daily workflow. Monitor it like a product.
- The generic Terraform pull-request lane runs credentialless source
validation only. Protected
mainand production dispatches use the apply identity. Other legacy pull-request OIDC paths remain until Group E. - Kargo writes narrow pin pull requests through a scoped GitHub App. Required checks pass before Kargo merges them.
- Running one engine means betting on Argo CD; Flux-only features (e.g. OCI artifact sources) are foregone until/unless Argo grows them.
- Three clusters ≈ three × (control plane + system nodes) baseline cost; accepted for the tier-0 isolation argument in §5. Everything else stays consolidated.