Platform architecture — living document
One picture of what actually runs and what is designed but not yet built. Convention: solid lines/boxes = live today; dashed = planned/designed. Update this document in the same PR as any change that alters the shape drawn here (new add-on, new stack, edge change) — it is part of the finishing routine alongside the ledger and roadmap. Rationale lives in the ADRs (0001 structure, 0002 delivery, 0003 portal, 0004 tenancy, 0005 gateway); this doc shows the resulting shape only.
Last verified against the live cluster: 2026-08-05.
1. Delivery planes — how anything gets anywhere
Two planes, per ADR-0002: Terraform makes cloud resources; Argo CD makes
cluster state. Nothing deploys by hand. The Argo-side file structure —
recipes/tickets/robots/fences, and the "how do I add…" recipes for platform,
cluster, and tenant changes — is documented in
kubernetes/README.md,
in the infrastructure repository.
flowchart LR
dev([engineer PR]) --> gh[GitHub<br/>OpenMined/infrastructure]
subgraph ci [Generic Terraform — PR credentialless, main OIDC]
check["generic Terraform PR checks<br/>format · validate · contracts<br/>no Azure credential"]
apply["apply identity (Contributor,<br/>ABAC-whitelisted RBAC Admin)<br/>main: plan + auto-apply generic non-prod<br/>prod: two-phase /apply-prod"]
end
gh -- pull_request --> check
gh -- merge to main --> apply
apply --> az[(Azure<br/>rg / aks / acr / dns<br/>state: rg-tfstate-*)]
gh -- "watched by" --> argo[Argo CD<br/>argocd ns, aks-dev]
argo -- "root app → kubernetes/clusters/<cluster>" --> k8s[(cluster state)]
sf([tenant merge<br/>OpenMined/screamingface]) -- "path-filtered CI<br/>(their gate)" --> acr2[(ACR<br/>main-<sha40>)]
acr2 -- "Warehouse poll (10m)" --> kargo[Kargo v1.11 headless<br/>kargo ns · ADR-0008]
kargo -- "Freight → render gate →<br/>two-line pin PR (om-pulse-bot)" --> gh
gh -- "3 required checks green →<br/>git-merge-pr (machine)" --> argo
- Trust root (
ci-identities) is human-applied via the break-glass runbook — the one exception to no-local-apply. - This source removes the generic pull-request Azure OIDC identity.
- Activation must apply that trust reduction and prove the live identity absent.
- Firecall roots use separate state containers and protected identities.
- Other legacy pull-request OIDC paths remain outside this generic lane. Firecall Group E removes or isolates them before activation.
- Tenant continuous delivery (ADR-0008, live 2026-08-05): the platform is
never the gate. A ScreamingFace merge reaches dev in ~15–20 min with no
human: Kargo mints Freight from the full-SHA tag, writes both pin fields
from that one tag, renders their chart at the exact commit against our
values BEFORE any PR exists, opens a two-line PR and merges it itself once
the three required checks pass. Watched by two independent alarms
(pin-staleness, argo-health). Human promotion = rollback/break-glass only
(
scripts/ops/promote-dev.sh). - Dev stacks reference prod-owned resources (ACR, DNS zone) by data-source name lookup, never remote state.
Pull-request Preview rail — designed, qualification pending
This rail is implemented in source. It is not live until one labelled fixture passes creation, workload, access, telemetry, deletion, and expiry qualification.
flowchart LR
pr["ScreamingFace PR<br/>changed paths + exact SHA"] -.-> ci["selective tenant CI<br/>changed images only"]
ci -.-> pacr["Preview ACR<br/>PR image tags"]
pr -. "preview label" .-> appset["Argo PR ApplicationSet"]
appset -.-> parent["trusted parent<br/>infrastructure main"]
parent -.-> ns["access + workload namespaces<br/>policy + generated secrets"]
parent -.-> project["per-PR AppProject"]
project -.-> child["restricted child<br/>exact tenant SHA"]
child -.-> runtime["selected runtime<br/>PR images + reviewed dependencies"]
pacr -.-> runtime
cf["Cloudflare Access + tunnel"] -.-> envoy["protected Envoy Gateway"]
envoy -.-> runtime
envoy -.-> access["per-PR access server<br/>one-hour namespace credential"]
access -.-> api["AKS API"]
runtime -.-> signoz["SigNoz logs and traces"]
envoy -.-> signoz
classDef planned stroke-dasharray: 5 5
class pr,ci,pacr,appset,parent,ns,project,child,runtime,cf,envoy,access,api,signoz planned
The tenant workflow builds only changed images. Runtime profiles add required services without rebuilding unchanged dependencies. Argo deploys unreviewed workloads; pull-request CI receives no Kubernetes credential.
The trusted parent owns namespace boundaries. The restricted child can create approved workload kinds only. Per-PR generated secrets and disposable storage end with namespace deletion.
The developer guide gives the complete workflow. ADR-0014 records the host namespace decision and its security boundaries.
2. aks-dev — everything on the cluster today
flowchart TB
inet([internet]) --> lb[Azure LB / public IP]
subgraph aks [aks-dev — K8s 1.35, DASv5 pools, AAD RBAC only, workload identity]
subgraph kube-system
cilium[Cilium dataplane<br/>NetworkPolicy ENFORCED<br/>deny-test passed 2026-07-18]
end
subgraph ingress-nginx-ns [ingress-nginx ns]
nginx[ingress-nginx<br/>platform ingress class]
end
subgraph argocd-ns [argocd ns]
argo[Argo CD v3.4<br/>argocd.dev.openmined.org<br/>admin auth until 3c SSO]
end
subgraph cert-manager-ns [cert-manager ns]
cm[cert-manager<br/>Let's Encrypt DNS-01]
end
subgraph external-dns-ns [external-dns ns]
edns[external-dns]
end
subgraph eg-ns [envoy-gateway-system ns]
eg[Envoy Gateway controller<br/>+ Gateway API CRDs + GatewayClass<br/>cf-protected: ONE shared listener]
end
subgraph kargo-ns [kargo ns]
kargoc[Kargo v1.11 headless<br/>controller + admission only<br/>WI → AcrPull]
end
subgraph signoz-ns [signoz ns]
signoz[SigNoz + ClickHouse<br/>ADR-0007 · logs/traces/metrics]
end
subgraph tenant [tenant namespaces — PSA baseline ENFORCED, quotas, default-deny]
sfk[sf-kargo<br/>Project/Warehouses/Stages<br/>+ git write credential]
sfa[sf-aigw<br/>aigw + console + pg]
sff[sf-fusion<br/>url4-cloud + nats]
sfe[sf-edge<br/>cloudflared tunnel]
sfs[sf-scoreboard<br/>(leaderboard arrives)]
end
end
lb --> nginx --> argo
cm -- "TXT challenges (workload identity)" --> dns[(Azure DNS<br/>dev.openmined.org<br/>delegated from Route53)]
edns -- "A/CNAME records from Ingress" --> dns
aks -- "AcrPull (kubelet identity)" --> acr[(ACR acropenmined)]
argo -- "syncs kubernetes/apps/*:<br/>portal, status, docs (see §4)" --> apps[portal / status / docs ns]
Everything above was delivered through the loop it draws — and since
2026-08-05 the loop closed for tenants too (ADR-0008): every boundary object
in the tenant namespaces (netpols, quotas, secrets rail, the tunnel) rides
the PLATFORM rail (project: default); tenant projects can author workload
kinds only — NetworkPolicy and ExternalSecret are no longer on their
whitelist, the ClusterSecretStore is conditioned to named namespaces, and
PSA enforce: baseline refuses privilege chart-authored or otherwise.
3. Tenant edge — Envoy Gateway (ADR-0005; enforcement LIVE Ring 0, edge planned)
Three decoupled planes: Cloudflare = edge, Access (→ Keycloak at 3c) =
identity, Envoy Gateway = enforcement. One shared platform-owned Gateway per
tier; tenants attach routes from their own namespaces. First tier:
cf-protected — tunnel-only, no public IP exists for tenant workloads.
Solid below = proven live 2026-07-20 against the pt-echo sim-tenant with
a controllable test issuer (the full 10-check conformance suite passed).
Dashed = the real Cloudflare edge (Ring 1) + first real tenant, still ahead.
flowchart TB
user([tenant's users]) -.-> cf
subgraph cf [Cloudflare — real edge, Ring 1 planned]
access[Access: OTP login,<br/>per-tenant app → unique AUD]
tun[Tunnel]
end
access --> tun
subgraph aks [aks-dev]
subgraph edge [pulse-edge ns — platform-owned]
cfd[cloudflared ×2<br/>token via ESO/Key Vault]
gw[Envoy Gateway: cf-protected tier<br/>ClusterIP only, 2 replicas + PDB<br/>strips X-User-*/X-Service-* then re-injects<br/>from validated JWT — never trusts inbound]
end
subgraph pt [pt-echo — sim-tenant, default-deny netpol]
route[HTTPRoute + platform SecurityPolicy<br/>+ BackendTrafficPolicy rate limits]
echo[echo instrument]
iss[test issuer<br/>JWKS + good/bad tokens]
end
subgraph sf [sf-* tenant ns — construct LIVE #138: AppProject + default-deny; workloads/routes planned]
sfroute[HTTPRoute: fusion.dev.screamingface.ai /v1*]
svc[fusion / scoreboard / aigateway<br/>aigateway: NO public hostname]
end
nginx[ingress-nginx<br/>platform apps *.dev.openmined.org]
end
tun -.-> cfd
cfd -.-> gw
gw -->|"stamped SecurityPolicy (JWKS, AUD, strip+re-inject)"| route
route --> echo
iss -->|JWKS| gw
gw -. planned .-> sfroute -.-> svc
classDef planned stroke-dasharray: 5 5
class cf,access,tun,user,cfd,sf,sfroute,svc planned
Key invariants (ADR-0005 §9), all exercised by the Ring-0 suite: header strip
+ re-inject (forged X-User-Email overwritten by the claim); JWT validated at
the gateway, never in backends (no/expired/garbage → 401); per-tenant AUD (a
foreign-AUD token → 403); rate limit + per-identity descriptor isolation;
listener allowedRoutes (a foreign namespace's route → NotAllowedByListeners);
backends reachable only via the proxy (direct curl blocked).
Build order (ScreamingFace track B): ~~B1 Cilium~~ ~~B2 Envoy Gateway~~
~~B3a tier Gateway + enforcement (Ring 0)~~ ~~B4 ESO/Key Vault~~ ~~Ring 1:
real Cloudflare tunnel + Access~~ done 2026-07-22 → B2 of the fusion trial
(aigateway + /claude). Backends read only the four identity-contract headers
— what makes the test-issuer → Cloudflare → Keycloak (roadmap 3c) swaps
invisible to every tenant.
LIVE 2026-07-22: fusion.testsfapp.net (ScreamingFace trial) serves url4
through this exact path — Cloudflare Access → tunnel → cf-protected Envoy →
platform SecurityPolicy → url4 — proven with real requests. Note: the tier
Gateway uses ONE shared http listener + host-routing, not per-tenant
listeners (ADR-0005 §4, amended —
Envoy Gateway's JWT aggregation breaks across merged same-port listeners).
Squatting prevention moves to an admission policy (GA prerequisite).
4. PULSE portal — the hub (ADR-0003; 4a + 4b live 2026-07-19)
One URL for the platform's storefront: services, health, docs, inventory.
Four replaceable panes; three live, inventory (4c) next.
Hub + status delivered via the platform-apps ApplicationSet
(clusters/*/apps/*.yaml enablement — the platform-owned-workload rail).
flowchart TB
devs([developers]) --> hub
subgraph aks [aks-dev]
subgraph portal-ns [portal ns]
hub[Homepage — pulse.dev.openmined.org<br/>tiles auto-discovered from Ingress<br/>annotations cluster-wide]
end
subgraph status-ns [status ns]
gatus[Gatus — status.pulse.dev.openmined.org<br/>checks: argocd 200 + cert validity,<br/>portal 200, cluster API]
end
subgraph docs-ns [docs ns]
docsite[docs site — pulse.dev.openmined.org/docs<br/>MkDocs image by CI → ACR, dev pin<br/>self-bumps via CI PR — no hands]
end
argo[argocd — annotated so its<br/>tile self-populates]
end
hub --> gatus
hub --> docsite
hub --> argo
classDef planned stroke-dasharray: 5 5
Docs strategy (agreed 2026-07-19): this file stays the single all-diagrams dump until the pieces exist. After implementation it splits into one doc per product — the platform treats each capability (AKS clusters, gateways, delivery pipeline, portal) as a product a tenant takes a lease on — each product doc covering: what it is, philosophy/how we operate it, its diagram, and getting a lease (what you get, what you owe, how to sign, how it renews or ends). Plus one combined overview. Audience: developers. Open decision, parked: whether ingress-nginx survives once the Envoy tier gateways exist (platform consuming its own gateway product) — revisit after track B3.