Skip to content

Platform architecture — living document

One picture of what actually runs and what is designed but not yet built. Convention: solid lines/boxes = live today; dashed = planned/designed. Update this document in the same PR as any change that alters the shape drawn here (new add-on, new stack, edge change) — it is part of the finishing routine alongside the ledger and roadmap. Rationale lives in the ADRs (0001 structure, 0002 delivery, 0003 portal, 0004 tenancy, 0005 gateway); this doc shows the resulting shape only.

Last verified against the live cluster: 2026-08-05.

1. Delivery planes — how anything gets anywhere

Two planes, per ADR-0002: Terraform makes cloud resources; Argo CD makes cluster state. Nothing deploys by hand. The Argo-side file structure — recipes/tickets/robots/fences, and the "how do I add…" recipes for platform, cluster, and tenant changes — is documented in kubernetes/README.md, in the infrastructure repository.

flowchart LR
    dev([engineer PR]) --> gh[GitHub<br/>OpenMined/infrastructure]

    subgraph ci [Generic Terraform — PR credentialless, main OIDC]
        check["generic Terraform PR checks<br/>format · validate · contracts<br/>no Azure credential"]
        apply["apply identity (Contributor,<br/>ABAC-whitelisted RBAC Admin)<br/>main: plan + auto-apply generic non-prod<br/>prod: two-phase /apply-prod"]
    end

    gh -- pull_request --> check
    gh -- merge to main --> apply
    apply --> az[(Azure<br/>rg / aks / acr / dns<br/>state: rg-tfstate-*)]

    gh -- "watched by" --> argo[Argo CD<br/>argocd ns, aks-dev]
    argo -- "root app → kubernetes/clusters/&lt;cluster&gt;" --> k8s[(cluster state)]

    sf([tenant merge<br/>OpenMined/screamingface]) -- "path-filtered CI<br/>(their gate)" --> acr2[(ACR<br/>main-&lt;sha40&gt;)]
    acr2 -- "Warehouse poll (10m)" --> kargo[Kargo v1.11 headless<br/>kargo ns · ADR-0008]
    kargo -- "Freight → render gate →<br/>two-line pin PR (om-pulse-bot)" --> gh
    gh -- "3 required checks green →<br/>git-merge-pr (machine)" --> argo
  • Trust root (ci-identities) is human-applied via the break-glass runbook — the one exception to no-local-apply.
  • This source removes the generic pull-request Azure OIDC identity.
  • Activation must apply that trust reduction and prove the live identity absent.
  • Firecall roots use separate state containers and protected identities.
  • Other legacy pull-request OIDC paths remain outside this generic lane. Firecall Group E removes or isolates them before activation.
  • Tenant continuous delivery (ADR-0008, live 2026-08-05): the platform is never the gate. A ScreamingFace merge reaches dev in ~15–20 min with no human: Kargo mints Freight from the full-SHA tag, writes both pin fields from that one tag, renders their chart at the exact commit against our values BEFORE any PR exists, opens a two-line PR and merges it itself once the three required checks pass. Watched by two independent alarms (pin-staleness, argo-health). Human promotion = rollback/break-glass only (scripts/ops/promote-dev.sh).
  • Dev stacks reference prod-owned resources (ACR, DNS zone) by data-source name lookup, never remote state.

Pull-request Preview rail — designed, qualification pending

This rail is implemented in source. It is not live until one labelled fixture passes creation, workload, access, telemetry, deletion, and expiry qualification.

flowchart LR
    pr["ScreamingFace PR<br/>changed paths + exact SHA"] -.-> ci["selective tenant CI<br/>changed images only"]
    ci -.-> pacr["Preview ACR<br/>PR image tags"]
    pr -. "preview label" .-> appset["Argo PR ApplicationSet"]
    appset -.-> parent["trusted parent<br/>infrastructure main"]
    parent -.-> ns["access + workload namespaces<br/>policy + generated secrets"]
    parent -.-> project["per-PR AppProject"]
    project -.-> child["restricted child<br/>exact tenant SHA"]
    child -.-> runtime["selected runtime<br/>PR images + reviewed dependencies"]
    pacr -.-> runtime
    cf["Cloudflare Access + tunnel"] -.-> envoy["protected Envoy Gateway"]
    envoy -.-> runtime
    envoy -.-> access["per-PR access server<br/>one-hour namespace credential"]
    access -.-> api["AKS API"]
    runtime -.-> signoz["SigNoz logs and traces"]
    envoy -.-> signoz

    classDef planned stroke-dasharray: 5 5
    class pr,ci,pacr,appset,parent,ns,project,child,runtime,cf,envoy,access,api,signoz planned

The tenant workflow builds only changed images. Runtime profiles add required services without rebuilding unchanged dependencies. Argo deploys unreviewed workloads; pull-request CI receives no Kubernetes credential.

The trusted parent owns namespace boundaries. The restricted child can create approved workload kinds only. Per-PR generated secrets and disposable storage end with namespace deletion.

The developer guide gives the complete workflow. ADR-0014 records the host namespace decision and its security boundaries.

2. aks-dev — everything on the cluster today

flowchart TB
    inet([internet]) --> lb[Azure LB / public IP]

    subgraph aks [aks-dev — K8s 1.35, DASv5 pools, AAD RBAC only, workload identity]
        subgraph kube-system
            cilium[Cilium dataplane<br/>NetworkPolicy ENFORCED<br/>deny-test passed 2026-07-18]
        end
        subgraph ingress-nginx-ns [ingress-nginx ns]
            nginx[ingress-nginx<br/>platform ingress class]
        end
        subgraph argocd-ns [argocd ns]
            argo[Argo CD v3.4<br/>argocd.dev.openmined.org<br/>admin auth until 3c SSO]
        end
        subgraph cert-manager-ns [cert-manager ns]
            cm[cert-manager<br/>Let's Encrypt DNS-01]
        end
        subgraph external-dns-ns [external-dns ns]
            edns[external-dns]
        end
        subgraph eg-ns [envoy-gateway-system ns]
            eg[Envoy Gateway controller<br/>+ Gateway API CRDs + GatewayClass<br/>cf-protected: ONE shared listener]
        end
        subgraph kargo-ns [kargo ns]
            kargoc[Kargo v1.11 headless<br/>controller + admission only<br/>WI → AcrPull]
        end
        subgraph signoz-ns [signoz ns]
            signoz[SigNoz + ClickHouse<br/>ADR-0007 · logs/traces/metrics]
        end
        subgraph tenant [tenant namespaces — PSA baseline ENFORCED, quotas, default-deny]
            sfk[sf-kargo<br/>Project/Warehouses/Stages<br/>+ git write credential]
            sfa[sf-aigw<br/>aigw + console + pg]
            sff[sf-fusion<br/>url4-cloud + nats]
            sfe[sf-edge<br/>cloudflared tunnel]
            sfs[sf-scoreboard<br/>(leaderboard arrives)]
        end
    end

    lb --> nginx --> argo
    cm -- "TXT challenges (workload identity)" --> dns[(Azure DNS<br/>dev.openmined.org<br/>delegated from Route53)]
    edns -- "A/CNAME records from Ingress" --> dns
    aks -- "AcrPull (kubelet identity)" --> acr[(ACR acropenmined)]
    argo -- "syncs kubernetes/apps/*:<br/>portal, status, docs (see §4)" --> apps[portal / status / docs ns]

Everything above was delivered through the loop it draws — and since 2026-08-05 the loop closed for tenants too (ADR-0008): every boundary object in the tenant namespaces (netpols, quotas, secrets rail, the tunnel) rides the PLATFORM rail (project: default); tenant projects can author workload kinds only — NetworkPolicy and ExternalSecret are no longer on their whitelist, the ClusterSecretStore is conditioned to named namespaces, and PSA enforce: baseline refuses privilege chart-authored or otherwise.

3. Tenant edge — Envoy Gateway (ADR-0005; enforcement LIVE Ring 0, edge planned)

Three decoupled planes: Cloudflare = edge, Access (→ Keycloak at 3c) = identity, Envoy Gateway = enforcement. One shared platform-owned Gateway per tier; tenants attach routes from their own namespaces. First tier: cf-protected — tunnel-only, no public IP exists for tenant workloads.

Solid below = proven live 2026-07-20 against the pt-echo sim-tenant with a controllable test issuer (the full 10-check conformance suite passed). Dashed = the real Cloudflare edge (Ring 1) + first real tenant, still ahead.

flowchart TB
    user([tenant's users]) -.-> cf

    subgraph cf [Cloudflare — real edge, Ring 1 planned]
        access[Access: OTP login,<br/>per-tenant app → unique AUD]
        tun[Tunnel]
    end
    access --> tun

    subgraph aks [aks-dev]
        subgraph edge [pulse-edge ns — platform-owned]
            cfd[cloudflared ×2<br/>token via ESO/Key Vault]
            gw[Envoy Gateway: cf-protected tier<br/>ClusterIP only, 2 replicas + PDB<br/>strips X-User-*/X-Service-* then re-injects<br/>from validated JWT — never trusts inbound]
        end
        subgraph pt [pt-echo — sim-tenant, default-deny netpol]
            route[HTTPRoute + platform SecurityPolicy<br/>+ BackendTrafficPolicy rate limits]
            echo[echo instrument]
            iss[test issuer<br/>JWKS + good/bad tokens]
        end
        subgraph sf [sf-* tenant ns — construct LIVE #138: AppProject + default-deny; workloads/routes planned]
            sfroute[HTTPRoute: fusion.dev.screamingface.ai /v1*]
            svc[fusion / scoreboard / aigateway<br/>aigateway: NO public hostname]
        end
        nginx[ingress-nginx<br/>platform apps *.dev.openmined.org]
    end

    tun -.-> cfd
    cfd -.-> gw
    gw -->|"stamped SecurityPolicy (JWKS, AUD, strip+re-inject)"| route
    route --> echo
    iss -->|JWKS| gw
    gw -. planned .-> sfroute -.-> svc

    classDef planned stroke-dasharray: 5 5
    class cf,access,tun,user,cfd,sf,sfroute,svc planned

Key invariants (ADR-0005 §9), all exercised by the Ring-0 suite: header strip + re-inject (forged X-User-Email overwritten by the claim); JWT validated at the gateway, never in backends (no/expired/garbage → 401); per-tenant AUD (a foreign-AUD token → 403); rate limit + per-identity descriptor isolation; listener allowedRoutes (a foreign namespace's route → NotAllowedByListeners); backends reachable only via the proxy (direct curl blocked).

Build order (ScreamingFace track B): ~~B1 Cilium~~ ~~B2 Envoy Gateway~~ ~~B3a tier Gateway + enforcement (Ring 0)~~ ~~B4 ESO/Key Vault~~ ~~Ring 1: real Cloudflare tunnel + Access~~ done 2026-07-22 → B2 of the fusion trial (aigateway + /claude). Backends read only the four identity-contract headers — what makes the test-issuer → Cloudflare → Keycloak (roadmap 3c) swaps invisible to every tenant.

LIVE 2026-07-22: fusion.testsfapp.net (ScreamingFace trial) serves url4 through this exact path — Cloudflare Access → tunnel → cf-protected Envoy → platform SecurityPolicy → url4 — proven with real requests. Note: the tier Gateway uses ONE shared http listener + host-routing, not per-tenant listeners (ADR-0005 §4, amended — Envoy Gateway's JWT aggregation breaks across merged same-port listeners). Squatting prevention moves to an admission policy (GA prerequisite).

4. PULSE portal — the hub (ADR-0003; 4a + 4b live 2026-07-19)

One URL for the platform's storefront: services, health, docs, inventory. Four replaceable panes; three live, inventory (4c) next. Hub + status delivered via the platform-apps ApplicationSet (clusters/*/apps/*.yaml enablement — the platform-owned-workload rail).

flowchart TB
    devs([developers]) --> hub

    subgraph aks [aks-dev]
        subgraph portal-ns [portal ns]
            hub[Homepage — pulse.dev.openmined.org<br/>tiles auto-discovered from Ingress<br/>annotations cluster-wide]
        end
        subgraph status-ns [status ns]
            gatus[Gatus — status.pulse.dev.openmined.org<br/>checks: argocd 200 + cert validity,<br/>portal 200, cluster API]
        end
        subgraph docs-ns [docs ns]
            docsite[docs site — pulse.dev.openmined.org/docs<br/>MkDocs image by CI → ACR, dev pin<br/>self-bumps via CI PR — no hands]
        end
        argo[argocd — annotated so its<br/>tile self-populates]
    end

    hub --> gatus
    hub --> docsite
    hub --> argo

    classDef planned stroke-dasharray: 5 5

Docs strategy (agreed 2026-07-19): this file stays the single all-diagrams dump until the pieces exist. After implementation it splits into one doc per product — the platform treats each capability (AKS clusters, gateways, delivery pipeline, portal) as a product a tenant takes a lease on — each product doc covering: what it is, philosophy/how we operate it, its diagram, and getting a lease (what you get, what you owe, how to sign, how it renews or ends). Plus one combined overview. Audience: developers. Open decision, parked: whether ingress-nginx survives once the Envoy tier gateways exist (platform consuming its own gateway product) — revisit after track B3.