ADR-0004: Platform tenancy — namespaces first, vcluster by trigger
Date: 2026-07-17 Status: accepted · amended 2026-07-18: the per-tenant Gateway edge (§1 table row 5, §2) is superseded by ADR-0005's shared tiered gateway. Tenants receive platform-authored hostname routes on a shared listener and Gateway. All other primitives here stand. · amended 2026-07-31: the NetworkPolicy row (§1 table row 3) is refined by ADR-0006 — the openings on top of the default-deny baseline are platform-authored, not "the tenant's own policies"; ADR-0006 also fixes which objects inside a tenant namespace the platform owns rather than the tenant's chart. · amended 2026-08-12: ADR-0008 supersedes manual pin approval for routine tenant application promotion. Every other primitive here stands.
Context
ScreamingFace is the first tenant: a product team whose workloads run on our
clusters but whose identity is their own — their domain (fusion.screamingface.ai),
their edge auth (Cloudflare Access), their release cadence. Tenant traffic does
not flow through *.dev.openmined.org, and tenant prod/dev map onto OUR
prod/dev clusters. The Fusion Edge brief (workload-deploy.md) also demands
enforced NetworkPolicies and a Gateway API data path. We need a tenancy model
that serves this tenant now and the next one without redesign.
Decision
1. A tenant is: an AppProject + a namespace set + a vault path + an edge
| Primitive | Concretely | Enforces |
|---|---|---|
| Argo AppProject per tenant | screamingface: allowed source repos (their repo + ours), allowed destination namespaces (sf-* only) |
a tenant's manifests cannot land outside their namespaces |
| Namespace set, tenant-prefixed | sf-edge, sf-url4, sf-aigw (one prefix, N namespaces per their architecture) |
naming collisions impossible; policy selectors trivial |
| Default-deny NetworkPolicies stamped per tenant namespace | deny-all ingress+egress baseline; the platform authors every approved opening | tenant workloads are isolated by default, including from other tenants |
Key Vault path per tenant (sf-* secret names) + ESO |
tenant Secrets materialize only in tenant namespaces | secret isolation without per-tenant vaults (upgrade path exists) |
| Edge per tenant | platform-authored routes and policies attach tenant hostnames to one shared tier Gateway | the platform enforces hostname, identity, and traffic policy |
2. Two data paths, by audience — this is ADR-0002 §4's pre-planned second path
Amended by ADR-0005 (2026-07-18): the tenant path below is now a shared, platform-owned tier Gateway and listener with hostname routes, not a
Gatewayper tenant. See ADR-0005 for the tier model, SecurityPolicy ownership, identity contract, and topology decisions.
- Platform apps (Argo CD, portal, docs): ingress-nginx under
*.dev.openmined.org. Unchanged. - Tenant apps: Gateway API via a shared, pinned Envoy Gateway controller
(
kubernetes/infrastructure/envoy-gateway/, enabled per cluster). The platform owns the shared tier Gateway, routes, policies,GatewayClass, and CRD versions. Tenant workloads receive traffic through approved routes.
3. Tenant DNS and TLS are the tenant's
fusion.screamingface.ai lives in THEIR Cloudflare zone. Preferred connectivity:
cloudflared Tunnel per tenant edge (outbound-only — no public origin IP, no
inbound exposure, no cert coupling; Envoy's listener can stay HTTP behind the
tunnel per the brief; the tunnel mechanics are ADR-0005 §7). Fallback: proxied
DNS to a dedicated LB IP. Either way, external-dns and our Azure zones are NOT
involved in tenant hostnames. Tenant dev hosts (<svc>.dev.screamingface.ai,
e.g. fusion.dev.screamingface.ai) are the tenant's naming call, terminating
on our dev cluster; prod is <svc>.screamingface.ai.
4. Tenant environments map to platform clusters
Tenant-dev runs on aks-dev, tenant-prod on aks-prod (roadmap 3b — a
prerequisite for any tenant prod; until it exists, trials run on aks-dev and
are labeled trials). Kargo writes immutable version-pin pull requests and
merges them only after required checks pass. ADR-0008 governs routine tenant
promotion. Production infrastructure changes remain human-gated.
5. vcluster: defined triggers, not a default
Namespaces + AppProject + NetworkPolicy + Gateway API cover Fusion v1 entirely. Introduce a vcluster for a tenant only when one of these becomes true:
- CRD/operator conflict — the tenant needs cluster-scoped APIs (versions or operators) that collide with platform or other-tenant installs.
- Self-service k8s API — the tenant needs to create cluster-scoped objects or namespaces themselves, i.e. wants their own control plane to administer.
- Control-plane blast radius — tenant workloads are hostile enough that a shared API server is itself the risk (webhooks, aggressive controllers).
Untrusted code execution (url4's /python, subprocess workers — RSV-1) is
NOT a vcluster trigger: vclusters share the node kernel. That is solved at the
node layer — AKS Pod Sandboxing (Kata microVMs) on a dedicated tainted pool —
which the platform provides per-tenant when RSV-1 lands.
6. Cluster capability required: enforced NetworkPolicies
aks-dev today runs Azure CNI overlay with networkPolicy: none — policies are
silently unenforced, which invalidates any tenant isolation claim. The aks
module gains network policy enforcement (Cilium dataplane preferred), applied
to dev first; enforcement becomes part of the module's default shape for every
future cluster. No tenant goes live before a positive deny-test passes.
Consequences
- First-tenant cost is real (netpol enablement, Envoy add-on, AppProject scaffolding); every later tenant is a prefix, a project, a vault path, and a platform-authored attachment to a shared Gateway.
- Two edge technologies to operate (nginx + Envoy Gateway). Accepted: different audiences, and consolidation onto Gateway API later remains open.
- Argo remains the single deploy path for tenant manifests (their repo as a source, scoped by AppProject) — tenant CI never touches the cluster.
- Rate-limit Redis and cloudflared run per tenant edge (in
sf-edge), not as shared platform services — isolation over efficiency at this scale.