ADR-0005: Multitenant gateway platform — shared tiered gateways, three planes
Date: 2026-07-18
Status: accepted in PR #34 · amended 2026-07-22 (§4: single shared listener adopted now,
not per-tenant listeners — forced by an Envoy Gateway JWT-aggregation bug; see
§4) · amended 2026-07-23 (§7: the mechanics of a federated tenant on the shared
tier — a tenant-owned cloudflared into the shared Gateway; ScreamingFace is
the first) · amended 2026-08-13 (§10: the preview exposure — a dedicated
preview subdomain with one preview-tier Access app and AUD per tenant) ·
amended 2026-08-19 (§11: dev-tier single sign-on — one tenant AUD for
user-facing dev services, with privileged surfaces like aigw-admin kept
separate; staging/prod keep per-service AUDs) · amended 2026-08-31 (§13: the
public-intake route class — the first fully-anonymous WRITE surface; dev-only
interim on the shared listener behind a BLOCKING cf-public split gate before
any staging/prod exposure) · §14 proposed 2026-09-08 (self-service route
declarations — a tenant declares a hostname in one file, the platform renders
the route and the tier policy, CI and a native ValidatingAdmissionPolicy enforce
hostname ownership, closing the control §4 deferred; Status: proposed until the
owner accepts) · §15 proposed 2026-09-22 (the cf-public gateway — the
public-intake class leaves the shared listener onto its own fleet and its own
tenant-owned tunnel per environment, with fail-closed true-client-IP detection;
analytics.dev.screamingface.ai joins the member list; staging admitted behind
the split, production behind a ladder; Status: proposed until the owner accepts)
· §16 accepted for implementation 2026-10-05 by owner decision (whole-public
hostnames: each leaderboard hostname moves whole to cf-public on dev,
staging and production — anonymous reads under a per-address and a
route-wide budget, writes behind a path-scoped Access application and the
Envoy-verified Access JWT; amends §14 and §15; the production conditions in
§16(g) are signed by the owner before the production Access edit)
Amends: ADR-0004 (supersedes its §2 per-tenant Gateway edge; everything else in
0004 — AppProject, namespaces, vault paths, netpol, vcluster triggers — stands)
Context
ADR-0004 gave each tenant their own Gateway API Gateway (own Envoy proxy
fleet) in their edge namespace. Design alignment on the gateway platform
(The original alignment draft is historical private material and has no
authority over this ADR.)
concluded that per-tenant gateways don't scale — the product is a small menu of
shared, platform-owned gateways that many tenants attach to — and contributed
the tier framing, the identity contract, and the Keycloak migration path. This
ADR distills that into decisions, resolving its open questions D1–D6.
Decision
1. Three decoupled planes
| Plane | Component | Owns | Never does |
|---|---|---|---|
| Edge | Cloudflare (Access + Tunnel) | DDoS, TLS at edge, first auth gate, origin unreachability | source of truth for identity |
| Identity | Cloudflare Access (OTP) now → Keycloak later (roadmap 3c) | who is this: users, SSO, tokens | route traffic |
| Enforcement | Envoy Gateway (Gateway API CRDs, pinned version) | JWT validation, routing, rate limits, header contract, tenant isolation | trust a header it didn't set |
Each plane is replaceable without rearchitecting the other two — this is what makes the Keycloak migration a config change (§7).
2. Shared tiered gateways, not per-tenant
One platform-owned Gateway per tier, in a platform edge namespace; a
dedicated proxy fleet per tier (Envoy Gateway mergeGateways remains a later
cost knob). Tenants attach HTTPRoutes from their own namespaces. First tier:
cf-protected (tunnel-only exposure, Access JWT auth, no public IP
exists). Future tiers (internal for east-west, partner-mtls) are defined by
the five-part tier contract (exposure · auth · traffic guarantees ·
availability · admission) and named by guarantee, not implementation.
3. A tenant, at the gateway layer, is {domain, aud, upstream} + a tier
One Access application per tenant → one unique AUD, which is the tenant boundary at the auth layer: a token minted for tenant A is cryptographically invalid on tenant B's routes. Never share an AUD; never wildcard audiences. The tuple survives the Keycloak migration (aud becomes a client ID). Everything tenant-side of onboarding is a PR of tenant-namespace resources, stamped from the /onboard-repo template.
4. Routing granularity: per-tenant listeners now, virtual hosts later
Now (tenant count ~1, no admission controller on the cluster): each tenant
gets a hostname-scoped listener on the tier Gateway with allowedRoutes
selecting only that tenant's namespaces — hostname squatting is structurally
impossible, no extra machinery.
Later (trigger: approaching the 64-listener cap, hostname churn, or Kyverno/ OPA landing for other reasons): collapse to one listener with host-header routing + an admission policy asserting a namespace may only use hostnames from its registered domain list. The switch is invisible to tenants.
AMENDED 2026-07-22 — "later" is now, forced by an Envoy Gateway bug. The per-tenant-listener model does not work with per-tenant JWT on Envoy Gateway. Multiple hostname listeners on the same port are merged by Envoy into one listener, and EG's JWT-requirement aggregation across that merge is broken: only the first merged listener's
SecurityPolicyrequirement is written into the shared JWT filter'srequirement_map, so every other tenant's route fails closed withFailed JWT authentication: Wrong requirement_name(deterministic; confirmed via Envoyconfig_dump; not fixed by controller/proxy restarts; area of envoyproxy/gateway#8313, which upstream closed as won't-fix). Discovered standing up the second JWT tenant (ScreamingFace fusion) alongside the pt-echo sim-tenant.Decision: adopt the single-listener host-routing model now. One listener → no Envoy merge → JWT aggregates across all tenant routes correctly. Tenants scope by their
HTTPRoutehostname. The lost property — structural hostname-squatting prevention (per-listenerallowedRoutes) — is deferred to an admission policy (Kyverno/OPA: a namespace may only use hostnames from its registered domain list). Accepted gap for the trial / small trusted tenant set; admission control is a tracked follow-up and a GA prerequisite. Revisit per-tenant listeners only if EG fixes the merge aggregation.
5. SecurityPolicy is platform-owned
The policy object is stamped by the platform from the onboarding template;
tenants cannot weaken their own tier guarantee. Invariant fields (JWT
extraction from Cf-Access-Jwt-Assertion, issuer/JWKS, claim→header mapping
to the identity contract, header stripping on the route) are template-fixed.
Tenants supply values: their AUD, allowed CORS origins, optional
claim-based authorization rules (e.g. required group), and their rate-limit
numbers (which live in BackendTrafficPolicy, tenant-tunable within tier
ceilings).
6. The identity contract (adopt now, never break)
| Header | Meaning | Source |
|---|---|---|
X-User-Email |
verified human identity | JWT email claim |
X-User-Id |
stable subject id | JWT sub |
X-Tenant |
tenant context of the route | route/policy config |
X-Service-Id |
machine identity | service-token claim (common_name) |
Set only by the gateway after cryptographic validation; all inbound instances
stripped unconditionally first. Backends read only these — never Cf-Access-*
or provider token internals. This contract is what makes swapping
Cloudflare→Keycloak invisible to every backend.
7. Cloudflare topology: centralized by default
Tenant hostnames live in the platform's Cloudflare account / Zero Trust
org: one issuer, one JWKS, one wildcard tunnel (≥2 cloudflared replicas),
tenant teams get scoped dashboard access to their own Access policies.
Federated (tenant-owned Cloudflare account, own issuer, own tunnel) remains
supported — provider config is per-policy — but is the exception, not the
default. ScreamingFace is grandfathered federated (their zone, their org,
per the sent proposal); revisit only with their agreement.
Amendment 2026-07-23 — how a federated tenant rides the shared tier. A
federated tenant does NOT get its own Gateway (that would reintroduce the
per-tenant edge §2 rejected). Instead: a tenant-owned cloudflared
Deployment in the tenant's edge namespace, running the tenant's own tunnel
token (from Key Vault via ESO, break-glass write, never in git), forwards to
the shared tier's cf-protected-internal Service. A platform-authored
HTTPRoute for the tenant hostname attaches to the shared listener. A
platform-authored SecurityPolicy uses the tenant issuer, AUD, and JWKS.
Per-route JWT providers aggregate cleanly on the one listener, so each tenant
keeps an independent authentication boundary. Result: shared tier, tenant
edge. The tenant controls its hostname zone and tunnel inputs. The platform
owns the controller, listener, route, and enforcement policy, as ADR-0006
requires.
Observed at onboarding (2026-07-29): screamingface.ai's Access lives in the
OpenMined Zero Trust org (issuer openmined.cloudflareaccess.com), not a
separate tenant org — so the live shape is "own zone + own tunnel, same org."
Mechanically identical (per-app AUDs remain the tenant/service boundary; the
tunnel + zone are tenant-scoped); the fully-federated own-org variant remains
supported for a tenant that brings one.
8. Keycloak path (executes with roadmap 3c, designed now)
- K1 (low risk, per-tenant, canary by PR): Keycloak becomes a generic-OIDC IdP inside Cloudflare Access. Same Cloudflare JWT downstream; zero gateway changes. Tenant A moves to SSO while tenant B stays on OTP.
- K2 (optional): gateway validates Keycloak tokens directly (issuer/JWKS swap in the SecurityPolicy); Cloudflare drops to pure edge.
- Tenant mapping: client-per-tenant in one realm (company-wide SSO goal).
- Keycloak is tier-0 once login depends on it (HA per roadmap 3c).
9. Security invariants and standing tests
Non-negotiable invariants: header strip + re-inject (the single most important
one); no reachable origin for cf-protected; JWT validated at the gateway
(issuer + audience + JWKS), never in backends; network path lockdown (backends
← proxy only, proxy ← cloudflared only, tenant↔tenant denied); per-tenant
BackendTrafficPolicy for blast radius; JWKS fail-closed with a measured
cached-key survival window; tenant-prefixed rate-limit descriptors in the
shared Redis; per-tenant telemetry partitioning.
The T1–T8 standing test checklist from the alignment doc (policy independence, cross-tenant token replay → 401, forged headers in-cluster → 401, forged headers through a valid session → claim-derived values win, tenant-B meltdown drill, tier-label admission rejection, JWKS outage drill, cloudflared/proxy kill mid-traffic) becomes the acceptance suite for the gateway platform — repeatable checks, not one-off pokes.
10. Amendment 2026-08-13 — the preview exposure (PVW-D04)
Routed by the preview-environments program (PVW-D04, work package
PVW-M0-W04 —
https://github.com/OpenMined/infrastructure/blob/main/docs/programs/preview-environments/decisions.md).
The owner decided DECIDE PVW-D04: OPTION A on 2026-08-13.
Preview environments ride the existing cf-protected tier under a dedicated
preview subdomain. The design goal that shaped everything: no per-preview
Cloudflare write, ever — Cloudflare's tunnel and Access write permissions
exist only at account scope, so any automated per-preview minting would
require token custody over the tenant's whole account.
- Hostname shape:
<app>-pr-<N>.preview.dev.<tenant-domain>(first tenant:preview.dev.screamingface.ai). Every wildcard the design needs is a full leftmost label, which is the only form DNS records and tunnel rows can express. - Five-part contract for the preview exposure: exposure — tunnel-only through the existing shared Gateway; auth — Access JWT validated against one preview-tier AUD per tenant; traffic — preview-class, non-gating, budget-capped; availability — best-effort; admission — machine-generated preview routes may claim only hostnames matching the preview pattern.
- One AUD per tenant per tier. All previews of one tenant share one Access application and one AUD. §3's rule protects the boundary between tenants and tiers and is unchanged; this amendment fixes its reading inside one tenant's preview tier, where per-preview AUDs would cost account-token custody plus app churn. Per-preview Access applications are recorded as the hardening path if per-preview audiences are ever needed.
- The one-time tenant ask — four dashboard actions, no token, nothing recurring: one proxied wildcard DNS record for the preview subdomain pointing at the tunnel; one wildcard tunnel public-hostname row to the shared origin service; one wildcard Access application on the same pattern with the same policy shape as the dev apps; one advanced-certificate wildcard SAN for the new depth (Total TLS does not cover tunnel-fronted hostnames).
- Platform side: one preview-tier
SecurityPolicyvalidating the preview AUD; per-preview platform-stamped HTTPRoutes on the single shared listener (§4 unchanged);X-Tenantstamped per route; the identity strip inherited from the tier ClientTrafficPolicy. - Rejected: flat names at the service depth (DNS and tunnel wildcards cannot express partial labels, forcing either a zone-wide wildcard or per-preview API writes); per-preview Access applications (above).
- Implementation checks: inspect the live certificate covering the current service depth before freezing the ask list; live-test that a wildcard tunnel row routes; note that wildcard Access apps receive no preemptive cookies (the first request redirects to login — accepted).
11. Amendment 2026-08-19 — dev-tier single sign-on (one tenant AUD for user-facing services)
Decided by the platform owner on 2026-08-19. Narrows §3's per-service AUD reading inside one tenant's dev tier, and only for user-facing services. §3's between-tenant and between-tier boundary is unchanged.
Decision. In the dev tier, a tenant's user-facing services share
one Cloudflare Access application and one AUD — one app scoped to the
explicit user-facing dev hostnames (first tenant:
fusion.dev.screamingface.ai and leaderboard.dev.screamingface.ai). A user
authenticates once and the resulting token validates on every listed
user-facing dev subdomain of that tenant. The platform SecurityPolicy for
each such route validates that single shared AUD.
- User-facing today:
fusion.devandleaderboard.dev(its write path). Both move to the shared AUD. Leaderboard reads stay public (no policy). - Added 2026-08-31:
reports-internal.dev.screamingface.ai(report-intake's identified-report path; tenant added the hostname to the shared app the same day). Owner ruling: shared AUD — filing an identified bug report is an action any general dev user may perform. Recorded dissent (public-route assessment, verbatim position): the analysts split on whether these writes pass the exception test AT ALL — tracker writes arguably FAIL the "action a general dev user must not perform" test, arguing for an own AUD; the assessment held the question genuinely open, "must be decided, not defaulted". The owner decided it as above. Chosen reopening condition: revisit toward an own AUD if authenticated-flood abuse shows up in triage (the identity path carries no rate limit and skips Turnstile by design). - Amended 2026-10-30:
leaderboard.devleaves the shared application. Its write path (/v1/scores) moves to its own dev Access application with its own AUD, whose one policy is Allow, Include Everyone (§16(j), open submission, OME-1286). An Access policy belongs to the whole application, not to one destination, so the shared application could not open the submit path without also openingfusion.devandreports-internal.dev. The trade-off below ("the same trusted team") no longer holds for leaderboard writes on dev; the write route carries the §16(j) write budget. Dev users now log in once for the engine and once for the leaderboard, as on staging and production. - New user-facing dev subdomains are added to this app deliberately (default-closed): a subdomain gets the shared login only once listed, so a future privileged subdomain is never swept in by default.
- Rejected: a
*.devwildcard app. A wildcard is default-open — it would formally coveraigw-adminand every future subdomain, staying safe only by Cloudflare most-specific-app precedence plus the platform's per-route AUD split (defense in depth, but implicit). The explicit hostname list is default-closed and states the intent directly, which matches a design that has a privileged exception to protect. (The single-label wildcard also never reached previews —*.preview.devis a level deeper — so the choice does not affect them either way.)
Privileged-surface exception (holds the line). Services that gate
administrative or credential access keep their own scoped AUD and their
own smaller allow-list, and are never folded into the shared dev AUD.
The concrete case: aigw-admin administers every tenant's provider
credentials and admits only the aigateway operator allow-list; sharing its
AUD would grant credential administration to every dev user. It stays
separate. The test for the exception is "does this surface authorize an
action a general dev user must not perform?" — if yes, its own AUD.
Accepted trade-off. One shared AUD means any dev-authenticated user's token is accepted by every user-facing dev service of that tenant — for example, a fusion user can write dev leaderboard scores. In dev this is acceptable: the data is disposable and the user population is the same trusted team. It is not acceptable where the action has integrity or privilege weight, which the exception above carves out.
Scope guard. This amendment applies to the dev tier only. Staging and production keep §3's per-service AUD as the rule — the STP program decides their access model under its own gates. Do not copy this dev simplification upward.
Implementation. The shared Access app — scoped to the user-facing dev
hostnames — is a one-time tenant dashboard action (returns the AUD). The
platform side points the fusion-cf-access and leaderboard-cf-access
policies at the shared AUD; aigw-admin-cf-access is untouched. Until the
tenant returns the shared AUD, the value stays a fail-closed PENDING_*
sentinel (blocked by the k8s-validate.sh tripwire), so the change cannot
land half-configured.
12. Amendment 2026-08-24 — preview implementation reference
ADR-0013 suspended the PVW delivery workflow. It did not change the Preview exposure decision in §10. The suspended PVW records remain historical evidence. The Preview architecture guide now defines the current implementation.
13. Amendment 2026-08-31 — the public-intake route class (first fully-anonymous write surface)
Decided by the platform owner on 2026-08-31, with the tenant's agreement to
deploy the interim shape now and harden later. Grounded in the 2026-08-28
public-route assessment
(docs/consultants/2026-08-28-report-intake-public-route-assessment-for-codex.md),
whose findings were adversarially verified against the tenant app's source,
Envoy Gateway and Cloudflare documentation, and the live cluster.
What this admits. A hostname posture this ADR did not previously allow: a public-intake hostname — no Cloudflare Access at all, whole surface anonymous including writes. The tier had ONE prior anonymous exception, read-only: leaderboard reads (method-split route with no SecurityPolicy, §11). The separate admin-edge rail carries an analogous empty-SecurityPolicy precedent (the status page) — but that is a different gateway with a different auth stack (Entra OIDC, public LB, no tunnel): precedent for the mechanism, not a tier exception. This class is anonymous WRITE, which is why it gets its own contract instead of riding either precedent.
Reconciliation with §2. §2 defines cf-protected as "tunnel-only exposure,
Access JWT auth, no public IP exists". That auth line is henceforth the tier
default, not an absolute: the recorded departures are §11's
leaderboard-reads exception and this class. The gate below exists precisely
because an anonymous-write surface does not belong on this tier — the
resolution is the class moving OUT (to the public gateway), never §2 being
redefined down. First and only member: reports.dev.screamingface.ai
(report-intake, anonymous error reports into the tenant's private tracker).
Its identified companion reports-internal.dev.screamingface.ai is a normal
§11 shared-AUD route, not part of this class.
The §2 five-part contract for this class (dev, interim), plus one class-specific obligation:
- Exposure — tunnel-only, no public IP. Unchanged from §2.
- Auth — none at the edge. The only per-request gate is Cloudflare
Turnstile, verified in-app against a platform-delivered secret
(
sf-report-intake-dev-turnstile-secret). Identity stays gateway-owned on this route like every other: the tier-wideearlyRequestHeadersstrip is inherited automatically, the route stampsX-Tenantwithset, and the route additionally removesCF-Connecting-IPtoward the backend (the app's own contract). A forgedCf-Access-Jwt-Assertionis inert here — no SecurityPolicy consumes it and the app does not read it. - Traffic — INTERIM: no per-caller rate limit exists (accepted risks
below). The route path-matches
/v1/reportsonly — never/, so/healthzand/readyz(an unbudgeted DB query) stay unpublished. Explicit per-route timeouts. - Availability — best-effort, same as the tier: the class rides the shared fleet's tier-0 posture until the split below; no additional guarantee.
- Admission — default-closed. The member list of this class is this
section; adding a hostname means amending it, and edge objects for a
public hostname are platform-PR-only (the tenant fence already rejects
chart-rendered edge objects). Enforcement, not convention: a
k8s-validate.shtripwire (same pattern as the §11PENDING_*sentinel) asserts every cf-protected route carrying no route-targeted JWT SecurityPolicy appears in the checked-in anonymous-route allowlist atscripts/ci/anonymous-routes.allowlist— route-level, not hostname-level, so method-split hostnames (leaderboard: gated writes, anonymous reads) are covered, and the allowlist inventories ALL deliberately-anonymous cf-protected routes annotated by class (§11 read exception vs §13 public-intake). Precedence when the records diverge: the allowlist is the ENFORCED record, this section is the decision record, gateway.md's posture table is the human view. On this listener absence of policy fails open into public — and preview churn has already produced orphaned SecurityPolicies — so the fail-open must be CI-visible, not review-remembered. - Monitoring (class obligation, additional to the §2 contract parts) —
mandatory Gatus posture pair: the public hostname answers
with NO Access redirect (a 302 to
cloudflareaccess.comappearing means an Access app crept over the hostname — scope-widening detection in reverse), and the identity hostname still redirects unauthenticated. Plus an alert on the anonymous-POST 503 rate: a broken Turnstile secret answers every anonymous report with 503 while the pod reports healthy — no liveness, readiness, or naive 200-check can see that failure.
The edge slice (next PR) is mechanically specified by this section:
(1) the two HTTPRoutes and the shared-AUD SecurityPolicy; (2) the
scripts/ci/anonymous-routes.allowlist file plus its k8s-validate.sh
tripwire; (3) the Gatus posture pair and the anonymous-503-rate alert;
(4) the attestation table's move into docs/tenants/screamingface/. If that
PR is ever authored without this session's context, this list is the
contract — none of the four may be dropped silently.
Boundary artifact — attestations instead of an AUD. Every other hostname's
recorded boundary is its AUD, verifiable from git and proven by cross-AUD 403.
A public-intake hostname has no AUD, so its boundary record is a dated,
named attestation set for the facts that live in the tenant's Cloudflare
dashboard and cannot be verified from git: no Access application covers the
hostname (attested — "no app", never a Bypass policy, which is origin-identical
but unlogged); Turnstile widget with REAL keys (test keys always pass);
zone plan tier (Free — a host- or method-scoped edge rate rule is not
expressible); Bot Fight Mode posture (ON, see risk 3); no pre-existing
DNS record on either hostname. The Turnstile sitekey is public and recorded
deliberately — 0x4AAAAAAEjIHU_eaxfIZ-u6 — because Cloudflare's always-pass
test sitekeys are well-known constants, so the recorded value is what lets a
future reviewer verify "real keys" without dashboard access. Attested by the
owner 2026-08-31; the table moves into docs/tenants/screamingface/ with the
edge slice. Re-attestation triggers: plan change, tunnel change, any Access
app change, any abuse incident.
Accepted interim risks (owner, 2026-08-31):
- No per-caller rate limiting. The app's in-process limiter collapses to one shared bucket behind the tunnel (its TCP peer is the proxy), the per-IP BackendTrafficPolicy design remains parked, and the Free plan cannot express a scoped edge rule. Until the gate below is paid, the only per-request gate is Turnstile — a cost-raiser (solver farms sell tokens), not a limiter.
- Shared blast radius. Anonymous traffic rides the same cf-protected Envoy fleet and the same tenant sf-edge tunnel as all authenticated traffic. A flood degrades the tenant's own product first.
- Bot Fight Mode is ON, zone-wide, unscopable on Free. It challenges at the edge BEFORE Turnstile ever runs and may block legitimate SDK POSTs. (Existing tenant traffic has run with it on — an inference from live service, not an attested fact.) Decision point: the first SDK acceptance test against the live hostname. If reports are challenged, the response — switching BFM off zone-wide (a tenant action), a plan change, or the Worker fallback — is decided THEN; none is pre-committed here.
- Turnstile-for-headless-SDK is an open tenant question. Turnstile is a browser widget; the hostname's primary caller is a Python SDK. Until the tenant answers how the SDK obtains a token, the anonymous path may 403 legitimate SDK reports. Accepted for dev; it blocks nothing platform-side.
THE GATE — this interim shape is dev-only and it expires (BLOCKING).
Before any of the following, the public-intake class MUST move off the
shared listener onto a dedicated public gateway (working name cf-public:
own EnvoyProxy fleet on the same controller — the admin-edge
gateway-per-trust-model precedent, ~2 pods — with label-scoped
allowedRoutes, clientIPDetection (customHeader: CF-Connecting-IP,
fail-closed) scoped to that tier only, a per-IP Global BackendTrafficPolicy on
the already-live rate-limit service, default-deny NetworkPolicy around the
fleet, and its own tunnel decision):
- (a) any staging or production exposure of any public-intake hostname. The staging-production program MUST treat "public-intake still on cf-protected" as a failed readiness gate of the same class as W24/W25 — a blocker, not a finding. Do not scope, waive, or defer it in an activation review.
- (b) admission of a second public-intake hostname, any tenant. The member list above may never grow while the class sits on cf-protected.
- (c) any tier-wide
clientIPDetectionon cf-protected — forbidden outright. That trust change alters client-IP semantics for every tenant route on the single shared listener and belongs only on the isolated public tier. If per-caller limiting is wanted, that IS the split trigger.
Trigger (a) is the owner's stated condition for accepting the interim shape ("must split before staging and prod prep", 2026-08-31); (b) and (c) are platform-derived conditions of the same gate, from the adversarially verified assessment. Removing or weakening any of the three is a new ADR-level decision, not an edit.
Where this gate is enforced. The delivery-program workflow is suspended (ADR-0013), so no STP document gates anything today — W24/W25 block production because CLAUDE.md's operating rules say so. This gate therefore lives in the same place: CLAUDE.md carries it (edited in this PR, adjacent to the W24/W25 block), and any future staging scoping MUST import it as a day-one blocker rather than rediscovering it here.
Naming note. docs/platform-delivery-plan.md §1.3 sketches a "cf-public"
tier for anonymous READS (scoreboard's case). The split gateway this gate
requires subsumes that sketch: one public gateway hosts both anonymous-read
and public-intake classes, with per-class routes and budgets. Where the §1.3
sketch and this section disagree, this section supersedes.
Rejected alternatives (assessed and adversarially verified; details in the assessment): tunnel-direct-to-app (no gateway in the path — deletes the identity strip, X-Tenant, all edge forensics, and conflates IP trust with identity trust in the app's own knobs; rejected with no adequate mitigation); Access Bypass or embedded service tokens (Bypass is origin-identical to no app minus the audit trail; a token in a public SDK is a public credential); Cloudflare Worker intake on the tenant zone (best invariant preservation, heaviest tenant coupling — held as the fallback if this class is ever declined or retired).
Corrective scope. docs/security/gateway.md and
docs/products/gateway-identity-flow.md stated the Access invariant
absolutely ("Cloudflare Access authenticates every request", "you cannot
smuggle the header past the edge") — already false for leaderboard reads and
the status page before this amendment. Both are edited in this PR to
tier-scoped statements, and gateway.md gains a per-hostname posture table so
exposure is reviewable from git even where it is not enforceable from git.
14. Amendment 2026-09-08 — self-service route declarations and hostname admission
Status: proposed. Decider: platform owner. Drafted with the Phase 1 pilot
plan (A3.5: "Envoy Gateway stays. Routes become self-service."; items P1-42,
P1-43, P1-44;
https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/phase-1-pilot-plan.md).
The object names below are the P1-42 design as specified on 2026-09-04. P1-42
has not merged as of 2026-09-08: there is no kubernetes/apps/tenant-edge, no
kubernetes/appsets/tenant-edge.yaml, and no hostname policy under
kubernetes/infrastructure/tenant-guardrails/. Acceptance conditions: P1-42
has merged, every name here has been re-read from origin/main and
corrected, and the owner has answered the pending decisions below. A position
the owner has not yet taken is marked Decision pending with its options
and its default.
What changes. Tenant edge objects are no longer hand-written by the
platform per hostname. A tenant declares one route in one file,
kubernetes/tenants/<tenant>/routes/<env>/<route>.yaml (the route
declaration), with these fields: tenant, route, env, app,
hostname, tier, namespace, service, port, identity, audience,
methods, rules[] (path type, path value, request timeout),
stripRequestHeaders, and a recorded-not-rendered rateBudget. The file is
the values file of the platform chart kubernetes/apps/tenant-edge, which
renders the HTTPRoute and, for identity: access-jwt, the tier's
SecurityPolicy. The ApplicationSet kubernetes/appsets/tenant-edge.yaml
(a git files generator over the declaration path;
preserveResourcesOnDeletion: true) delivers one Application per declaration
under the AppProject <tenant>-edge (first: screamingface-edge), whose
whitelist holds only the edge kinds (HTTPRoute, SecurityPolicy,
BackendTrafficPolicy) and whose destinations are the tenant's workload
namespaces, never its edge or delivery namespaces. §5 is unchanged: the policy is platform-owned and the declaration
supplies values only; the issuer, the JWKS URI, the header extraction, and
the claim-to-header map are template invariants. §6 is unchanged: the
tier-wide identity strip is inherited, and the route stamps X-Tenant with
set. One hostname may carry two declarations only when their methods are
disjoint and non-empty (the leaderboard read/write split): a SecurityPolicy
targets a whole route, so a method split is two routes.
Tiers by name. Every declaration names one tier. The chart's fail
guards and the CI validator enforce the same table.
| Tier | Identity | Audience | Methods | Environments |
|---|---|---|---|---|
user |
Access JWT | on dev may equal the shared dev AUD (§11); on staging and prod unique per hostname (§11 scope guard; pending decision below) |
any | any |
privileged |
Access JWT | own AUD, never shared (the §11 exception) | any | any |
anonymous-read |
none | absent | non-empty subset of GET HEAD OPTIONS |
Amended by §16: dev, staging, prod, for a hostname on the §16 list only; gateway cf-public; budget key source-ip with requestsPerMinute and routePerMinute required once single-homed. (Before §16: dev only; the §11 read exception.) |
public-intake |
none | absent | any; never path / |
dev only, §13 member list only, at most one declaration (§13 gate, not a decision) |
An identity: none declaration still needs its line in
scripts/ci/anonymous-routes.allowlist (<tenant>/<env>/<route> <class>);
the validator checks every anonymous declaration against that file and flags
stale lines. The allowlist stays the enforced record of §13; this section
stays the decision record.
The admission control §4 deferred, now specified. §4 (amended
2026-07-22) moved the tier to one shared listener and deferred structural
hostname-squatting prevention to "an admission policy (Kyverno/OPA)". It is a
native ValidatingAdmissionPolicy, tenant-httproute-hostnames, bound
to every namespace that carries the openmined.org/tenant label, with
failurePolicy: Fail and a per-cluster param ConfigMap
tenant-hostname-suffixes (namespace kube-system,
parameterNotFoundAction: Deny) that registers the suffixes a tenant may
claim on that cluster (dev: dev.screamingface.ai; staging and production
register their own labels, so a dev hostname cannot be claimed on staging).
It denies an HTTPRoute with no hostname, a wildcard hostname, or a hostname
outside the tenant's registered suffixes. No Kyverno, no OPA: the platform's
tenant guardrails are native admission policies already
(kubernetes/infrastructure/tenant-guardrails/), and one rule does not
justify a second admission engine. Preview namespaces carry the tenant label
and comply. The Consequences bullet "before GA / second tenant — the hostname
admission policy" is satisfied by this section once it is accepted and the
policy is live.
Enforcement, in order. (1) The CI validator
scripts/ci/tenant-edge-validate.sh, run by scripts/ci/k8s-validate.sh
inside the required validate check, with named checks: check_schema;
check_path_agrees (the path segments equal tenant, env, route);
check_hostname_suffix (against the same ConfigMap the policy reads);
check_namespace_owned (a tenant-labelled Namespace in the registry and a
destination of the edge AppProject); check_service_exists (the Service and
port exist in the tenant chart at the pinned chart ref, rendered with our
values); check_tier_rules; check_render (one HTTPRoute; one
SecurityPolicy if and only if access-jwt); check_hostname_unique;
check_anonymous_allowlist. (2) The chart's fail guards, which stop Argo
from rendering a declaration that bypassed CI. (3) The AppProject fence,
which can deliver no other kind. (4) The admission policy, which denies a
hostname the tenant does not own regardless of how the object arrived. Every
tenant-side change lands as a pull request in this repository; Argo delivers
from main only.
Reconciliation with §4, §11, §13. §4's single listener stands;
per-tenant listeners remain rejected while the Envoy Gateway merge behaviour
is unchanged. §11's shared dev AUD is expressed as a user declaration whose
audience equals the shared AUD; the validator rejects that value on
privileged. §13's gate is untouched: a public-intake declaration is valid
on dev only, for the §13 member list only, never with path /, and at most
one may exist while the class sits on cf-protected; the split to cf-public
remains the blocking condition for any staging or production exposure.
rateBudget is recorded, not rendered: rate limiting is parked, per-path
budgets need separate routes, and Envoy Gateway 1.5 rejects sectionName on
a BackendTrafficPolicy at admission while the CRD schema accepts it.
Nothing here un-parks it.
Delivery of tenant changes (P1-44). A tenant may keep its declarations in
its own repository under platform/routes/; a Kargo Warehouse watches that
folder and a PromotionTask copies the files into this repository as a pull
request that must pass the validator. Production declarations
(routes/prod/) are never auto-merged.
- Decision pending: merge policy for mirror pull requests on dev and staging. Options: (A) automatic when CI is green, the plan's proposal (P1-44); the validator, the AppProject fence, and the admission policy are the controls. (B) Human merge until the validator has run on the six migrated dev routes and one staging change, then (A) for dev and staging only, the WS4 specification's proposal. Default: (A), the plan's. The two records disagree; the owner settles it.
- Decision pending: Access application shape on staging (P1-41). Options: (A)
one application and one AUD per hostname, §3 and the §11 scope guard as
written; the validator enforces it on
stagingandprod. (B) One application per tier (one user-facing AUD forfusion,leaderboard,reports-internal; one privileged AUD foraigw-admin), which needs a §11 scope-guard reconciliation in this section before any staging declaration may share an AUD. Default: (A). - Decision pending:
anonymous-readon staging (P1-41, P1-42). Options: (A) dev only; staging sits behind the tester allow list (plan A5), so the staging leaderboard is oneuserroute with all methods. (B) Allowenv: stagingfor reads as on dev. Default: (A). Resolved 2026-10-05 by §16: (B), extended toprod, for a hostname on the §16 list only, oncf-public, never oncf-protectedonce single-homed. - Decision pending: the
X-Tenantvalue the chart stamps. Options: (A) the declaration'snamespace; five of the six live routes do this, migrating Applied 2026-09-08 (P1-43, #886): the chart stamps the route's namespace; the tenant's code ignores the header (their tests assert identity derived from it is dropped), so it stays edge forensics. The owner can still change it here.aigw-adminchanges its value fromscreamingfacetosf-aigw, and no tenant code reads the header (a search of the tenant repository returned nothing on 2026-09-04). (B) A per-declaration override field. Default: (A).
Rejected alternatives. Tenant-authored HTTPRoute objects in tenant
namespaces: the AppProject fence and ADR-0006 forbid tenant edge objects, and
a route without its tier policy fails open into public on this listener. A
Kyverno or OPA engine for one rule. Per-tenant listeners: §4's Envoy Gateway
merge bug is unchanged. A per-hostname Access application minted by
automation: Cloudflare write permissions exist only at account scope (§10).
Corrective scope on acceptance. docs/security/gateway.md marks hostname
squatting as a known gap and derives its posture table by hand; on acceptance
the gap line closes to "ADR-0005 §14 (P1-42)" and the table is derived from
the declarations after P1-43. docs/conventions.md still names "the Kyverno
admission policy" as the tenancy backstop; it changes to the native policy.
Both edits land in the acceptance PR, not in this draft.
15. Amendment 2026-09-22 — the cf-public gateway: the public-intake class leaves the shared listener, on its own tunnel
Status: proposed. Decider: platform owner; the decisions recorded below were
taken by the owner on 2026-09-22. Trigger: OME-1175 ("allow analytics
ingestion without Cloudflare Access login"), which asks for a second
anonymous-write hostname — the §13 gate (b) condition, word for word.
analytics.dev.screamingface.ai was placed on the user tier (#958) because
that gate forbade the alternative; the tenant's own service contract
(ScreamingFace docs/spec/2026-09-09-OME-1152-analytics-service-spec.md)
adds "no embedded shared SDK secret, CAPTCHA or login is added to this
anonymous endpoint" and "ingress also needs deployment-owned global abuse
protection before public launch" — so the §13 rejected alternatives stay
rejected and the split is the only answer. Grounded in §13's gate text, the
2026-08-28 assessment (Option B; Phase 1 items 7–9), the
docs/platform-delivery-plan.md §2 runbook stub (superseded here where they
differ), and a read-only review of the draft against both clusters and the
Cloudflare and Envoy documentation on 2026-09-22. Every object shape below
passed kubectl apply --dry-run=server against aks-dev (Envoy Gateway
v1.5.0) the same day. Acceptance conditions: the five pull requests of the
split have merged on dev, the two dev hostnames answer through the new tunnel,
the proofs in the working plan pass live, the corrective scope has landed, and
docs/security/gateway.md carries the re-attested rows.
What changes. A second platform-owned Gateway, cf-public in
pulse-edge, with its own EnvoyProxy fleet cf-public-proxy, serves every
public-intake declaration, and a second, tenant-owned Cloudflare Tunnel per
environment carries the public hostnames to it. The route chart maps the tier
to the gateway (public-intake → cf-public; every other tier →
cf-protected); a declaration still names no gateway. cf-protected keeps
§2's contract as written, with the §11 leaderboard-reads exception as its
only anonymous route. (Amended by §16: a hostname on the §16 list also maps
to cf-public, and the leaderboard read route leaves cf-protected.) The §13 member list grows to
reports.dev.screamingface.ai, analytics.dev.screamingface.ai, and — once
the staging slice lands — reports.staging.screamingface.ai and
analytics.staging.screamingface.ai.
The five-part contract for cf-public (a tier, not an interim):
- Exposure — tunnel-only,
ClusterIP, no public IP. Each environment's public hostnames ride their own tenant-owned tunnel (sf-public-<env>, created in ScreamingFace's Cloudflare account: acfargotunnel.comCNAME proxies only same-account records), whose connectors run as Deploymentcloudflared-publicinsf-edgebeside the existingcloudflared. Every public tunnel route names the stable Servicecf-public-internal.envoy-gateway-system.svc.cluster.local:80(80 → 10080, thecf-protected-internalalias pattern). The public connector's NetworkPolicy reaches only thecf-publicfleet; the existing connector's reaches onlycf-protected. §7's observed shape becomes "two tenant-owned tunnels into two platform tiers". The token record issf-public-tunnel-token-<env>in that environment's vault, written by the owner's break-glass window before the connector merges, and listed indocs/runbooks/cloudflare-tunnel-token-rotation.md. - Auth — none at the edge, by definition. No Access application covers a
cf-publichostname (attested per hostname, never a Bypass policy); noSecurityPolicyrenders for the tier. (Amended by §16(c): on a §16 hostname auser-tier JWT SecurityPolicy renders oncf-public, and a path-scoped Access application covers/v1/scoresonly: the write path and its score-detail reads.) The application owns any per-request gate (Turnstile for report-intake; none for analytics, by contract). Identity stays gateway-owned: aClientTrafficPolicyon thecf-publicGateway carries the sameearlyRequestHeadersstrip ascf-protected-identity-strip, the route stampsX-Tenantwithset, and a public-intake declaration stripsCF-Connecting-IPandX-User-Emailtoward the backend. - Traffic — the tier can limit per caller, and the owner decides when it
does. The same
ClientTrafficPolicysetsclientIPDetection.customHeader{name: CF-Connecting-IP, failClosed: true}: Envoy's trusted client address becomes Cloudflare's, and a request that carries no such header is refused with 403. §13 gate (c) stands:cf-protectednever gets this.rateBudgetis rendered for this tier only —{requestsPerMinute: N, key: source-ip, routePerMinute: M}becomes one GlobalBackendTrafficPolicyper route on the live Redis-backed limiter: twosourceCIDR {type: Distinct}rules (0.0.0.0/0,::/0) at N per minute and one unselected rule at M per minute, the route ceiling; whole-route policies only (Envoy Gateway 1.5 rejectssectionNameat admission), and never while a route is attached to both gateways (oncf-protectedthe trusted address is the connector pod, so a per-IP rule there would key every caller onto two buckets). Owner decision 2026-09-22: no numbers on dev or staging — the users are known — so no policy renders today; the numbers are a production-ladder item. The limiter runs fail-open with one replica today (norateLimit.failClosed); the ladder closes that. IPv6 keys are per address (/128), so the route ceiling is the IPv6 flood control; Cloudflare's Pseudo IPv4 is evaluated with the Bot Fight Mode decision. Every other tier keepsrateBudgetrecorded, not rendered. (Amended by §16(d):rateBudgetrenders on any route that resolves tocf-public; a single-homedanonymous-readroute requiresrequestsPerMinuteandroutePerMinuteon every environment, which supersedes "no numbers on dev or staging" for that class only.) - Availability — its own tier-0 fleet: 2 replicas,
envoyPDB minAvailable: 1, container requests and a memory limit (no CPU limit — CFS throttling under a flood), hostname topology spread withwhenUnsatisfiable: ScheduleAnyway(each cluster has one untainted user node today, so the spread takes effect when a second exists). No tracing stanza: the staging overlay removes tracing by thecf-protected-proxyname only, and anonymous traffic is not traced by default. The fleet lives inpulse-edge/base, so it renders wherever the tier is enabled: aks-dev and aks-staging today; aks-prod does not enablepulse-edge, and the pull request that createsoverlays/prod(STP-P1-45) is a production change under the ask-first rule and renders this fleet with it. A flood oncf-publicdegradescf-publicand its own tunnel; the tenant's authenticated product keeps its tunnel and its fleet. - Admission — default-closed at three layers. (1) The Gateway's
allowedRoutes.namespaces.from: Selectoradmits only namespaces labelledopenmined.org/public-routes: "true", set only in the platform-owned namespace registry (kubernetes/tenants/<tenant>/namespaces.yaml, todaysf-report-intakeandsf-analytics); both tenant fences blacklist cluster-scoped kinds, so a tenant cannot self-label, and a forged route in any other namespace cannot attach. (2) The validator'scheck_tier_rules: a public-intake hostname is in the member list above, its namespace carries the label, its environment isdevor (after the staging slice)staging—prodis denied with a message that names the production ladder below — no rule publishes/, and the allowlist line carries classpublic-intake-ADR-0005-s13; the "at most one declaration" rule is removed, because gate (b) is satisfied by the split.check_renderasserts the parentRefs the tier implies and aBackendTrafficPolicyif and only if a budget is set and the route is single-homed. (3) Thek8s-validate.shanonymous-route tripwire treats acf-publicparent exactly like a policy-lesscf-protectedone. Thetenant-httproute-hostnamesadmission policy applies unchanged. The fleet's own default-deny NetworkPolicy (cf-public-fleetinenvoy-gateway-system: ingress 10080 from the public connector only; egress DNS, the controller's xDS ports, the limiter, and the two backend ports) is the only guard against an in-cluster caller forgingCF-Connecting-IPat the stable Service — the custom-header extension trusts its upstream by design. (Amended by §16(e): egress addssf-scoreboard9106 and TCP 443 to Cloudflare's published IPv4 ranges for the Access JWKS;check_tier_rulesadmitsanonymous-readanduseron a §16 hostname.) - Monitoring (class obligation) — a Gatus posture check per public
hostname with
ignore-redirect(a 302 tocloudflareaccess.commeans an Access application crept back; the expected answer is the application's own refusal code); the identity companions unchanged; the report-intake 503-rate alert unchanged; analytics gains its delivery-503 alert and a 429-rate signal. Recorded limit: Gatus cannot tell Envoy's fail-closed 403 (the header missing — the zone's "Remove visitor IP headers" transform switched on) from the application's 403 on reports; the transform is therefore an attested fact per hostname, re-attested with every tunnel change. Staging rows are board-only; paging is a ladder item.
Access log (owner decision 2026-09-22). The cf-public fleet logs the
cf-protected field set minus x-forwarded-for and
downstream_remote_address: the callers are anonymous reporters and SDK
users, the analytics contract promises no IP reaches analytics data, and
cf-ray keeps every request attributable in the tenant's Cloudflare
dashboard. Exception: downstream_remote_address stays on from the fleet's
first merge until the detection and spoof proofs have run (it is the only
observable for "Envoy saw the real address, not the client-supplied
header"), then leaves in the analytics change. Incident mode is a one-line
change re-adding the field (Envoy Gateway pushes access-log configuration
over xDS without a restart) with a stated removal condition.
Transition — no dark window for reports.dev. A hostname's DNS record
names one tunnel, so the move is a switch, not a fade; the chart's platform
constant publicIntakeDualHome makes both paths valid while the switch
happens. Order: (1) the fleet lands inert in pulse-edge/base (nothing
carries the label); (2) the dev public tunnel exists and its token is in the
dev vault; the public connector, the quota fix (explicit CPU limits on both
connectors — the namespace ceiling is fully consumed today), the chart
mapping with dual-home ON, the labels, the validator and tripwire changes and
the report-intake ingress rule land; reports-public is attached to both
gateways; (3) the hostname moves: a throwaway canary route on the new tunnel
proves the tunnel-to-fleet path (Envoy 404), the reports.dev ingress rule
is added to the new tunnel's configuration through the API (the dashboard
refuses a hostname whose record exists), the existing proxied CNAME's target
is edited from the old tunnel's cfargotunnel.com name to the new one
(atomic at the edge), the rule is removed from the old tunnel's configuration
through the API — never a dashboard route delete, whose effect on the DNS
record is undocumented — and the hostname is re-attested; (4) dual-home goes
OFF and analytics flips to public-intake in the same change (identity
none, methods: [POST], the strips, the allowlist line); its hostname
moves the same way, and the owner removes it from the shared dev Access
application last; (5) records; (6) the staging slice. Analytics is not
dual-homed: no SDK caller exists yet on the tenant's main.
Staging (owner decision 2026-09-22: admitted). A public-intake
declaration with env: staging is valid once, on aks-staging: the fleet
renders from base (it does on merge), the staging public tunnel exists with
its token in the staging vault and its connector in sf-edge, the edge
AppProject lists the two namespaces on aks-staging, the report-intake staging
workload ticket exists (the workload before the route), the two hostnames are
in the member list, and each carries its attestation set (no Access
application, no pre-existing DNS record, Turnstile staging widget with real
keys for reports, Bot Fight Mode status, plan, transform OFF). The deny
fixture public-intake-staging becomes public-intake-prod. Taken
2026-10-23 for analytics.staging.screamingface.ai only (owner go): the
member list gains that one hostname, the chart guard and the validator admit
env: staging and deny env: prod, and the edge AppProject lists
sf-analytics on aks-staging. The staging route carries POST /v1/events,
with no budget; the Colab consent bridge (GET and POST under /bridge)
joins it the same day by owner decision, before its browser acceptance.
reports.staging.screamingface.ai stays out of the member list until its
own change. The owner's attestations are recorded in
docs/tenants/screamingface/apps/analytics.md (staging table).
Production taken 2026-10-23 for analytics.screamingface.ai only, by
owner decision, before the rest of the ladder: the member list becomes per
environment, the validator and the chart guard admit that one hostname on
env: prod, and the route carries the staging shape and limits (120 a minute
per address, 1200 for the route, the Colab bridge on). Ladder items still
open, to follow after go-live: (7) paging and (10) production log export
(the staging log alerts exist; production follows), (8) the staging shut-off
drill, (9) the strip and spoof proofs on the production proxy, and (11) the
tenant's SDK back-off check. Items (3) and (4) are the owner's baseline and
acceptance of the same day.
Production ladder (strict upwards). env: prod stays denied by the chart
guard and the validator until a later amendment records evidence for each
item (amended by §16(g): for the anonymous-read class, §16 is that
amendment; public-intake on prod stays denied): (1) aks-prod registered and pulse-edge/overlays/prod rendering this
base; the production vault admits sf-edge. (2) A separate production public
tunnel in the tenant's account, its token in the production vault, at least
two connectors on at least two nodes with a PDB, the sf-edge quota sized
for four pods. (3) Budgets set from at least two weeks of staging 2xx/429
data, with the IPv6 posture recorded. (4) rateLimit.failClosed: true
(controller-wide; it also governs preview policies) and the limiter and its
Redis at two replicas with PDBs, or an explicit acceptance. (5) The
production zone's plan, WAF and Bot Fight Mode posture decided; a Turnstile
production widget with real keys. (6) The full attestation set per production
hostname, re-attested on every tunnel change. (7) Paging: a Gatus alerting
provider with alerts on the production rows, the 503-rate and 429-rate alerts
wired to a pager. (8) A kill-switch drill on staging with measured
time-to-dark for the DNS switch, the tunnel-rule removal and the connector
scaled to zero through git (kubectl scale is reverted by self-heal), and a
runbook. (9) A cf-public identity-strip sentinel and the recorded
CF-Connecting-IP spoof proof. (10) A retention bound for the access log
and the incident-mode toggle documented; production log export verified.
(11) Production declarations never auto-merge (§14), and the SDK's 429
back-off verified by the tenant.
What this satisfies, and what it does not. §13 gate (b): satisfied — the
class is off the shared listener, so a second member is admitted. Gate (c):
unchanged and still forbidden on cf-protected. Gate (a): the "public-intake
still on cf-protected" blocker is replaced, not removed — staging exposure
requires the staging fleet, tunnel and attestations above; production
exposure requires the ladder — and CLAUDE.md's gate paragraph is rewritten on
acceptance to say exactly that, in the W24/W25 blocker class. §11 is
untouched. §14's tier table changes in the public-intake row only:
environments dev, staging (§15); prod denied by validator, gateway
cf-public, budget optional until the ladder, count limit gone.
Rejected alternatives, for OME-1175 specifically. A service token shipped
in the SDK (a public credential; §13; forbidden by the tenant contract). An
Access Bypass (§13). Amending gate (b) to admit two hostnames on the shared
listener (keeps the shared blast radius and no per-caller control — the
pressure the gate exists to refuse). One public tunnel shared by dev and
staging (Cloudflare balances a tunnel across all its connectors, so traffic
would reach a cluster without the route). Moving the leaderboard hostname now
(D4: its reads are method-split from Access-gated writes and one hostname has
one tunnel; the whole-hostname move is the named follow-up, after which
cf-protected carries no anonymous route). (Amended by §16: the follow-up is
taken 2026-10-05.) A Cloudflare Worker intake (held
as the fallback, unchanged). Tunnel direct to the app (§13).
- Bot Fight Mode and Pseudo IPv4: decided at the first SDK acceptance test, never mid-incident (§13 risk 3, unchanged).
Corrective scope on acceptance. CLAUDE.md's gate paragraph (as above);
docs/security/gateway.md (the analytics rows; the reports.staging row
rewritten from "must not exist"; the known-gaps rate-limit line says
"rendered on cf-public only when a budget is set, parked elsewhere");
docs/runbooks/declare-a-route.md (tier table; "what a declaration does not
change" gains the per-tier tunnel and origin);
docs/runbooks/cloudflare-tunnel-token-rotation.md (two record rows) and
docs/runbooks/secret-rotation.md (the public tunnel token);
kubernetes/apps/tenant-edge/README.md; the report-intake app doc's open
item; a new docs/tenants/screamingface/apps/analytics.md with its
attestation set; docs/architecture/platform.md (two tier gateways, two
tenant tunnels). Known proof gaps, recorded: the identity-strip sentinel
(pt-echo) covers cf-protected only; Gatus cannot distinguish the two 403s
on reports.
16. Amendment 2026-10-05 — whole-public hostnames: anonymous reads on cf-public, Access-JWT writes
Status: accepted for implementation 2026-10-05 by owner decision (option 3,
all three environments). Decider: platform owner. The production conditions
in (g) are signed by the owner before the production Access edit. §14 and
§15 keep their own status lines; the statements this section changes carry
an "amended by §16" note where they stand. Trigger: the owner's goal of an
anonymous ScreamingFace leaderboard on dev, staging and production, with
writes still behind Cloudflare Access. Grounded in a read-only review of
origin/main at 92679702, the three clusters, the tenant's scoreboard
source at the staging and production pin 68cad9d9 and the dev pin
8c4db51e, and the tenant's SDK at e9e850e8 on 2026-10-05, with a
red-team pass over the plan the same day.
Where it stands today. On dev the leaderboard reads are anonymous, but on
cf-protected: that is the interim shape §13 warns about, and the §11 read
exception is the only anonymous route on that tier. On staging and production
one user declaration per hostname carries every method, and Cloudflare
Access covers the whole hostname. The cf-public fleet renders on all three
clusters but carries traffic on dev only.
(a) The anonymous-read class leaves cf-protected. A leaderboard hostname
moves WHOLE to cf-public, reads and writes together, because one DNS record
names one tunnel. An anonymous route renders on cf-protected ONLY as the
second parent of a hostname on the dual-home list in (b), while its tunnel
move is in progress — never once the hostname is single-homed. After each
move, cf-protected carries no anonymous route for that hostname, and after
the last move it carries none at all; §2's "Access JWT auth" line is again
the whole of that tier's auth contract. The chart and the validator refuse
anonymous-read on an unlisted hostname, so an anonymous route on
cf-protected (outside the dual-home window) cannot render. Accepted for the
dual-home window only: the cf-protected fleet has no NetworkPolicy, so an
in-cluster pod can read the staging or production leaderboard without Access
and without a budget while the hostname is dual-homed. The identity strip
still applies, so nothing private leaks; the window runs from the split merge
to the single-home merge and is kept as short as the owner's switch steps
allow.
(b) The member list is two platform-owned chart constants. No new tier,
CRD, controller or Cloudflare product. Two template defines in
kubernetes/apps/tenant-edge/templates/_helpers.tpl, each on one line so the
validator reads them by regex (the publicIntakeDualHome pattern), each a
comma list that no declaration can set:
tenant-edge.publicHostnames— the whole-public hostnames.tenant-edge.publicHostnamesDualHome— the listed hostnames whose tunnel move is in progress. Every entry is also inpublicHostnames; the chart and the validator fail a dual-home entry that is not.
The members are leaderboard.dev.screamingface.ai,
leaderboard.staging.screamingface.ai and leaderboard.screamingface.ai.
Each joins the list in its own environment's change, after that
environment's prerequisites in (e) and (i) are live. On a listed hostname
the tiers map as follows: anonymous-read and user render on cf-public;
privileged never renders (the chart and the validator fail it). Every
declaration on one hostname resolves to the same parentRefs: one hostname,
one tunnel. The mechanism change extends the validator rule
check_hostname_unique to assert this; today that rule checks only one
tenant, environment and namespace per hostname, disjoint methods and the
hand-written allowlist. A route's namespace must
carry openmined.org/public-routes: "true", set in the platform-owned
namespace registry (§15 Admission), so sf-scoreboard joins sf-report-intake
and sf-analytics. The allowlist class for the read route is renamed from
anonymous-read-exception-ADR-0005-s11 to anonymous-read-ADR-0005-s16.
Each listed hostname keeps the dev method split as two declarations:
leaderboard— tieranonymous-read, identitynone, methodsGET HEAD OPTIONS, no SecurityPolicy, the budget in (d).leaderboard-writes— tieruser, identityaccess-jwt, methodsPOST PUT PATCH DELETE CONNECT TRACE, the chart's unchanged JWT SecurityPolicy with the hostname's existing AUD (dev: the shared §11 AUD; staging and production: their own per-hostname AUD, §11 scope guard).
Together the two cover all nine Gateway API methods. Read scope stays
PathPrefix / (dev parity): /healthz, /docs, /redoc, /openapi.json
and, where the pin serves it, /readyz (a database query) become public
under the budget. The tenant confirms that data surface before any staging or
production Access edit ((g)).
(c) Auth — amends §15 Auth. On a listed hostname, a user-tier JWT
SecurityPolicy may render on cf-public. The policy targets the route by
name, so it applies on whichever gateway the route attaches to; issuer, JWKS
URI, header extraction (Cf-Access-Jwt-Assertion) and the claim-to-header
map stay template invariants (§5, §14). Cloudflare Access covers only the
write path /v1/scores — the only write endpoint at both tenant pins:
- The Access application is path-scoped: destination
<hostname>/v1/scores. On staging and production the owner edits the existing application IN PLACE (destination<hostname>becomes<hostname>/v1/scores), so the AUD is unchanged. Never delete and recreate it: a new application issues a new AUD, and then every write answers 401. Never add a Bypass policy (§13). The policy and its Include list stay as they are (superseded 2026-10-22 by (j): Include Everyone). Attested per hostname, re-attested on every tunnel or Access change. Dev already has this shape on the shared dev application. - Access matches host and path, not method. So
GET /v1/scores/{id}(the portal's score detail) stays login-gated, as on dev. - The identity boundary is the edge, not the app. The scoreboard app trusts
X-User-Emailfrom any peer in the pod CIDR and verifies no token itself (tenantapps/scoreboard/src/scoreboard/core/auth/cloudflare_identity.py, module docstring: "This service therefore does no token work of its own"). The comment intemplates/securitypolicy.yamlthat says the application verifies the Access JWT itself is wrong for this app and is corrected with the mechanism. The boundary is therefore two things: thecf-publicidentity strip (cf-public-client-ip,earlyRequestHeaders.removeofX-User-Email,X-User-Id,X-Service-Id), and thesf-scoreboardingress selector, whosecf-publicandcf-protectedpeers each name bothgateway.envoyproxy.io/owning-gateway-nameandgateway.envoyproxy.io/owning-gateway-namespace: pulse-edge. No static check of the strip exists today (k8s-validate.shchecks only that thept-echosentinel has noclaimToHeaders). The mechanism change adds one tok8s-validate.sh:cf-protected-identity-stripandcf-public-client-ipeach target their own Gateway and removeX-User-Email,X-User-IdandX-Service-Id; no overlay may patch either. - Any future tenant write endpoint must sit under an Access-covered path
before it ships. Reason: Cloudflare turns the SDK's
Cf-Access-TokenintoCf-Access-Jwt-Assertiononly on a path an Access application covers; on any other path a write meets the Envoy JWT check with no token and answers 401. - The SDK learns Access per request, so a path-scoped application still works
for SDK submissions.
sync_auth_flowreads the Access challenge from the response, logs in, and re-sends withCf-Access-Token(packages/screamingface/src/screamingface/_access/auth.py:289-342);client.login()logs in to the engine origin only (client.py:141-146); after a login challenge the first request raisesaccess_reauthenticatedunless the caller marked it replay-safe (_require_replay_safe,auth.py:566-585). The tenant's real SDK submission after each Access edit is the end-to-end proof. Corrected 2026-10-22 ((j)): this bullet was wrong for ScreamingFace clients before 0.1.6, and the proof it names was not run. - Logged-in readers lose their private-board rows on the read path on
staging and production (dev parity): the read route has no SecurityPolicy,
so no verified identity reaches the app on a read. The owner tells the
tenant. Optional identity on the read route (
jwt.optional) is a later, separate ADR item.
(d) Traffic — amends §15 Traffic. rateBudget may render on any route
that resolves to cf-public, not only public-intake. Key source-ip is
valid if and only if the route resolves to cf-public (on cf-protected the
trusted address is the connector pod). A single-homed anonymous-read route
REQUIRES {key: source-ip, requestsPerMinute: N, routePerMinute: M}, on every
environment including dev; the chart guard and the validator fail it
otherwise. routePerMinute is mandatory, not optional, because IPv6 buckets
are per /128: anyone with an IPv6 range otherwise avoids the per-address
budget, and the leaderboard hostnames answer over IPv6. The route ceiling is
the IPv6 flood control. A budget never renders while the hostname is
dual-homed (§15). Starting numbers on all three environments: 600 requests
per minute per client address, 6000 per minute for the route, revised from
2xx/429 data. Sizing: one portal index view is /v1/benchmarks plus one
/v1/leaderboard/{id}?top=200 per board (65 boards on dev), about 66 API
calls plus static assets, and each board call runs a whole-board pass in the
app, which has no rate limit of its own. This supersedes the 2026-09-22
"no numbers on dev or staging" decision for this route class only;
public-intake is unchanged. Accepted, revisit: the limiter runs fail-open
with one replica (no rateLimit.failClosed), as §15 recorded.
(e) Admission and the fleet contract — amends §15 Admission. The
cf-public-fleet NetworkPolicy's egress gains exactly two rules:
sf-scoreboardTCP 9106, the leaderboard backend. It may land early, with the inert preparation: no route reaches it until a hostname is listed.- TCP 443 to Cloudflare's 15 published IPv4 ranges
(
https://www.cloudflare.com/ips-v4), for the Access JWKS. This rule lands in the change that first puts a JWT route oncf-public, not before. The Envoy proxy, not the controller, fetches the JWKS (https://openmined.cloudflareaccess.com/cdn-cgi/access/certs); without this rule every write fails closed with 401. An FQDN policy is not available: Ciliumenable-l7-proxyisfalseon aks-dev, aks-staging and aks-prod (read 2026-10-05), and AKS ACNS FQDN filtering is new spend. The Access host's own addresses (today inside104.16.0.0/13) can change, so/32s are not used. Precedent for the shape only (anipBlockegress to Cloudflare on 443):kubernetes/apps/firecall-edge/overlays/dev/activation/cloudflare-egress.yaml. That file pins exact tunnel-edge/32and/128addresses, the address choice this rule rejects. Accepted: the ranges front much of the Cloudflare-proxied internet, so a compromised public proxy gains that outbound path; never0.0.0.0/0without an owner decision.
The sf-scoreboard ingress gains one cf-public peer as in (c). Nothing else
may reach port 9106 except the Gatus pod and, until the last move is done,
the cf-protected fleet. The fleet's ingress stays the public connector only
on 10080 (§15): that remains the only guard against an in-cluster caller
forging CF-Connecting-IP.
(f) Transition, per hostname, and the kill switch. Prerequisites: the
namespace label, the two egress rules and the ingress peer live, or landing
in the same change as step 1 on the first environment (the 443 rule, (e));
the environment's public tunnel, record and connector live ((i)); and, before
the staging move, a checked-in runbook (under docs/runbooks/) for moving a
hostname between tunnels and its revert. Then, in this order:
- The split (staging and production) and the hostname on both lists merge
while Access still covers the whole hostname. Both routes attach to
cf-publicandcf-protected. - The owner adds the hostname's ingress rule on
sf-public-<env>through the API (the dashboard refuses a hostname whose record exists), with the origincf-public-internal.envoy-gateway-system.svc.cluster.local:80. - The platform confirms that both
cloudflared-publicpods logged the new configuration with the hostname. Adding the rule before the CNAME switch avoids the 2026-09-23 404. - The owner edits the proxied CNAME's target to the new tunnel's
cfargotunnel.comname. - The owner removes the rule from the old tunnel through the API — never a dashboard route delete. Afterwards the hostname appears in exactly one connector's latest configuration.
- The hostname leaves the dual-home list: single-homed, the budget renders.
- On the environment's
cf-publicpods: the strip proof and theCF-Connecting-IPspoof proof are recorded, and a burst test shows 429 above the per-address number and 200 after the window. The burst test runs on dev and then on staging before the staging Access edit. The tenant has confirmed the data surface, and on production the owner has signed (g). - The Access edit, LAST, in place ((c)).
On dev the shared application is already path-scoped and reads are already public, so step 1's Access condition and step 8 do not apply. Step 7's strip, spoof and burst proofs are still recorded on dev after step 6, and the staging move does not start without them. The tenant confirmation gates only the staging and production Access edits.
Rollback before step 5: the CNAME goes back. Between step 5 and step 6: the rule is re-added on the old tunnel through the API, then the CNAME goes back. After step 6: revert the single-home change first. Kill switch for exposure: restore the whole-host Access destination. It takes seconds, needs no deploy, and leaves writes and the tunnel untouched.
(g) Production conditions (signed by the owner before the production Access
edit). env: prod for anonymous-read is admitted by this section for the
listed production hostname; public-intake on prod stays denied (§15). The
§15 production ladder, item by item, for this class. The disposition column
is the target the owner signs. "Met" means the owner signs against recorded
evidence; on 2026-10-05 only item (1) and the declaration half of item (11)
have that evidence, and every other "met" or "required" item gets its
evidence before the production Access edit. "Required" means the item is a
condition in its own right. "Accepted in writing" means the owner's signature
accepts the gap:
| §15 ladder item | Disposition at signing, for anonymous-read |
Evidence or reason |
|---|---|---|
(1) aks-prod registered, pulse-edge/overlays/prod renders the base, the production vault admits sf-edge |
met | P1-25, P1-45: the production edge and its tenant connector run |
(2) separate production public tunnel and record, ≥2 connectors on ≥2 nodes, PDB, sf-edge quota for four pods |
met, except two nodes: accepted in writing | sf-public-prod, cloudflared-public 2/2 with its PDB; one user node per cluster (a second is new spend), the same failure domain as cf-protected today |
| (3) budgets from two weeks of staging 2xx/429 data, IPv6 posture recorded | accepted in writing | the starting numbers in (d), revised from data; IPv6 posture recorded in (d) (routePerMinute mandatory) |
(4) rateLimit.failClosed and two limiter and Redis replicas with PDBs, or explicit acceptance |
accepted in writing | fail-open with one replica (d); the alternative is controller-wide |
| (5) zone plan, WAF and Bot Fight Mode decided; production Turnstile with real keys | met for plan and BFM (attested); Turnstile not applicable | the read route has no app-side gate |
| (6) full attestation set per production hostname | met | path-scoped Access application with its AUD unchanged; no whole-host application; no Bypass; "Remove visitor IP headers" OFF; BFM status; plan |
| (7) paging provider and alerts | accepted in writing | no paging provider; rows are board-only |
| (8) kill-switch drill on staging with measured time-to-dark, and a runbook | met by the Access-revert drill | for this class the exposure switch is the Access destination (f), not the DNS, tunnel rule or connector (those take reads and writes dark together); the staging drill times the revert; the checked-in tunnel-move runbook is a prerequisite of the staging move (f) |
(9) cf-public identity-strip sentinel and the CF-Connecting-IP spoof proof |
strip proof required; spoof proof required; standing sentinel accepted in writing | the static k8s-validate.sh assertion that the mechanism change adds ((c)), plus a strip proof recorded on the staging and production cf-public pods before each Access edit; the spoof proof is recorded on the same pods before each Access edit, because the source-ip budget keys on CF-Connecting-IP and the proof shows a caller cannot choose its own bucket (no spoof proof for cf-public is in the ledger today); a standing pt-echo route on cf-public is extra surface |
| (10) access-log retention bound, incident-mode toggle, production log export | accepted in writing | the fleet log omits client-IP fields (§15); cf-ray keeps requests attributable; production export is open under WS6 |
| (11) production declarations never auto-merge; SDK 429 back-off verified | met for declarations; back-off accepted in writing until the tenant verifies it | production declarations change only by a reviewed pull request; the numbers in (d) sit well above one portal view |
Plus, required for this class:
- Staging complete, with a timed Access-revert drill recorded.
- The production public connector 2/2 with its PDB.
- The strip proof and the
CF-Connecting-IPspoof proof on the staging and productioncf-publicpods, each recorded before that environment's Access edit. - A recorded burst test (429 above the per-address number, 200 after the window) on dev and then on staging, before the staging Access edit.
- The checked-in tunnel-move runbook, merged before the staging move.
- The tenant's written confirmation of the public data surface: submitter
email local parts; full author emails where local parts collide;
/docs,/redoc,/openapi.json; the private board's metadata (its listing is by design, its rows are not). - The budget live on the single-homed production route before the edit.
- IPv6 per-address bypass covered by
routePerMinute.
(h) Monitoring per environment (class obligation). Four Gatus rows per
listed hostname, ignore-redirect where a 302 is the answer:
- Public read:
GET /v1/benchmarksanswers 200 (a 302 means an Access application crept back over the read path). - Write refusal:
POST /v1/scoresanswers 302 to Access (any other answer, Envoy's 401 included, means the path-scoped application is gone). - Write JWT:
POST /v1/benchmarksanswers 401Jwt is missing(Envoy's check on a path Access does not cover). - JWKS reachability:
POST /v1/benchmarkswith a crafted RS256 token sent inCf-Access-Jwt-Assertion(the extraction header in (c)) — JWT header{alg: RS256, kid: probe}, issuerhttps://openmined.cloudflareaccess.com, the hostname's full 64-character AUD (a short one answers 403Audiences in Jwt are not allowed), a futureexp, any base64url signature — answers 401Jwks doesn't have key to match kid or alg from Jwt. The answerJwks remote fetch is failedmeans the 443 egress in (e) broke. No other row sees that failure: the no-token row is answered before any key lookup.
The existing staging and production rows that expect a refusal on reads
change in the same window as that environment's Access edit. Outside-in
probes are the default proof. A kubectl port-forward or a crafted probe into
the staging or production clusters needs the owner's per-action go-ahead.
(i) Records. One tunnel per environment, sf-public-<env>
(sf-public-dev, sf-public-staging, sf-public-prod), in the tenant's
Cloudflare account. One record per environment,
sf-public-tunnel-token-<env>, in that environment's vault
(kv-pulse-dev-eastus, kv-pulse-staging-eastus,
kv-openmined-prod-eastus), written only with
scripts/ops/write-vault-record.sh --shape tunnel-token (break-glass window,
revoke, zero standing access), before that environment's connector merges.
Never reuse a token across environments: Cloudflare balances a tunnel across
all its connectors, so traffic would reach a cluster without the route.
(j) Amendment 2026-10-22 — open submission, a write budget, and a correction (OME-1286, OME-1649). Decided by the owner on 2026-10-22.
- Open submission. On every environment, the Access application on a listed hostname's write path has one policy: Allow, Include Everyone. Any person who can log in may submit. The login stays: a submission is never anonymous, and the row's submitter is the verified email that Envoy stamps. This supersedes "the policy and its Include list stay as they are" in (c).
- One application per hostname stays. The tenant proposed folding
/v1/scoresinto the engine's application (OME-1649). The owner rejected it: §11's scope guard forbids a shared AUD on staging and production, every engine user could then write scores, and Include Everyone on a shared application would open the engine too. For the same reason, dev's write path moves to its own application before its policy opens (§11 is amended when the owner creates that application). - Seats. Every Access login takes one seat from the
openminedZero Trust team. Every Access application draws from that one pool, staff applications included. On the free plan the pool is 50 seats, and when it is full Cloudflare refuses the next login of anyone. The plan, seat expiry and the seat count are the owner's (new spend). Owner decision 2026-10-29: open admission is capped at 50 people a month; seats are not reserved in advance (36 in use that day); seat expiry is one month. The seat watch (.github/workflows/cloudflare-seats.yml, runbookdocs/runbooks/cloudflare-seats.md) counts the seats in use every six hours and warns in Slack from 80% of the plan's seats (once a day) and from 90% (every run), so seats are bought when they are needed. - Write budget — amends (d). The write route on a listed hostname carries
{key: source-ip, requestsPerMinute: 30, routePerMinute: 300}on every environment, revised from 2xx/429 data. It renders as BackendTrafficPolicy<route>-public-budget. - Correction to (c). The bullet "a path-scoped application still works
for SDK submissions" was wrong for ScreamingFace clients before 0.1.6.
Those clients ended the Access login on the host root
/. After the 2026-10-07 edits/is public, so Cloudflare never transferred the application token, and a submission to staging or production waited out the 300-second login (OME-1649). Dev worked only becauseleaderboard.devshared the engine's application. Client 0.1.6 (tenant #1486) ends the login on the challenged path. The "real SDK submission after each Access edit" proof was not run in October. - Required proof after every Access edit on a listed hostname. One real
submission with the current released client (0.1.6 or later) by a person on
no allow list. The row shows that person's email; an anonymous
POST /v1/scoresstill answers 302; and the environment's engine hostname still refuses that person. The result goes in the hostname's attestation table. A reading of the client's challenge handling is not this proof.
Rejected alternatives. Anonymous reads on cf-protected (option 2): the
§13 interim shape made permanent, with no per-caller budget, because §13
gate (c) forbids clientIPDetection there. Writes on a separate hostname
that stays on cf-protected: keeps §15's "no SecurityPolicy on cf-public"
but needs a tenant SDK and portal change and breaks older SDKs. A new tier
for JWT writes on cf-public: more ADR, validator and documentation surface
than two constants. An Access Bypass on the read paths (§13).
Corrective scope, with the change that makes each record false.
docs/security/gateway.md posture rows and its rate-limit gap line;
docs/runbooks/declare-a-route.md; kubernetes/apps/tenant-edge/README.md
and values.yaml; scripts/ci/k8s-validate.sh (the new static strip
assertion in (c), and the anonymous-route tripwire comment that calls every
cf-public parent "public BY DEFINITION"), with the mechanism change;
docs/runbooks/cloudflare-tunnel-token-rotation.md (staging and production
public rows); a runbook for moving a hostname between tunnels and its revert,
before the staging move ((f)); a docs/tenants/screamingface/apps/scoreboard.md attestation table;
docs/tenants/screamingface/index.md ("No public door on production") and the
aks-prod pulse-edge.yaml "DARK" comment, with the production move. CLAUDE.md
carries this section's gate in the W24/W25 blocker class (this change).
Consequences
- Stage 3 of the ScreamingFace plan builds the
cf-protectedtier Gateway (platform namespace) instead of scaffolding for a tenant-owned Gateway; their WU-4/WU-5 shrink accordingly (communicated on their #413). - The platform inherits obligations that per-tenant gateways didn't have: route/policy templating in /onboard-repo, descriptor prefixing, telemetry partitioning, and (before GA / second tenant) the hostname admission policy.
- ScreamingFace needs one follow-up:
SecurityPolicymoves platform-side (they supply AUD + specifics), amending the earlier "stays in your repo" message. - Two edge stacks continue (ingress-nginx for platform apps under
*.dev.openmined.org, the tier gateways for tenants) — consolidation onto Gateway API stays a later deliberate call. - A shared data plane means a proxy-fleet incident is platform-wide; the tier gateway is operated as tier-0 (HA replicas, PDB, kill drills per T8).