Skip to content

ADR-0005: Multitenant gateway platform — shared tiered gateways, three planes

Date: 2026-07-18 Status: accepted in PR #34 · amended 2026-07-22 (§4: single shared listener adopted now, not per-tenant listeners — forced by an Envoy Gateway JWT-aggregation bug; see §4) · amended 2026-07-23 (§7: the mechanics of a federated tenant on the shared tier — a tenant-owned cloudflared into the shared Gateway; ScreamingFace is the first) · amended 2026-08-13 (§10: the preview exposure — a dedicated preview subdomain with one preview-tier Access app and AUD per tenant) · amended 2026-08-19 (§11: dev-tier single sign-on — one tenant AUD for user-facing dev services, with privileged surfaces like aigw-admin kept separate; staging/prod keep per-service AUDs) · amended 2026-08-31 (§13: the public-intake route class — the first fully-anonymous WRITE surface; dev-only interim on the shared listener behind a BLOCKING cf-public split gate before any staging/prod exposure) · §14 proposed 2026-09-08 (self-service route declarations — a tenant declares a hostname in one file, the platform renders the route and the tier policy, CI and a native ValidatingAdmissionPolicy enforce hostname ownership, closing the control §4 deferred; Status: proposed until the owner accepts) · §15 proposed 2026-09-22 (the cf-public gateway — the public-intake class leaves the shared listener onto its own fleet and its own tenant-owned tunnel per environment, with fail-closed true-client-IP detection; analytics.dev.screamingface.ai joins the member list; staging admitted behind the split, production behind a ladder; Status: proposed until the owner accepts) · §16 accepted for implementation 2026-10-05 by owner decision (whole-public hostnames: each leaderboard hostname moves whole to cf-public on dev, staging and production — anonymous reads under a per-address and a route-wide budget, writes behind a path-scoped Access application and the Envoy-verified Access JWT; amends §14 and §15; the production conditions in §16(g) are signed by the owner before the production Access edit) Amends: ADR-0004 (supersedes its §2 per-tenant Gateway edge; everything else in 0004 — AppProject, namespaces, vault paths, netpol, vcluster triggers — stands)

Context

ADR-0004 gave each tenant their own Gateway API Gateway (own Envoy proxy fleet) in their edge namespace. Design alignment on the gateway platform (The original alignment draft is historical private material and has no authority over this ADR.) concluded that per-tenant gateways don't scale — the product is a small menu of shared, platform-owned gateways that many tenants attach to — and contributed the tier framing, the identity contract, and the Keycloak migration path. This ADR distills that into decisions, resolving its open questions D1–D6.

Decision

1. Three decoupled planes

Plane Component Owns Never does
Edge Cloudflare (Access + Tunnel) DDoS, TLS at edge, first auth gate, origin unreachability source of truth for identity
Identity Cloudflare Access (OTP) now → Keycloak later (roadmap 3c) who is this: users, SSO, tokens route traffic
Enforcement Envoy Gateway (Gateway API CRDs, pinned version) JWT validation, routing, rate limits, header contract, tenant isolation trust a header it didn't set

Each plane is replaceable without rearchitecting the other two — this is what makes the Keycloak migration a config change (§7).

2. Shared tiered gateways, not per-tenant

One platform-owned Gateway per tier, in a platform edge namespace; a dedicated proxy fleet per tier (Envoy Gateway mergeGateways remains a later cost knob). Tenants attach HTTPRoutes from their own namespaces. First tier: cf-protected (tunnel-only exposure, Access JWT auth, no public IP exists). Future tiers (internal for east-west, partner-mtls) are defined by the five-part tier contract (exposure · auth · traffic guarantees · availability · admission) and named by guarantee, not implementation.

3. A tenant, at the gateway layer, is {domain, aud, upstream} + a tier

One Access application per tenant → one unique AUD, which is the tenant boundary at the auth layer: a token minted for tenant A is cryptographically invalid on tenant B's routes. Never share an AUD; never wildcard audiences. The tuple survives the Keycloak migration (aud becomes a client ID). Everything tenant-side of onboarding is a PR of tenant-namespace resources, stamped from the /onboard-repo template.

4. Routing granularity: per-tenant listeners now, virtual hosts later

Now (tenant count ~1, no admission controller on the cluster): each tenant gets a hostname-scoped listener on the tier Gateway with allowedRoutes selecting only that tenant's namespaces — hostname squatting is structurally impossible, no extra machinery.

Later (trigger: approaching the 64-listener cap, hostname churn, or Kyverno/ OPA landing for other reasons): collapse to one listener with host-header routing + an admission policy asserting a namespace may only use hostnames from its registered domain list. The switch is invisible to tenants.

AMENDED 2026-07-22 — "later" is now, forced by an Envoy Gateway bug. The per-tenant-listener model does not work with per-tenant JWT on Envoy Gateway. Multiple hostname listeners on the same port are merged by Envoy into one listener, and EG's JWT-requirement aggregation across that merge is broken: only the first merged listener's SecurityPolicy requirement is written into the shared JWT filter's requirement_map, so every other tenant's route fails closed with Failed JWT authentication: Wrong requirement_name (deterministic; confirmed via Envoy config_dump; not fixed by controller/proxy restarts; area of envoyproxy/gateway#8313, which upstream closed as won't-fix). Discovered standing up the second JWT tenant (ScreamingFace fusion) alongside the pt-echo sim-tenant.

Decision: adopt the single-listener host-routing model now. One listener → no Envoy merge → JWT aggregates across all tenant routes correctly. Tenants scope by their HTTPRoute hostname. The lost property — structural hostname-squatting prevention (per-listener allowedRoutes) — is deferred to an admission policy (Kyverno/OPA: a namespace may only use hostnames from its registered domain list). Accepted gap for the trial / small trusted tenant set; admission control is a tracked follow-up and a GA prerequisite. Revisit per-tenant listeners only if EG fixes the merge aggregation.

5. SecurityPolicy is platform-owned

The policy object is stamped by the platform from the onboarding template; tenants cannot weaken their own tier guarantee. Invariant fields (JWT extraction from Cf-Access-Jwt-Assertion, issuer/JWKS, claim→header mapping to the identity contract, header stripping on the route) are template-fixed. Tenants supply values: their AUD, allowed CORS origins, optional claim-based authorization rules (e.g. required group), and their rate-limit numbers (which live in BackendTrafficPolicy, tenant-tunable within tier ceilings).

6. The identity contract (adopt now, never break)

Header Meaning Source
X-User-Email verified human identity JWT email claim
X-User-Id stable subject id JWT sub
X-Tenant tenant context of the route route/policy config
X-Service-Id machine identity service-token claim (common_name)

Set only by the gateway after cryptographic validation; all inbound instances stripped unconditionally first. Backends read only these — never Cf-Access-* or provider token internals. This contract is what makes swapping Cloudflare→Keycloak invisible to every backend.

7. Cloudflare topology: centralized by default

Tenant hostnames live in the platform's Cloudflare account / Zero Trust org: one issuer, one JWKS, one wildcard tunnel (≥2 cloudflared replicas), tenant teams get scoped dashboard access to their own Access policies. Federated (tenant-owned Cloudflare account, own issuer, own tunnel) remains supported — provider config is per-policy — but is the exception, not the default. ScreamingFace is grandfathered federated (their zone, their org, per the sent proposal); revisit only with their agreement.

Amendment 2026-07-23 — how a federated tenant rides the shared tier. A federated tenant does NOT get its own Gateway (that would reintroduce the per-tenant edge §2 rejected). Instead: a tenant-owned cloudflared Deployment in the tenant's edge namespace, running the tenant's own tunnel token (from Key Vault via ESO, break-glass write, never in git), forwards to the shared tier's cf-protected-internal Service. A platform-authored HTTPRoute for the tenant hostname attaches to the shared listener. A platform-authored SecurityPolicy uses the tenant issuer, AUD, and JWKS. Per-route JWT providers aggregate cleanly on the one listener, so each tenant keeps an independent authentication boundary. Result: shared tier, tenant edge. The tenant controls its hostname zone and tunnel inputs. The platform owns the controller, listener, route, and enforcement policy, as ADR-0006 requires.

Observed at onboarding (2026-07-29): screamingface.ai's Access lives in the OpenMined Zero Trust org (issuer openmined.cloudflareaccess.com), not a separate tenant org — so the live shape is "own zone + own tunnel, same org." Mechanically identical (per-app AUDs remain the tenant/service boundary; the tunnel + zone are tenant-scoped); the fully-federated own-org variant remains supported for a tenant that brings one.

8. Keycloak path (executes with roadmap 3c, designed now)

  • K1 (low risk, per-tenant, canary by PR): Keycloak becomes a generic-OIDC IdP inside Cloudflare Access. Same Cloudflare JWT downstream; zero gateway changes. Tenant A moves to SSO while tenant B stays on OTP.
  • K2 (optional): gateway validates Keycloak tokens directly (issuer/JWKS swap in the SecurityPolicy); Cloudflare drops to pure edge.
  • Tenant mapping: client-per-tenant in one realm (company-wide SSO goal).
  • Keycloak is tier-0 once login depends on it (HA per roadmap 3c).

9. Security invariants and standing tests

Non-negotiable invariants: header strip + re-inject (the single most important one); no reachable origin for cf-protected; JWT validated at the gateway (issuer + audience + JWKS), never in backends; network path lockdown (backends ← proxy only, proxy ← cloudflared only, tenant↔tenant denied); per-tenant BackendTrafficPolicy for blast radius; JWKS fail-closed with a measured cached-key survival window; tenant-prefixed rate-limit descriptors in the shared Redis; per-tenant telemetry partitioning.

The T1–T8 standing test checklist from the alignment doc (policy independence, cross-tenant token replay → 401, forged headers in-cluster → 401, forged headers through a valid session → claim-derived values win, tenant-B meltdown drill, tier-label admission rejection, JWKS outage drill, cloudflared/proxy kill mid-traffic) becomes the acceptance suite for the gateway platform — repeatable checks, not one-off pokes.

10. Amendment 2026-08-13 — the preview exposure (PVW-D04)

Routed by the preview-environments program (PVW-D04, work package PVW-M0-W04 — https://github.com/OpenMined/infrastructure/blob/main/docs/programs/preview-environments/decisions.md). The owner decided DECIDE PVW-D04: OPTION A on 2026-08-13.

Preview environments ride the existing cf-protected tier under a dedicated preview subdomain. The design goal that shaped everything: no per-preview Cloudflare write, ever — Cloudflare's tunnel and Access write permissions exist only at account scope, so any automated per-preview minting would require token custody over the tenant's whole account.

  • Hostname shape: <app>-pr-<N>.preview.dev.<tenant-domain> (first tenant: preview.dev.screamingface.ai). Every wildcard the design needs is a full leftmost label, which is the only form DNS records and tunnel rows can express.
  • Five-part contract for the preview exposure: exposure — tunnel-only through the existing shared Gateway; auth — Access JWT validated against one preview-tier AUD per tenant; traffic — preview-class, non-gating, budget-capped; availability — best-effort; admission — machine-generated preview routes may claim only hostnames matching the preview pattern.
  • One AUD per tenant per tier. All previews of one tenant share one Access application and one AUD. §3's rule protects the boundary between tenants and tiers and is unchanged; this amendment fixes its reading inside one tenant's preview tier, where per-preview AUDs would cost account-token custody plus app churn. Per-preview Access applications are recorded as the hardening path if per-preview audiences are ever needed.
  • The one-time tenant ask — four dashboard actions, no token, nothing recurring: one proxied wildcard DNS record for the preview subdomain pointing at the tunnel; one wildcard tunnel public-hostname row to the shared origin service; one wildcard Access application on the same pattern with the same policy shape as the dev apps; one advanced-certificate wildcard SAN for the new depth (Total TLS does not cover tunnel-fronted hostnames).
  • Platform side: one preview-tier SecurityPolicy validating the preview AUD; per-preview platform-stamped HTTPRoutes on the single shared listener (§4 unchanged); X-Tenant stamped per route; the identity strip inherited from the tier ClientTrafficPolicy.
  • Rejected: flat names at the service depth (DNS and tunnel wildcards cannot express partial labels, forcing either a zone-wide wildcard or per-preview API writes); per-preview Access applications (above).
  • Implementation checks: inspect the live certificate covering the current service depth before freezing the ask list; live-test that a wildcard tunnel row routes; note that wildcard Access apps receive no preemptive cookies (the first request redirects to login — accepted).

11. Amendment 2026-08-19 — dev-tier single sign-on (one tenant AUD for user-facing services)

Decided by the platform owner on 2026-08-19. Narrows §3's per-service AUD reading inside one tenant's dev tier, and only for user-facing services. §3's between-tenant and between-tier boundary is unchanged.

Decision. In the dev tier, a tenant's user-facing services share one Cloudflare Access application and one AUD — one app scoped to the explicit user-facing dev hostnames (first tenant: fusion.dev.screamingface.ai and leaderboard.dev.screamingface.ai). A user authenticates once and the resulting token validates on every listed user-facing dev subdomain of that tenant. The platform SecurityPolicy for each such route validates that single shared AUD.

  • User-facing today: fusion.dev and leaderboard.dev (its write path). Both move to the shared AUD. Leaderboard reads stay public (no policy).
  • Added 2026-08-31: reports-internal.dev.screamingface.ai (report-intake's identified-report path; tenant added the hostname to the shared app the same day). Owner ruling: shared AUD — filing an identified bug report is an action any general dev user may perform. Recorded dissent (public-route assessment, verbatim position): the analysts split on whether these writes pass the exception test AT ALL — tracker writes arguably FAIL the "action a general dev user must not perform" test, arguing for an own AUD; the assessment held the question genuinely open, "must be decided, not defaulted". The owner decided it as above. Chosen reopening condition: revisit toward an own AUD if authenticated-flood abuse shows up in triage (the identity path carries no rate limit and skips Turnstile by design).
  • Amended 2026-10-30: leaderboard.dev leaves the shared application. Its write path (/v1/scores) moves to its own dev Access application with its own AUD, whose one policy is Allow, Include Everyone (§16(j), open submission, OME-1286). An Access policy belongs to the whole application, not to one destination, so the shared application could not open the submit path without also opening fusion.dev and reports-internal.dev. The trade-off below ("the same trusted team") no longer holds for leaderboard writes on dev; the write route carries the §16(j) write budget. Dev users now log in once for the engine and once for the leaderboard, as on staging and production.
  • New user-facing dev subdomains are added to this app deliberately (default-closed): a subdomain gets the shared login only once listed, so a future privileged subdomain is never swept in by default.
  • Rejected: a *.dev wildcard app. A wildcard is default-open — it would formally cover aigw-admin and every future subdomain, staying safe only by Cloudflare most-specific-app precedence plus the platform's per-route AUD split (defense in depth, but implicit). The explicit hostname list is default-closed and states the intent directly, which matches a design that has a privileged exception to protect. (The single-label wildcard also never reached previews — *.preview.dev is a level deeper — so the choice does not affect them either way.)

Privileged-surface exception (holds the line). Services that gate administrative or credential access keep their own scoped AUD and their own smaller allow-list, and are never folded into the shared dev AUD. The concrete case: aigw-admin administers every tenant's provider credentials and admits only the aigateway operator allow-list; sharing its AUD would grant credential administration to every dev user. It stays separate. The test for the exception is "does this surface authorize an action a general dev user must not perform?" — if yes, its own AUD.

Accepted trade-off. One shared AUD means any dev-authenticated user's token is accepted by every user-facing dev service of that tenant — for example, a fusion user can write dev leaderboard scores. In dev this is acceptable: the data is disposable and the user population is the same trusted team. It is not acceptable where the action has integrity or privilege weight, which the exception above carves out.

Scope guard. This amendment applies to the dev tier only. Staging and production keep §3's per-service AUD as the rule — the STP program decides their access model under its own gates. Do not copy this dev simplification upward.

Implementation. The shared Access app — scoped to the user-facing dev hostnames — is a one-time tenant dashboard action (returns the AUD). The platform side points the fusion-cf-access and leaderboard-cf-access policies at the shared AUD; aigw-admin-cf-access is untouched. Until the tenant returns the shared AUD, the value stays a fail-closed PENDING_* sentinel (blocked by the k8s-validate.sh tripwire), so the change cannot land half-configured.

12. Amendment 2026-08-24 — preview implementation reference

ADR-0013 suspended the PVW delivery workflow. It did not change the Preview exposure decision in §10. The suspended PVW records remain historical evidence. The Preview architecture guide now defines the current implementation.

13. Amendment 2026-08-31 — the public-intake route class (first fully-anonymous write surface)

Decided by the platform owner on 2026-08-31, with the tenant's agreement to deploy the interim shape now and harden later. Grounded in the 2026-08-28 public-route assessment (docs/consultants/2026-08-28-report-intake-public-route-assessment-for-codex.md), whose findings were adversarially verified against the tenant app's source, Envoy Gateway and Cloudflare documentation, and the live cluster.

What this admits. A hostname posture this ADR did not previously allow: a public-intake hostname — no Cloudflare Access at all, whole surface anonymous including writes. The tier had ONE prior anonymous exception, read-only: leaderboard reads (method-split route with no SecurityPolicy, §11). The separate admin-edge rail carries an analogous empty-SecurityPolicy precedent (the status page) — but that is a different gateway with a different auth stack (Entra OIDC, public LB, no tunnel): precedent for the mechanism, not a tier exception. This class is anonymous WRITE, which is why it gets its own contract instead of riding either precedent.

Reconciliation with §2. §2 defines cf-protected as "tunnel-only exposure, Access JWT auth, no public IP exists". That auth line is henceforth the tier default, not an absolute: the recorded departures are §11's leaderboard-reads exception and this class. The gate below exists precisely because an anonymous-write surface does not belong on this tier — the resolution is the class moving OUT (to the public gateway), never §2 being redefined down. First and only member: reports.dev.screamingface.ai (report-intake, anonymous error reports into the tenant's private tracker). Its identified companion reports-internal.dev.screamingface.ai is a normal §11 shared-AUD route, not part of this class.

The §2 five-part contract for this class (dev, interim), plus one class-specific obligation:

  • Exposure — tunnel-only, no public IP. Unchanged from §2.
  • Auth — none at the edge. The only per-request gate is Cloudflare Turnstile, verified in-app against a platform-delivered secret (sf-report-intake-dev-turnstile-secret). Identity stays gateway-owned on this route like every other: the tier-wide earlyRequestHeaders strip is inherited automatically, the route stamps X-Tenant with set, and the route additionally removes CF-Connecting-IP toward the backend (the app's own contract). A forged Cf-Access-Jwt-Assertion is inert here — no SecurityPolicy consumes it and the app does not read it.
  • Traffic — INTERIM: no per-caller rate limit exists (accepted risks below). The route path-matches /v1/reports only — never /, so /healthz and /readyz (an unbudgeted DB query) stay unpublished. Explicit per-route timeouts.
  • Availability — best-effort, same as the tier: the class rides the shared fleet's tier-0 posture until the split below; no additional guarantee.
  • Admission — default-closed. The member list of this class is this section; adding a hostname means amending it, and edge objects for a public hostname are platform-PR-only (the tenant fence already rejects chart-rendered edge objects). Enforcement, not convention: a k8s-validate.sh tripwire (same pattern as the §11 PENDING_* sentinel) asserts every cf-protected route carrying no route-targeted JWT SecurityPolicy appears in the checked-in anonymous-route allowlist at scripts/ci/anonymous-routes.allowlist — route-level, not hostname-level, so method-split hostnames (leaderboard: gated writes, anonymous reads) are covered, and the allowlist inventories ALL deliberately-anonymous cf-protected routes annotated by class (§11 read exception vs §13 public-intake). Precedence when the records diverge: the allowlist is the ENFORCED record, this section is the decision record, gateway.md's posture table is the human view. On this listener absence of policy fails open into public — and preview churn has already produced orphaned SecurityPolicies — so the fail-open must be CI-visible, not review-remembered.
  • Monitoring (class obligation, additional to the §2 contract parts) — mandatory Gatus posture pair: the public hostname answers with NO Access redirect (a 302 to cloudflareaccess.com appearing means an Access app crept over the hostname — scope-widening detection in reverse), and the identity hostname still redirects unauthenticated. Plus an alert on the anonymous-POST 503 rate: a broken Turnstile secret answers every anonymous report with 503 while the pod reports healthy — no liveness, readiness, or naive 200-check can see that failure.

The edge slice (next PR) is mechanically specified by this section: (1) the two HTTPRoutes and the shared-AUD SecurityPolicy; (2) the scripts/ci/anonymous-routes.allowlist file plus its k8s-validate.sh tripwire; (3) the Gatus posture pair and the anonymous-503-rate alert; (4) the attestation table's move into docs/tenants/screamingface/. If that PR is ever authored without this session's context, this list is the contract — none of the four may be dropped silently.

Boundary artifact — attestations instead of an AUD. Every other hostname's recorded boundary is its AUD, verifiable from git and proven by cross-AUD 403. A public-intake hostname has no AUD, so its boundary record is a dated, named attestation set for the facts that live in the tenant's Cloudflare dashboard and cannot be verified from git: no Access application covers the hostname (attested — "no app", never a Bypass policy, which is origin-identical but unlogged); Turnstile widget with REAL keys (test keys always pass); zone plan tier (Free — a host- or method-scoped edge rate rule is not expressible); Bot Fight Mode posture (ON, see risk 3); no pre-existing DNS record on either hostname. The Turnstile sitekey is public and recorded deliberately — 0x4AAAAAAEjIHU_eaxfIZ-u6 — because Cloudflare's always-pass test sitekeys are well-known constants, so the recorded value is what lets a future reviewer verify "real keys" without dashboard access. Attested by the owner 2026-08-31; the table moves into docs/tenants/screamingface/ with the edge slice. Re-attestation triggers: plan change, tunnel change, any Access app change, any abuse incident.

Accepted interim risks (owner, 2026-08-31):

  1. No per-caller rate limiting. The app's in-process limiter collapses to one shared bucket behind the tunnel (its TCP peer is the proxy), the per-IP BackendTrafficPolicy design remains parked, and the Free plan cannot express a scoped edge rule. Until the gate below is paid, the only per-request gate is Turnstile — a cost-raiser (solver farms sell tokens), not a limiter.
  2. Shared blast radius. Anonymous traffic rides the same cf-protected Envoy fleet and the same tenant sf-edge tunnel as all authenticated traffic. A flood degrades the tenant's own product first.
  3. Bot Fight Mode is ON, zone-wide, unscopable on Free. It challenges at the edge BEFORE Turnstile ever runs and may block legitimate SDK POSTs. (Existing tenant traffic has run with it on — an inference from live service, not an attested fact.) Decision point: the first SDK acceptance test against the live hostname. If reports are challenged, the response — switching BFM off zone-wide (a tenant action), a plan change, or the Worker fallback — is decided THEN; none is pre-committed here.
  4. Turnstile-for-headless-SDK is an open tenant question. Turnstile is a browser widget; the hostname's primary caller is a Python SDK. Until the tenant answers how the SDK obtains a token, the anonymous path may 403 legitimate SDK reports. Accepted for dev; it blocks nothing platform-side.

THE GATE — this interim shape is dev-only and it expires (BLOCKING). Before any of the following, the public-intake class MUST move off the shared listener onto a dedicated public gateway (working name cf-public: own EnvoyProxy fleet on the same controller — the admin-edge gateway-per-trust-model precedent, ~2 pods — with label-scoped allowedRoutes, clientIPDetection (customHeader: CF-Connecting-IP, fail-closed) scoped to that tier only, a per-IP Global BackendTrafficPolicy on the already-live rate-limit service, default-deny NetworkPolicy around the fleet, and its own tunnel decision):

  • (a) any staging or production exposure of any public-intake hostname. The staging-production program MUST treat "public-intake still on cf-protected" as a failed readiness gate of the same class as W24/W25 — a blocker, not a finding. Do not scope, waive, or defer it in an activation review.
  • (b) admission of a second public-intake hostname, any tenant. The member list above may never grow while the class sits on cf-protected.
  • (c) any tier-wide clientIPDetection on cf-protected — forbidden outright. That trust change alters client-IP semantics for every tenant route on the single shared listener and belongs only on the isolated public tier. If per-caller limiting is wanted, that IS the split trigger.

Trigger (a) is the owner's stated condition for accepting the interim shape ("must split before staging and prod prep", 2026-08-31); (b) and (c) are platform-derived conditions of the same gate, from the adversarially verified assessment. Removing or weakening any of the three is a new ADR-level decision, not an edit.

Where this gate is enforced. The delivery-program workflow is suspended (ADR-0013), so no STP document gates anything today — W24/W25 block production because CLAUDE.md's operating rules say so. This gate therefore lives in the same place: CLAUDE.md carries it (edited in this PR, adjacent to the W24/W25 block), and any future staging scoping MUST import it as a day-one blocker rather than rediscovering it here.

Naming note. docs/platform-delivery-plan.md §1.3 sketches a "cf-public" tier for anonymous READS (scoreboard's case). The split gateway this gate requires subsumes that sketch: one public gateway hosts both anonymous-read and public-intake classes, with per-class routes and budgets. Where the §1.3 sketch and this section disagree, this section supersedes.

Rejected alternatives (assessed and adversarially verified; details in the assessment): tunnel-direct-to-app (no gateway in the path — deletes the identity strip, X-Tenant, all edge forensics, and conflates IP trust with identity trust in the app's own knobs; rejected with no adequate mitigation); Access Bypass or embedded service tokens (Bypass is origin-identical to no app minus the audit trail; a token in a public SDK is a public credential); Cloudflare Worker intake on the tenant zone (best invariant preservation, heaviest tenant coupling — held as the fallback if this class is ever declined or retired).

Corrective scope. docs/security/gateway.md and docs/products/gateway-identity-flow.md stated the Access invariant absolutely ("Cloudflare Access authenticates every request", "you cannot smuggle the header past the edge") — already false for leaderboard reads and the status page before this amendment. Both are edited in this PR to tier-scoped statements, and gateway.md gains a per-hostname posture table so exposure is reviewable from git even where it is not enforceable from git.

14. Amendment 2026-09-08 — self-service route declarations and hostname admission

Status: proposed. Decider: platform owner. Drafted with the Phase 1 pilot plan (A3.5: "Envoy Gateway stays. Routes become self-service."; items P1-42, P1-43, P1-44; https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/phase-1-pilot-plan.md). The object names below are the P1-42 design as specified on 2026-09-04. P1-42 has not merged as of 2026-09-08: there is no kubernetes/apps/tenant-edge, no kubernetes/appsets/tenant-edge.yaml, and no hostname policy under kubernetes/infrastructure/tenant-guardrails/. Acceptance conditions: P1-42 has merged, every name here has been re-read from origin/main and corrected, and the owner has answered the pending decisions below. A position the owner has not yet taken is marked Decision pending with its options and its default.

What changes. Tenant edge objects are no longer hand-written by the platform per hostname. A tenant declares one route in one file, kubernetes/tenants/<tenant>/routes/<env>/<route>.yaml (the route declaration), with these fields: tenant, route, env, app, hostname, tier, namespace, service, port, identity, audience, methods, rules[] (path type, path value, request timeout), stripRequestHeaders, and a recorded-not-rendered rateBudget. The file is the values file of the platform chart kubernetes/apps/tenant-edge, which renders the HTTPRoute and, for identity: access-jwt, the tier's SecurityPolicy. The ApplicationSet kubernetes/appsets/tenant-edge.yaml (a git files generator over the declaration path; preserveResourcesOnDeletion: true) delivers one Application per declaration under the AppProject <tenant>-edge (first: screamingface-edge), whose whitelist holds only the edge kinds (HTTPRoute, SecurityPolicy, BackendTrafficPolicy) and whose destinations are the tenant's workload namespaces, never its edge or delivery namespaces. §5 is unchanged: the policy is platform-owned and the declaration supplies values only; the issuer, the JWKS URI, the header extraction, and the claim-to-header map are template invariants. §6 is unchanged: the tier-wide identity strip is inherited, and the route stamps X-Tenant with set. One hostname may carry two declarations only when their methods are disjoint and non-empty (the leaderboard read/write split): a SecurityPolicy targets a whole route, so a method split is two routes.

Tiers by name. Every declaration names one tier. The chart's fail guards and the CI validator enforce the same table.

Tier Identity Audience Methods Environments
user Access JWT on dev may equal the shared dev AUD (§11); on staging and prod unique per hostname (§11 scope guard; pending decision below) any any
privileged Access JWT own AUD, never shared (the §11 exception) any any
anonymous-read none absent non-empty subset of GET HEAD OPTIONS Amended by §16: dev, staging, prod, for a hostname on the §16 list only; gateway cf-public; budget key source-ip with requestsPerMinute and routePerMinute required once single-homed. (Before §16: dev only; the §11 read exception.)
public-intake none absent any; never path / dev only, §13 member list only, at most one declaration (§13 gate, not a decision)

An identity: none declaration still needs its line in scripts/ci/anonymous-routes.allowlist (<tenant>/<env>/<route> <class>); the validator checks every anonymous declaration against that file and flags stale lines. The allowlist stays the enforced record of §13; this section stays the decision record.

The admission control §4 deferred, now specified. §4 (amended 2026-07-22) moved the tier to one shared listener and deferred structural hostname-squatting prevention to "an admission policy (Kyverno/OPA)". It is a native ValidatingAdmissionPolicy, tenant-httproute-hostnames, bound to every namespace that carries the openmined.org/tenant label, with failurePolicy: Fail and a per-cluster param ConfigMap tenant-hostname-suffixes (namespace kube-system, parameterNotFoundAction: Deny) that registers the suffixes a tenant may claim on that cluster (dev: dev.screamingface.ai; staging and production register their own labels, so a dev hostname cannot be claimed on staging). It denies an HTTPRoute with no hostname, a wildcard hostname, or a hostname outside the tenant's registered suffixes. No Kyverno, no OPA: the platform's tenant guardrails are native admission policies already (kubernetes/infrastructure/tenant-guardrails/), and one rule does not justify a second admission engine. Preview namespaces carry the tenant label and comply. The Consequences bullet "before GA / second tenant — the hostname admission policy" is satisfied by this section once it is accepted and the policy is live.

Enforcement, in order. (1) The CI validator scripts/ci/tenant-edge-validate.sh, run by scripts/ci/k8s-validate.sh inside the required validate check, with named checks: check_schema; check_path_agrees (the path segments equal tenant, env, route); check_hostname_suffix (against the same ConfigMap the policy reads); check_namespace_owned (a tenant-labelled Namespace in the registry and a destination of the edge AppProject); check_service_exists (the Service and port exist in the tenant chart at the pinned chart ref, rendered with our values); check_tier_rules; check_render (one HTTPRoute; one SecurityPolicy if and only if access-jwt); check_hostname_unique; check_anonymous_allowlist. (2) The chart's fail guards, which stop Argo from rendering a declaration that bypassed CI. (3) The AppProject fence, which can deliver no other kind. (4) The admission policy, which denies a hostname the tenant does not own regardless of how the object arrived. Every tenant-side change lands as a pull request in this repository; Argo delivers from main only.

Reconciliation with §4, §11, §13. §4's single listener stands; per-tenant listeners remain rejected while the Envoy Gateway merge behaviour is unchanged. §11's shared dev AUD is expressed as a user declaration whose audience equals the shared AUD; the validator rejects that value on privileged. §13's gate is untouched: a public-intake declaration is valid on dev only, for the §13 member list only, never with path /, and at most one may exist while the class sits on cf-protected; the split to cf-public remains the blocking condition for any staging or production exposure. rateBudget is recorded, not rendered: rate limiting is parked, per-path budgets need separate routes, and Envoy Gateway 1.5 rejects sectionName on a BackendTrafficPolicy at admission while the CRD schema accepts it. Nothing here un-parks it.

Delivery of tenant changes (P1-44). A tenant may keep its declarations in its own repository under platform/routes/; a Kargo Warehouse watches that folder and a PromotionTask copies the files into this repository as a pull request that must pass the validator. Production declarations (routes/prod/) are never auto-merged.

  • Decision pending: merge policy for mirror pull requests on dev and staging. Options: (A) automatic when CI is green, the plan's proposal (P1-44); the validator, the AppProject fence, and the admission policy are the controls. (B) Human merge until the validator has run on the six migrated dev routes and one staging change, then (A) for dev and staging only, the WS4 specification's proposal. Default: (A), the plan's. The two records disagree; the owner settles it.
  • Decision pending: Access application shape on staging (P1-41). Options: (A) one application and one AUD per hostname, §3 and the §11 scope guard as written; the validator enforces it on staging and prod. (B) One application per tier (one user-facing AUD for fusion, leaderboard, reports-internal; one privileged AUD for aigw-admin), which needs a §11 scope-guard reconciliation in this section before any staging declaration may share an AUD. Default: (A).
  • Decision pending: anonymous-read on staging (P1-41, P1-42). Options: (A) dev only; staging sits behind the tester allow list (plan A5), so the staging leaderboard is one user route with all methods. (B) Allow env: staging for reads as on dev. Default: (A). Resolved 2026-10-05 by §16: (B), extended to prod, for a hostname on the §16 list only, on cf-public, never on cf-protected once single-homed.
  • Decision pending: the X-Tenant value the chart stamps. Options: (A) the declaration's namespace; five of the six live routes do this, migrating Applied 2026-09-08 (P1-43, #886): the chart stamps the route's namespace; the tenant's code ignores the header (their tests assert identity derived from it is dropped), so it stays edge forensics. The owner can still change it here. aigw-admin changes its value from screamingface to sf-aigw, and no tenant code reads the header (a search of the tenant repository returned nothing on 2026-09-04). (B) A per-declaration override field. Default: (A).

Rejected alternatives. Tenant-authored HTTPRoute objects in tenant namespaces: the AppProject fence and ADR-0006 forbid tenant edge objects, and a route without its tier policy fails open into public on this listener. A Kyverno or OPA engine for one rule. Per-tenant listeners: §4's Envoy Gateway merge bug is unchanged. A per-hostname Access application minted by automation: Cloudflare write permissions exist only at account scope (§10).

Corrective scope on acceptance. docs/security/gateway.md marks hostname squatting as a known gap and derives its posture table by hand; on acceptance the gap line closes to "ADR-0005 §14 (P1-42)" and the table is derived from the declarations after P1-43. docs/conventions.md still names "the Kyverno admission policy" as the tenancy backstop; it changes to the native policy. Both edits land in the acceptance PR, not in this draft.

15. Amendment 2026-09-22 — the cf-public gateway: the public-intake class leaves the shared listener, on its own tunnel

Status: proposed. Decider: platform owner; the decisions recorded below were taken by the owner on 2026-09-22. Trigger: OME-1175 ("allow analytics ingestion without Cloudflare Access login"), which asks for a second anonymous-write hostname — the §13 gate (b) condition, word for word. analytics.dev.screamingface.ai was placed on the user tier (#958) because that gate forbade the alternative; the tenant's own service contract (ScreamingFace docs/spec/2026-09-09-OME-1152-analytics-service-spec.md) adds "no embedded shared SDK secret, CAPTCHA or login is added to this anonymous endpoint" and "ingress also needs deployment-owned global abuse protection before public launch" — so the §13 rejected alternatives stay rejected and the split is the only answer. Grounded in §13's gate text, the 2026-08-28 assessment (Option B; Phase 1 items 7–9), the docs/platform-delivery-plan.md §2 runbook stub (superseded here where they differ), and a read-only review of the draft against both clusters and the Cloudflare and Envoy documentation on 2026-09-22. Every object shape below passed kubectl apply --dry-run=server against aks-dev (Envoy Gateway v1.5.0) the same day. Acceptance conditions: the five pull requests of the split have merged on dev, the two dev hostnames answer through the new tunnel, the proofs in the working plan pass live, the corrective scope has landed, and docs/security/gateway.md carries the re-attested rows.

What changes. A second platform-owned Gateway, cf-public in pulse-edge, with its own EnvoyProxy fleet cf-public-proxy, serves every public-intake declaration, and a second, tenant-owned Cloudflare Tunnel per environment carries the public hostnames to it. The route chart maps the tier to the gateway (public-intake → cf-public; every other tier → cf-protected); a declaration still names no gateway. cf-protected keeps §2's contract as written, with the §11 leaderboard-reads exception as its only anonymous route. (Amended by §16: a hostname on the §16 list also maps to cf-public, and the leaderboard read route leaves cf-protected.) The §13 member list grows to reports.dev.screamingface.ai, analytics.dev.screamingface.ai, and — once the staging slice lands — reports.staging.screamingface.ai and analytics.staging.screamingface.ai.

The five-part contract for cf-public (a tier, not an interim):

  • Exposure — tunnel-only, ClusterIP, no public IP. Each environment's public hostnames ride their own tenant-owned tunnel (sf-public-<env>, created in ScreamingFace's Cloudflare account: a cfargotunnel.com CNAME proxies only same-account records), whose connectors run as Deployment cloudflared-public in sf-edge beside the existing cloudflared. Every public tunnel route names the stable Service cf-public-internal.envoy-gateway-system.svc.cluster.local:80 (80 → 10080, the cf-protected-internal alias pattern). The public connector's NetworkPolicy reaches only the cf-public fleet; the existing connector's reaches only cf-protected. §7's observed shape becomes "two tenant-owned tunnels into two platform tiers". The token record is sf-public-tunnel-token-<env> in that environment's vault, written by the owner's break-glass window before the connector merges, and listed in docs/runbooks/cloudflare-tunnel-token-rotation.md.
  • Auth — none at the edge, by definition. No Access application covers a cf-public hostname (attested per hostname, never a Bypass policy); no SecurityPolicy renders for the tier. (Amended by §16(c): on a §16 hostname a user-tier JWT SecurityPolicy renders on cf-public, and a path-scoped Access application covers /v1/scores only: the write path and its score-detail reads.) The application owns any per-request gate (Turnstile for report-intake; none for analytics, by contract). Identity stays gateway-owned: a ClientTrafficPolicy on the cf-public Gateway carries the same earlyRequestHeaders strip as cf-protected-identity-strip, the route stamps X-Tenant with set, and a public-intake declaration strips CF-Connecting-IP and X-User-Email toward the backend.
  • Traffic — the tier can limit per caller, and the owner decides when it does. The same ClientTrafficPolicy sets clientIPDetection.customHeader {name: CF-Connecting-IP, failClosed: true}: Envoy's trusted client address becomes Cloudflare's, and a request that carries no such header is refused with 403. §13 gate (c) stands: cf-protected never gets this. rateBudget is rendered for this tier only — {requestsPerMinute: N, key: source-ip, routePerMinute: M} becomes one Global BackendTrafficPolicy per route on the live Redis-backed limiter: two sourceCIDR {type: Distinct} rules (0.0.0.0/0, ::/0) at N per minute and one unselected rule at M per minute, the route ceiling; whole-route policies only (Envoy Gateway 1.5 rejects sectionName at admission), and never while a route is attached to both gateways (on cf-protected the trusted address is the connector pod, so a per-IP rule there would key every caller onto two buckets). Owner decision 2026-09-22: no numbers on dev or staging — the users are known — so no policy renders today; the numbers are a production-ladder item. The limiter runs fail-open with one replica today (no rateLimit.failClosed); the ladder closes that. IPv6 keys are per address (/128), so the route ceiling is the IPv6 flood control; Cloudflare's Pseudo IPv4 is evaluated with the Bot Fight Mode decision. Every other tier keeps rateBudget recorded, not rendered. (Amended by §16(d): rateBudget renders on any route that resolves to cf-public; a single-homed anonymous-read route requires requestsPerMinute and routePerMinute on every environment, which supersedes "no numbers on dev or staging" for that class only.)
  • Availability — its own tier-0 fleet: 2 replicas, envoyPDB minAvailable: 1, container requests and a memory limit (no CPU limit — CFS throttling under a flood), hostname topology spread with whenUnsatisfiable: ScheduleAnyway (each cluster has one untainted user node today, so the spread takes effect when a second exists). No tracing stanza: the staging overlay removes tracing by the cf-protected-proxy name only, and anonymous traffic is not traced by default. The fleet lives in pulse-edge/base, so it renders wherever the tier is enabled: aks-dev and aks-staging today; aks-prod does not enable pulse-edge, and the pull request that creates overlays/prod (STP-P1-45) is a production change under the ask-first rule and renders this fleet with it. A flood on cf-public degrades cf-public and its own tunnel; the tenant's authenticated product keeps its tunnel and its fleet.
  • Admission — default-closed at three layers. (1) The Gateway's allowedRoutes.namespaces.from: Selector admits only namespaces labelled openmined.org/public-routes: "true", set only in the platform-owned namespace registry (kubernetes/tenants/<tenant>/namespaces.yaml, today sf-report-intake and sf-analytics); both tenant fences blacklist cluster-scoped kinds, so a tenant cannot self-label, and a forged route in any other namespace cannot attach. (2) The validator's check_tier_rules: a public-intake hostname is in the member list above, its namespace carries the label, its environment is dev or (after the staging slice) staging — prod is denied with a message that names the production ladder below — no rule publishes /, and the allowlist line carries class public-intake-ADR-0005-s13; the "at most one declaration" rule is removed, because gate (b) is satisfied by the split. check_render asserts the parentRefs the tier implies and a BackendTrafficPolicy if and only if a budget is set and the route is single-homed. (3) The k8s-validate.sh anonymous-route tripwire treats a cf-public parent exactly like a policy-less cf-protected one. The tenant-httproute-hostnames admission policy applies unchanged. The fleet's own default-deny NetworkPolicy (cf-public-fleet in envoy-gateway-system: ingress 10080 from the public connector only; egress DNS, the controller's xDS ports, the limiter, and the two backend ports) is the only guard against an in-cluster caller forging CF-Connecting-IP at the stable Service — the custom-header extension trusts its upstream by design. (Amended by §16(e): egress adds sf-scoreboard 9106 and TCP 443 to Cloudflare's published IPv4 ranges for the Access JWKS; check_tier_rules admits anonymous-read and user on a §16 hostname.)
  • Monitoring (class obligation) — a Gatus posture check per public hostname with ignore-redirect (a 302 to cloudflareaccess.com means an Access application crept back; the expected answer is the application's own refusal code); the identity companions unchanged; the report-intake 503-rate alert unchanged; analytics gains its delivery-503 alert and a 429-rate signal. Recorded limit: Gatus cannot tell Envoy's fail-closed 403 (the header missing — the zone's "Remove visitor IP headers" transform switched on) from the application's 403 on reports; the transform is therefore an attested fact per hostname, re-attested with every tunnel change. Staging rows are board-only; paging is a ladder item.

Access log (owner decision 2026-09-22). The cf-public fleet logs the cf-protected field set minus x-forwarded-for and downstream_remote_address: the callers are anonymous reporters and SDK users, the analytics contract promises no IP reaches analytics data, and cf-ray keeps every request attributable in the tenant's Cloudflare dashboard. Exception: downstream_remote_address stays on from the fleet's first merge until the detection and spoof proofs have run (it is the only observable for "Envoy saw the real address, not the client-supplied header"), then leaves in the analytics change. Incident mode is a one-line change re-adding the field (Envoy Gateway pushes access-log configuration over xDS without a restart) with a stated removal condition.

Transition — no dark window for reports.dev. A hostname's DNS record names one tunnel, so the move is a switch, not a fade; the chart's platform constant publicIntakeDualHome makes both paths valid while the switch happens. Order: (1) the fleet lands inert in pulse-edge/base (nothing carries the label); (2) the dev public tunnel exists and its token is in the dev vault; the public connector, the quota fix (explicit CPU limits on both connectors — the namespace ceiling is fully consumed today), the chart mapping with dual-home ON, the labels, the validator and tripwire changes and the report-intake ingress rule land; reports-public is attached to both gateways; (3) the hostname moves: a throwaway canary route on the new tunnel proves the tunnel-to-fleet path (Envoy 404), the reports.dev ingress rule is added to the new tunnel's configuration through the API (the dashboard refuses a hostname whose record exists), the existing proxied CNAME's target is edited from the old tunnel's cfargotunnel.com name to the new one (atomic at the edge), the rule is removed from the old tunnel's configuration through the API — never a dashboard route delete, whose effect on the DNS record is undocumented — and the hostname is re-attested; (4) dual-home goes OFF and analytics flips to public-intake in the same change (identity none, methods: [POST], the strips, the allowlist line); its hostname moves the same way, and the owner removes it from the shared dev Access application last; (5) records; (6) the staging slice. Analytics is not dual-homed: no SDK caller exists yet on the tenant's main.

Staging (owner decision 2026-09-22: admitted). A public-intake declaration with env: staging is valid once, on aks-staging: the fleet renders from base (it does on merge), the staging public tunnel exists with its token in the staging vault and its connector in sf-edge, the edge AppProject lists the two namespaces on aks-staging, the report-intake staging workload ticket exists (the workload before the route), the two hostnames are in the member list, and each carries its attestation set (no Access application, no pre-existing DNS record, Turnstile staging widget with real keys for reports, Bot Fight Mode status, plan, transform OFF). The deny fixture public-intake-staging becomes public-intake-prod. Taken 2026-10-23 for analytics.staging.screamingface.ai only (owner go): the member list gains that one hostname, the chart guard and the validator admit env: staging and deny env: prod, and the edge AppProject lists sf-analytics on aks-staging. The staging route carries POST /v1/events, with no budget; the Colab consent bridge (GET and POST under /bridge) joins it the same day by owner decision, before its browser acceptance. reports.staging.screamingface.ai stays out of the member list until its own change. The owner's attestations are recorded in docs/tenants/screamingface/apps/analytics.md (staging table). Production taken 2026-10-23 for analytics.screamingface.ai only, by owner decision, before the rest of the ladder: the member list becomes per environment, the validator and the chart guard admit that one hostname on env: prod, and the route carries the staging shape and limits (120 a minute per address, 1200 for the route, the Colab bridge on). Ladder items still open, to follow after go-live: (7) paging and (10) production log export (the staging log alerts exist; production follows), (8) the staging shut-off drill, (9) the strip and spoof proofs on the production proxy, and (11) the tenant's SDK back-off check. Items (3) and (4) are the owner's baseline and acceptance of the same day.

Production ladder (strict upwards). env: prod stays denied by the chart guard and the validator until a later amendment records evidence for each item (amended by §16(g): for the anonymous-read class, §16 is that amendment; public-intake on prod stays denied): (1) aks-prod registered and pulse-edge/overlays/prod rendering this base; the production vault admits sf-edge. (2) A separate production public tunnel in the tenant's account, its token in the production vault, at least two connectors on at least two nodes with a PDB, the sf-edge quota sized for four pods. (3) Budgets set from at least two weeks of staging 2xx/429 data, with the IPv6 posture recorded. (4) rateLimit.failClosed: true (controller-wide; it also governs preview policies) and the limiter and its Redis at two replicas with PDBs, or an explicit acceptance. (5) The production zone's plan, WAF and Bot Fight Mode posture decided; a Turnstile production widget with real keys. (6) The full attestation set per production hostname, re-attested on every tunnel change. (7) Paging: a Gatus alerting provider with alerts on the production rows, the 503-rate and 429-rate alerts wired to a pager. (8) A kill-switch drill on staging with measured time-to-dark for the DNS switch, the tunnel-rule removal and the connector scaled to zero through git (kubectl scale is reverted by self-heal), and a runbook. (9) A cf-public identity-strip sentinel and the recorded CF-Connecting-IP spoof proof. (10) A retention bound for the access log and the incident-mode toggle documented; production log export verified. (11) Production declarations never auto-merge (§14), and the SDK's 429 back-off verified by the tenant.

What this satisfies, and what it does not. §13 gate (b): satisfied — the class is off the shared listener, so a second member is admitted. Gate (c): unchanged and still forbidden on cf-protected. Gate (a): the "public-intake still on cf-protected" blocker is replaced, not removed — staging exposure requires the staging fleet, tunnel and attestations above; production exposure requires the ladder — and CLAUDE.md's gate paragraph is rewritten on acceptance to say exactly that, in the W24/W25 blocker class. §11 is untouched. §14's tier table changes in the public-intake row only: environments dev, staging (§15); prod denied by validator, gateway cf-public, budget optional until the ladder, count limit gone.

Rejected alternatives, for OME-1175 specifically. A service token shipped in the SDK (a public credential; §13; forbidden by the tenant contract). An Access Bypass (§13). Amending gate (b) to admit two hostnames on the shared listener (keeps the shared blast radius and no per-caller control — the pressure the gate exists to refuse). One public tunnel shared by dev and staging (Cloudflare balances a tunnel across all its connectors, so traffic would reach a cluster without the route). Moving the leaderboard hostname now (D4: its reads are method-split from Access-gated writes and one hostname has one tunnel; the whole-hostname move is the named follow-up, after which cf-protected carries no anonymous route). (Amended by §16: the follow-up is taken 2026-10-05.) A Cloudflare Worker intake (held as the fallback, unchanged). Tunnel direct to the app (§13).

  • Bot Fight Mode and Pseudo IPv4: decided at the first SDK acceptance test, never mid-incident (§13 risk 3, unchanged).

Corrective scope on acceptance. CLAUDE.md's gate paragraph (as above); docs/security/gateway.md (the analytics rows; the reports.staging row rewritten from "must not exist"; the known-gaps rate-limit line says "rendered on cf-public only when a budget is set, parked elsewhere"); docs/runbooks/declare-a-route.md (tier table; "what a declaration does not change" gains the per-tier tunnel and origin); docs/runbooks/cloudflare-tunnel-token-rotation.md (two record rows) and docs/runbooks/secret-rotation.md (the public tunnel token); kubernetes/apps/tenant-edge/README.md; the report-intake app doc's open item; a new docs/tenants/screamingface/apps/analytics.md with its attestation set; docs/architecture/platform.md (two tier gateways, two tenant tunnels). Known proof gaps, recorded: the identity-strip sentinel (pt-echo) covers cf-protected only; Gatus cannot distinguish the two 403s on reports.

16. Amendment 2026-10-05 — whole-public hostnames: anonymous reads on cf-public, Access-JWT writes

Status: accepted for implementation 2026-10-05 by owner decision (option 3, all three environments). Decider: platform owner. The production conditions in (g) are signed by the owner before the production Access edit. §14 and §15 keep their own status lines; the statements this section changes carry an "amended by §16" note where they stand. Trigger: the owner's goal of an anonymous ScreamingFace leaderboard on dev, staging and production, with writes still behind Cloudflare Access. Grounded in a read-only review of origin/main at 92679702, the three clusters, the tenant's scoreboard source at the staging and production pin 68cad9d9 and the dev pin 8c4db51e, and the tenant's SDK at e9e850e8 on 2026-10-05, with a red-team pass over the plan the same day.

Where it stands today. On dev the leaderboard reads are anonymous, but on cf-protected: that is the interim shape §13 warns about, and the §11 read exception is the only anonymous route on that tier. On staging and production one user declaration per hostname carries every method, and Cloudflare Access covers the whole hostname. The cf-public fleet renders on all three clusters but carries traffic on dev only.

(a) The anonymous-read class leaves cf-protected. A leaderboard hostname moves WHOLE to cf-public, reads and writes together, because one DNS record names one tunnel. An anonymous route renders on cf-protected ONLY as the second parent of a hostname on the dual-home list in (b), while its tunnel move is in progress — never once the hostname is single-homed. After each move, cf-protected carries no anonymous route for that hostname, and after the last move it carries none at all; §2's "Access JWT auth" line is again the whole of that tier's auth contract. The chart and the validator refuse anonymous-read on an unlisted hostname, so an anonymous route on cf-protected (outside the dual-home window) cannot render. Accepted for the dual-home window only: the cf-protected fleet has no NetworkPolicy, so an in-cluster pod can read the staging or production leaderboard without Access and without a budget while the hostname is dual-homed. The identity strip still applies, so nothing private leaks; the window runs from the split merge to the single-home merge and is kept as short as the owner's switch steps allow.

(b) The member list is two platform-owned chart constants. No new tier, CRD, controller or Cloudflare product. Two template defines in kubernetes/apps/tenant-edge/templates/_helpers.tpl, each on one line so the validator reads them by regex (the publicIntakeDualHome pattern), each a comma list that no declaration can set:

  • tenant-edge.publicHostnames — the whole-public hostnames.
  • tenant-edge.publicHostnamesDualHome — the listed hostnames whose tunnel move is in progress. Every entry is also in publicHostnames; the chart and the validator fail a dual-home entry that is not.

The members are leaderboard.dev.screamingface.ai, leaderboard.staging.screamingface.ai and leaderboard.screamingface.ai. Each joins the list in its own environment's change, after that environment's prerequisites in (e) and (i) are live. On a listed hostname the tiers map as follows: anonymous-read and user render on cf-public; privileged never renders (the chart and the validator fail it). Every declaration on one hostname resolves to the same parentRefs: one hostname, one tunnel. The mechanism change extends the validator rule check_hostname_unique to assert this; today that rule checks only one tenant, environment and namespace per hostname, disjoint methods and the hand-written allowlist. A route's namespace must carry openmined.org/public-routes: "true", set in the platform-owned namespace registry (§15 Admission), so sf-scoreboard joins sf-report-intake and sf-analytics. The allowlist class for the read route is renamed from anonymous-read-exception-ADR-0005-s11 to anonymous-read-ADR-0005-s16.

Each listed hostname keeps the dev method split as two declarations:

  • leaderboard — tier anonymous-read, identity none, methods GET HEAD OPTIONS, no SecurityPolicy, the budget in (d).
  • leaderboard-writes — tier user, identity access-jwt, methods POST PUT PATCH DELETE CONNECT TRACE, the chart's unchanged JWT SecurityPolicy with the hostname's existing AUD (dev: the shared §11 AUD; staging and production: their own per-hostname AUD, §11 scope guard).

Together the two cover all nine Gateway API methods. Read scope stays PathPrefix / (dev parity): /healthz, /docs, /redoc, /openapi.json and, where the pin serves it, /readyz (a database query) become public under the budget. The tenant confirms that data surface before any staging or production Access edit ((g)).

(c) Auth — amends §15 Auth. On a listed hostname, a user-tier JWT SecurityPolicy may render on cf-public. The policy targets the route by name, so it applies on whichever gateway the route attaches to; issuer, JWKS URI, header extraction (Cf-Access-Jwt-Assertion) and the claim-to-header map stay template invariants (§5, §14). Cloudflare Access covers only the write path /v1/scores — the only write endpoint at both tenant pins:

  • The Access application is path-scoped: destination <hostname>/v1/scores. On staging and production the owner edits the existing application IN PLACE (destination <hostname> becomes <hostname>/v1/scores), so the AUD is unchanged. Never delete and recreate it: a new application issues a new AUD, and then every write answers 401. Never add a Bypass policy (§13). The policy and its Include list stay as they are (superseded 2026-10-22 by (j): Include Everyone). Attested per hostname, re-attested on every tunnel or Access change. Dev already has this shape on the shared dev application.
  • Access matches host and path, not method. So GET /v1/scores/{id} (the portal's score detail) stays login-gated, as on dev.
  • The identity boundary is the edge, not the app. The scoreboard app trusts X-User-Email from any peer in the pod CIDR and verifies no token itself (tenant apps/scoreboard/src/scoreboard/core/auth/cloudflare_identity.py, module docstring: "This service therefore does no token work of its own"). The comment in templates/securitypolicy.yaml that says the application verifies the Access JWT itself is wrong for this app and is corrected with the mechanism. The boundary is therefore two things: the cf-public identity strip (cf-public-client-ip, earlyRequestHeaders.remove of X-User-Email, X-User-Id, X-Service-Id), and the sf-scoreboard ingress selector, whose cf-public and cf-protected peers each name both gateway.envoyproxy.io/owning-gateway-name and gateway.envoyproxy.io/owning-gateway-namespace: pulse-edge. No static check of the strip exists today (k8s-validate.sh checks only that the pt-echo sentinel has no claimToHeaders). The mechanism change adds one to k8s-validate.sh: cf-protected-identity-strip and cf-public-client-ip each target their own Gateway and remove X-User-Email, X-User-Id and X-Service-Id; no overlay may patch either.
  • Any future tenant write endpoint must sit under an Access-covered path before it ships. Reason: Cloudflare turns the SDK's Cf-Access-Token into Cf-Access-Jwt-Assertion only on a path an Access application covers; on any other path a write meets the Envoy JWT check with no token and answers 401.
  • The SDK learns Access per request, so a path-scoped application still works for SDK submissions. sync_auth_flow reads the Access challenge from the response, logs in, and re-sends with Cf-Access-Token (packages/screamingface/src/screamingface/_access/auth.py:289-342); client.login() logs in to the engine origin only (client.py:141-146); after a login challenge the first request raises access_reauthenticated unless the caller marked it replay-safe (_require_replay_safe, auth.py:566-585). The tenant's real SDK submission after each Access edit is the end-to-end proof. Corrected 2026-10-22 ((j)): this bullet was wrong for ScreamingFace clients before 0.1.6, and the proof it names was not run.
  • Logged-in readers lose their private-board rows on the read path on staging and production (dev parity): the read route has no SecurityPolicy, so no verified identity reaches the app on a read. The owner tells the tenant. Optional identity on the read route (jwt.optional) is a later, separate ADR item.

(d) Traffic — amends §15 Traffic. rateBudget may render on any route that resolves to cf-public, not only public-intake. Key source-ip is valid if and only if the route resolves to cf-public (on cf-protected the trusted address is the connector pod). A single-homed anonymous-read route REQUIRES {key: source-ip, requestsPerMinute: N, routePerMinute: M}, on every environment including dev; the chart guard and the validator fail it otherwise. routePerMinute is mandatory, not optional, because IPv6 buckets are per /128: anyone with an IPv6 range otherwise avoids the per-address budget, and the leaderboard hostnames answer over IPv6. The route ceiling is the IPv6 flood control. A budget never renders while the hostname is dual-homed (§15). Starting numbers on all three environments: 600 requests per minute per client address, 6000 per minute for the route, revised from 2xx/429 data. Sizing: one portal index view is /v1/benchmarks plus one /v1/leaderboard/{id}?top=200 per board (65 boards on dev), about 66 API calls plus static assets, and each board call runs a whole-board pass in the app, which has no rate limit of its own. This supersedes the 2026-09-22 "no numbers on dev or staging" decision for this route class only; public-intake is unchanged. Accepted, revisit: the limiter runs fail-open with one replica (no rateLimit.failClosed), as §15 recorded.

(e) Admission and the fleet contract — amends §15 Admission. The cf-public-fleet NetworkPolicy's egress gains exactly two rules:

  • sf-scoreboard TCP 9106, the leaderboard backend. It may land early, with the inert preparation: no route reaches it until a hostname is listed.
  • TCP 443 to Cloudflare's 15 published IPv4 ranges (https://www.cloudflare.com/ips-v4), for the Access JWKS. This rule lands in the change that first puts a JWT route on cf-public, not before. The Envoy proxy, not the controller, fetches the JWKS (https://openmined.cloudflareaccess.com/cdn-cgi/access/certs); without this rule every write fails closed with 401. An FQDN policy is not available: Cilium enable-l7-proxy is false on aks-dev, aks-staging and aks-prod (read 2026-10-05), and AKS ACNS FQDN filtering is new spend. The Access host's own addresses (today inside 104.16.0.0/13) can change, so /32s are not used. Precedent for the shape only (an ipBlock egress to Cloudflare on 443): kubernetes/apps/firecall-edge/overlays/dev/activation/cloudflare-egress.yaml. That file pins exact tunnel-edge /32 and /128 addresses, the address choice this rule rejects. Accepted: the ranges front much of the Cloudflare-proxied internet, so a compromised public proxy gains that outbound path; never 0.0.0.0/0 without an owner decision.

The sf-scoreboard ingress gains one cf-public peer as in (c). Nothing else may reach port 9106 except the Gatus pod and, until the last move is done, the cf-protected fleet. The fleet's ingress stays the public connector only on 10080 (§15): that remains the only guard against an in-cluster caller forging CF-Connecting-IP.

(f) Transition, per hostname, and the kill switch. Prerequisites: the namespace label, the two egress rules and the ingress peer live, or landing in the same change as step 1 on the first environment (the 443 rule, (e)); the environment's public tunnel, record and connector live ((i)); and, before the staging move, a checked-in runbook (under docs/runbooks/) for moving a hostname between tunnels and its revert. Then, in this order:

  1. The split (staging and production) and the hostname on both lists merge while Access still covers the whole hostname. Both routes attach to cf-public and cf-protected.
  2. The owner adds the hostname's ingress rule on sf-public-<env> through the API (the dashboard refuses a hostname whose record exists), with the origin cf-public-internal.envoy-gateway-system.svc.cluster.local:80.
  3. The platform confirms that both cloudflared-public pods logged the new configuration with the hostname. Adding the rule before the CNAME switch avoids the 2026-09-23 404.
  4. The owner edits the proxied CNAME's target to the new tunnel's cfargotunnel.com name.
  5. The owner removes the rule from the old tunnel through the API — never a dashboard route delete. Afterwards the hostname appears in exactly one connector's latest configuration.
  6. The hostname leaves the dual-home list: single-homed, the budget renders.
  7. On the environment's cf-public pods: the strip proof and the CF-Connecting-IP spoof proof are recorded, and a burst test shows 429 above the per-address number and 200 after the window. The burst test runs on dev and then on staging before the staging Access edit. The tenant has confirmed the data surface, and on production the owner has signed (g).
  8. The Access edit, LAST, in place ((c)).

On dev the shared application is already path-scoped and reads are already public, so step 1's Access condition and step 8 do not apply. Step 7's strip, spoof and burst proofs are still recorded on dev after step 6, and the staging move does not start without them. The tenant confirmation gates only the staging and production Access edits.

Rollback before step 5: the CNAME goes back. Between step 5 and step 6: the rule is re-added on the old tunnel through the API, then the CNAME goes back. After step 6: revert the single-home change first. Kill switch for exposure: restore the whole-host Access destination. It takes seconds, needs no deploy, and leaves writes and the tunnel untouched.

(g) Production conditions (signed by the owner before the production Access edit). env: prod for anonymous-read is admitted by this section for the listed production hostname; public-intake on prod stays denied (§15). The §15 production ladder, item by item, for this class. The disposition column is the target the owner signs. "Met" means the owner signs against recorded evidence; on 2026-10-05 only item (1) and the declaration half of item (11) have that evidence, and every other "met" or "required" item gets its evidence before the production Access edit. "Required" means the item is a condition in its own right. "Accepted in writing" means the owner's signature accepts the gap:

§15 ladder item Disposition at signing, for anonymous-read Evidence or reason
(1) aks-prod registered, pulse-edge/overlays/prod renders the base, the production vault admits sf-edge met P1-25, P1-45: the production edge and its tenant connector run
(2) separate production public tunnel and record, ≥2 connectors on ≥2 nodes, PDB, sf-edge quota for four pods met, except two nodes: accepted in writing sf-public-prod, cloudflared-public 2/2 with its PDB; one user node per cluster (a second is new spend), the same failure domain as cf-protected today
(3) budgets from two weeks of staging 2xx/429 data, IPv6 posture recorded accepted in writing the starting numbers in (d), revised from data; IPv6 posture recorded in (d) (routePerMinute mandatory)
(4) rateLimit.failClosed and two limiter and Redis replicas with PDBs, or explicit acceptance accepted in writing fail-open with one replica (d); the alternative is controller-wide
(5) zone plan, WAF and Bot Fight Mode decided; production Turnstile with real keys met for plan and BFM (attested); Turnstile not applicable the read route has no app-side gate
(6) full attestation set per production hostname met path-scoped Access application with its AUD unchanged; no whole-host application; no Bypass; "Remove visitor IP headers" OFF; BFM status; plan
(7) paging provider and alerts accepted in writing no paging provider; rows are board-only
(8) kill-switch drill on staging with measured time-to-dark, and a runbook met by the Access-revert drill for this class the exposure switch is the Access destination (f), not the DNS, tunnel rule or connector (those take reads and writes dark together); the staging drill times the revert; the checked-in tunnel-move runbook is a prerequisite of the staging move (f)
(9) cf-public identity-strip sentinel and the CF-Connecting-IP spoof proof strip proof required; spoof proof required; standing sentinel accepted in writing the static k8s-validate.sh assertion that the mechanism change adds ((c)), plus a strip proof recorded on the staging and production cf-public pods before each Access edit; the spoof proof is recorded on the same pods before each Access edit, because the source-ip budget keys on CF-Connecting-IP and the proof shows a caller cannot choose its own bucket (no spoof proof for cf-public is in the ledger today); a standing pt-echo route on cf-public is extra surface
(10) access-log retention bound, incident-mode toggle, production log export accepted in writing the fleet log omits client-IP fields (§15); cf-ray keeps requests attributable; production export is open under WS6
(11) production declarations never auto-merge; SDK 429 back-off verified met for declarations; back-off accepted in writing until the tenant verifies it production declarations change only by a reviewed pull request; the numbers in (d) sit well above one portal view

Plus, required for this class:

  • Staging complete, with a timed Access-revert drill recorded.
  • The production public connector 2/2 with its PDB.
  • The strip proof and the CF-Connecting-IP spoof proof on the staging and production cf-public pods, each recorded before that environment's Access edit.
  • A recorded burst test (429 above the per-address number, 200 after the window) on dev and then on staging, before the staging Access edit.
  • The checked-in tunnel-move runbook, merged before the staging move.
  • The tenant's written confirmation of the public data surface: submitter email local parts; full author emails where local parts collide; /docs, /redoc, /openapi.json; the private board's metadata (its listing is by design, its rows are not).
  • The budget live on the single-homed production route before the edit.
  • IPv6 per-address bypass covered by routePerMinute.

(h) Monitoring per environment (class obligation). Four Gatus rows per listed hostname, ignore-redirect where a 302 is the answer:

  • Public read: GET /v1/benchmarks answers 200 (a 302 means an Access application crept back over the read path).
  • Write refusal: POST /v1/scores answers 302 to Access (any other answer, Envoy's 401 included, means the path-scoped application is gone).
  • Write JWT: POST /v1/benchmarks answers 401 Jwt is missing (Envoy's check on a path Access does not cover).
  • JWKS reachability: POST /v1/benchmarks with a crafted RS256 token sent in Cf-Access-Jwt-Assertion (the extraction header in (c)) — JWT header {alg: RS256, kid: probe}, issuer https://openmined.cloudflareaccess.com, the hostname's full 64-character AUD (a short one answers 403 Audiences in Jwt are not allowed), a future exp, any base64url signature — answers 401 Jwks doesn't have key to match kid or alg from Jwt. The answer Jwks remote fetch is failed means the 443 egress in (e) broke. No other row sees that failure: the no-token row is answered before any key lookup.

The existing staging and production rows that expect a refusal on reads change in the same window as that environment's Access edit. Outside-in probes are the default proof. A kubectl port-forward or a crafted probe into the staging or production clusters needs the owner's per-action go-ahead.

(i) Records. One tunnel per environment, sf-public-<env> (sf-public-dev, sf-public-staging, sf-public-prod), in the tenant's Cloudflare account. One record per environment, sf-public-tunnel-token-<env>, in that environment's vault (kv-pulse-dev-eastus, kv-pulse-staging-eastus, kv-openmined-prod-eastus), written only with scripts/ops/write-vault-record.sh --shape tunnel-token (break-glass window, revoke, zero standing access), before that environment's connector merges. Never reuse a token across environments: Cloudflare balances a tunnel across all its connectors, so traffic would reach a cluster without the route.

(j) Amendment 2026-10-22 — open submission, a write budget, and a correction (OME-1286, OME-1649). Decided by the owner on 2026-10-22.

  • Open submission. On every environment, the Access application on a listed hostname's write path has one policy: Allow, Include Everyone. Any person who can log in may submit. The login stays: a submission is never anonymous, and the row's submitter is the verified email that Envoy stamps. This supersedes "the policy and its Include list stay as they are" in (c).
  • One application per hostname stays. The tenant proposed folding /v1/scores into the engine's application (OME-1649). The owner rejected it: §11's scope guard forbids a shared AUD on staging and production, every engine user could then write scores, and Include Everyone on a shared application would open the engine too. For the same reason, dev's write path moves to its own application before its policy opens (§11 is amended when the owner creates that application).
  • Seats. Every Access login takes one seat from the openmined Zero Trust team. Every Access application draws from that one pool, staff applications included. On the free plan the pool is 50 seats, and when it is full Cloudflare refuses the next login of anyone. The plan, seat expiry and the seat count are the owner's (new spend). Owner decision 2026-10-29: open admission is capped at 50 people a month; seats are not reserved in advance (36 in use that day); seat expiry is one month. The seat watch (.github/workflows/cloudflare-seats.yml, runbook docs/runbooks/cloudflare-seats.md) counts the seats in use every six hours and warns in Slack from 80% of the plan's seats (once a day) and from 90% (every run), so seats are bought when they are needed.
  • Write budget — amends (d). The write route on a listed hostname carries {key: source-ip, requestsPerMinute: 30, routePerMinute: 300} on every environment, revised from 2xx/429 data. It renders as BackendTrafficPolicy <route>-public-budget.
  • Correction to (c). The bullet "a path-scoped application still works for SDK submissions" was wrong for ScreamingFace clients before 0.1.6. Those clients ended the Access login on the host root /. After the 2026-10-07 edits / is public, so Cloudflare never transferred the application token, and a submission to staging or production waited out the 300-second login (OME-1649). Dev worked only because leaderboard.dev shared the engine's application. Client 0.1.6 (tenant #1486) ends the login on the challenged path. The "real SDK submission after each Access edit" proof was not run in October.
  • Required proof after every Access edit on a listed hostname. One real submission with the current released client (0.1.6 or later) by a person on no allow list. The row shows that person's email; an anonymous POST /v1/scores still answers 302; and the environment's engine hostname still refuses that person. The result goes in the hostname's attestation table. A reading of the client's challenge handling is not this proof.

Rejected alternatives. Anonymous reads on cf-protected (option 2): the §13 interim shape made permanent, with no per-caller budget, because §13 gate (c) forbids clientIPDetection there. Writes on a separate hostname that stays on cf-protected: keeps §15's "no SecurityPolicy on cf-public" but needs a tenant SDK and portal change and breaks older SDKs. A new tier for JWT writes on cf-public: more ADR, validator and documentation surface than two constants. An Access Bypass on the read paths (§13).

Corrective scope, with the change that makes each record false. docs/security/gateway.md posture rows and its rate-limit gap line; docs/runbooks/declare-a-route.md; kubernetes/apps/tenant-edge/README.md and values.yaml; scripts/ci/k8s-validate.sh (the new static strip assertion in (c), and the anonymous-route tripwire comment that calls every cf-public parent "public BY DEFINITION"), with the mechanism change; docs/runbooks/cloudflare-tunnel-token-rotation.md (staging and production public rows); a runbook for moving a hostname between tunnels and its revert, before the staging move ((f)); a docs/tenants/screamingface/apps/scoreboard.md attestation table; docs/tenants/screamingface/index.md ("No public door on production") and the aks-prod pulse-edge.yaml "DARK" comment, with the production move. CLAUDE.md carries this section's gate in the W24/W25 blocker class (this change).

Consequences

  • Stage 3 of the ScreamingFace plan builds the cf-protected tier Gateway (platform namespace) instead of scaffolding for a tenant-owned Gateway; their WU-4/WU-5 shrink accordingly (communicated on their #413).
  • The platform inherits obligations that per-tenant gateways didn't have: route/policy templating in /onboard-repo, descriptor prefixing, telemetry partitioning, and (before GA / second tenant) the hostname admission policy.
  • ScreamingFace needs one follow-up: SecurityPolicy moves platform-side (they supply AUD + specifics), amending the earlier "stays in your repo" message.
  • Two edge stacks continue (ingress-nginx for platform apps under *.dev.openmined.org, the tier gateways for tenants) — consolidation onto Gateway API stays a later deliberate call.
  • A shared data plane means a proxy-fleet incident is platform-wide; the tier gateway is operated as tier-0 (HA replicas, PDB, kill drills per T8).