ScreamingFace on PULSE
ScreamingFace dev is online on PULSE through the permanent
fusion.dev.screamingface.ai hostname. The public route serves the
tenant-owned url4-cloud workload; the chart-deployed aigateway runs privately
behind it.
| Field | Current value |
|---|---|
| Environments | Dev on aks-dev (<app>.dev.screamingface.ai), staging on aks-staging (<app>.staging.screamingface.ai), production on aks-prod (bare names: fusion.screamingface.ai, leaderboard.screamingface.ai, aigw-admin.screamingface.ai) |
| Public URL | fusion.dev.screamingface.ai |
| Service status | Tenant · sf-fusion on the PULSE status board |
| Observability | Platform-operated SigNoz — logs are live; developer access is being set up |
| Tenant source revision | per environment, in the chart tickets under kubernetes/tenants/screamingface/charts/ (<app>.yaml dev, <app>-staging.yaml, <app>-prod.yaml); dev moves automatically, staging and production by a Kargo press |
| Infrastructure revision | main of OpenMined/infrastructure; every landed change has a ledger row (docs/ledger.md) |
| Last full functional verification | 2026-07-31, including a post-retirement run — evidence |
| Owners | ScreamingFace application team · OpenMined PULSE platform team |
| Production | Live, dark (2026-09-30; complete 2026-10-03): all six workloads run on aks-prod at staging's pins — engine, leaderboard, AI gateway and its console, report-intake, analytics; the engine and the AI gateway console behind Cloudflare Access for named operators (reports-internal.screamingface.ai follows once its tunnel row and Access application exist). The leaderboard's reads are public since 2026-10-07 (ADR-0005 §16); its submissions need an Access login. The gateway roster and console admins match dev and staging. |
Dev is online; provider completion is not yet accepted
The public token, WebSocket, Runner, JetStream, and aigateway transport paths are proven. A provider-backed completion is still blocked by the model/profile contract described under known limitations. Do not interpret a green health check as proof of provider execution.
What is running
- Cloudflare Access and a tenant tunnel protect the public hostname before traffic reaches the cluster.
- The shared platform-owned Envoy
cf-protectedgateway validates the route's identity contract, strips reserved identity headers, and routes/wsplus REST traffic to url4-cloud. - url4-cloud accepts requests, mints short-lived capability tickets, and creates ephemeral Runner Jobs.
- NATS JetStream carries the Runner lifecycle and result stream.
- The private aigateway handles model routing and identity-derived profiles.
- Platform-owned PostgreSQL stores aigateway state in the isolated
sf-aigwnamespace.
The previous trial stack was retired on 2026-07-31 after its observation and continuous-health gates passed. Its Deployments, database PVC, ExternalSecrets, and obsolete NetworkPolicies are gone. It is no longer a rollback path; the tenant-owned workloads are the only application stack serving this environment.
One request, end to end
flowchart LR
C[Developer or service client]
CF[Cloudflare Access<br/>and tenant tunnel]
EG[Envoy Gateway<br/>cf-protected tier]
U[url4-cloud]
R[Ephemeral Runner Job]
N[NATS JetStream]
A[aigateway]
DB[(PostgreSQL)]
P[Model provider]
C --> CF --> EG --> U
U --> R
U <--> N
R <--> N
R --> A
A --> DB
A -. provider completion<br/>not yet accepted .-> P
The WebSocket connection and application subscription are distinct. A client
must connect with a valid ticket and then send ai.url4.attach; merely opening
the socket does not start event delivery.
Proof at a glance
| Proof class | Current evidence | What it proves |
|---|---|---|
| Deployment provenance | Tenant chart revision, infrastructure revision, and both running image digests agree with the frozen rollout inputs | The intended immutable source and artifacts reached the cluster |
| Continuous checks | Edge enforcement, url4-cloud health, and AI gateway health run every 60 seconds | The edge still rejects unauthenticated traffic and both processes answer their health endpoints |
| Full functional acceptance | Public six-event lifecycle completed in 3.0 seconds; an attached socket stayed open for 110 seconds; fresh post-hold and post-retirement runs completed | Token, public route, WebSocket attach, Runner, JetStream, timeout, and the surviving post-retirement path worked together |
| Known boundary | Provider-backed completion remains unproven | Health and transport success must not be presented as successful model execution |
See Evidence and status for exact digests, check semantics, timestamps, and limitations.
Observability
PULSE operates SigNoz as the shared logs, traces, and metrics backend, using one store and UI under ADR-0007. ScreamingFace container logs are collected today with zero application changes.
Developer access is being set up. The current SigNoz Community Edition has no per-user data scoping, so the platform must settle the tenant-access boundary before inviting ScreamingFace developers; the link identifies the platform board but does not promise self-service access yet.
Keys and identity at a glance
No credential value belongs in source control, documentation, task trackers, or logs.
| Credential or trust category | Why it exists | Lifecycle consequence |
|---|---|---|
| aigateway database password | Authenticates aigateway to PostgreSQL | Rotate as a new generation and verify both consumers |
| aigateway JWT signing secret | Signs aigateway tokens | Coordinated rotation |
| aigateway admin and provisioning credentials | Controls administrative bootstrap and profile provisioning | Rotate or revoke after use |
| aigateway encryption master key | Encrypts stored provider credentials | Do not rotate in place without an application-supported re-encryption path |
| url4-cloud JWT signing secret | Signs short-lived capability tickets | Coordinated rotation with url4-cloud |
| GitHub App private key | Lets Argo read the private tenant chart repository | Overlapping-key cutover, functional Git verification, then revoke the old key |
| Azure federated identity | Lets tenant CI push images to ACR | Trust relationship; there is no stored Azure client secret |
| Cloudflare service identity | Authenticates non-interactive public checks | Client credential managed outside Git and documentation; rotate/revoke through the Cloudflare lifecycle |
Platform runtime credentials are delivered from Azure Key Vault through External Secrets Operator. Provider credentials belong to aigateway profiles; they are not chart values or platform deployment credentials.
How a ScreamingFace change reaches dev
- A change merges in
OpenMined/screamingface. - GitHub Actions uses Azure OIDC to build and push immutable images to the platform registry.
- Kargo discovers the accepted image, writes the matching chart commit and image tag, and opens a two-line infrastructure pull request.
- Required checks gate the pull request. Kargo merges it automatically when the checks pass.
- Argo reads the private chart with a GitHub App credential, combines it with platform values and companion resources, and reconciles the fenced namespaces.
- CI, Argo, continuous checks, and release-specific functional acceptance prove different parts of the result.
Tenant main advances routine application pins automatically under ADR-0008.
Changes to chart contracts, permissions, network boundaries, migrations, data, identity, or public routing still need platform review. If an active program governs the change, they also need an approved work package. Otherwise, use the normal Asana task and pull request flow.
Known limitations
- The deployed default Anthropic route uses a bare model identifier while aigateway requires a provider-prefixed canonical identifier.
- A connected WebSocket can be counted before the client successfully sends
ai.url4.attach; subscriber readiness should instead depend on protocol attach or auto-subscribe. - Header-derived identities still need an approved provider profile and an enforceable spend cap before provider completion can be accepted.
- Capability tickets expire after 60 seconds. An established socket can outlive its ticket, but a new run needs a freshly minted token.
- Public programmatic clients need a real User-Agent; Cloudflare bot protection
rejected the default
Python-urllibUser-Agent during acceptance. - Staging and production exist (2026-09-08 and 2026-09-22). On production the leaderboard's READS are public since 2026-10-07 (ADR-0005 §16: public tunnel, per-address budget; submissions still need Access). Owner decision 2026-10-22 (§16(j), OME-1286): on every environment, any person who can log in may submit, under a per-address write budget; the scoreboard attestation tables record each environment's Access edit. Every other production hostname is behind Cloudflare Access for named operators.
Go deeper
- Architecture — runtime flow, namespaces, ownership, trust, and state boundaries.
- Delivery: repository to cluster — the image and chart transports, immutable pin, and review flow.
- Evidence and status — how to interpret Argo, digests, Gatus, and functional proof.
- Credentials and identity — credential categories, ownership, rotation consequences, and identity layers.
- url4-cloud — execution protocol, Runner, JetStream, ports, and failure signatures.
- aigateway — identity, profiles, provider dispatch, database, and migration contracts.
- scoreboard — the leaderboard hostnames, anonymous
reads on
cf-public, the identity boundary, and the attestation record. - Deployment decisions — the practical consequences of the governing ADRs and the decisions still open.
- Making a change — the developer workflow, evidence gates, wider-review triggers, and rollback model.
- Tenant catalog — the PULSE tenancy model and ownership split.
- Gateway identity and traffic flow — the shared edge identity contract.
- ADR-0006 — why the tenant chart owns workloads while the platform owns the trust boundary.
The staging and production program owns the environment-specific readiness contract. Staging and production run every app; production is behind Access except the leaderboard's public reads, with the first rehearsed promotion (dev → staging → production by named people) still to come.