ScreamingFace pull-request Preview environments
Owner: Platform owner Updated: 2026-08-24 Architecture decision: ADR-0014
What a Preview gives you
A Preview is one dependency-complete ScreamingFace runtime for one pull request. It uses selected pull-request images and reviewed dependency images. It disappears when the Preview ends.
Each active Preview provides the services required by its selected components:
- Engine at
fusion-pr-<number>.preview.dev.screamingface.ai, when selected; - AI Gateway API at
aigw-pr-<number>.preview.dev.screamingface.ai, when required; - AI Gateway console at
console-pr-<number>.preview.dev.screamingface.ai, when selected; - Scoreboard at
leaderboard-pr-<number>.preview.dev.screamingface.ai, when selected; - NATS and Garage object storage, when Engine is selected;
- a PostgreSQL database for each selected component that needs one;
- a Kubernetes namespace named
sf-preview-pr-<number>; - a one-hour Kubernetes credential for the pull-request author;
- filtered SigNoz links for logs and traces; and
- automatic deletion after close, merge, label removal, or expiry.
Preview runs unreviewed pull-request code. The platform copies no secret from dev, staging, or production. Preview data is disposable.
Start a Preview
Prerequisites
Install these local tools:
ghfor GitHub;cloudflaredfor Cloudflare Access;kubectlfor Kubernetes; andjqfor the optional provider-profile command.
Authenticate gh with the GitHub account that owns the pull request.
The pull request must come from the ScreamingFace repository, not a fork.
Warning: the preview label authorizes unreviewed code on the dev cluster.
Maintainers must never add this label to a fork. Tenant automation removes
Preview labels from forks, but a short reconciliation race can still exist.
Open or update the pull request
A change under a deployable ScreamingFace path starts the Preview workflow. The workflow selects a runtime profile and the images changed by that pull request.
The pull request uses these labels:
| Label | Meaning |
|---|---|
preview-building |
The required pull-request images are not ready yet. |
preview-queued |
Images are ready, but three Preview slots are active. |
preview |
Argo can create or update the Preview. |
preview-expired |
The 72-hour limit removed the active Preview. |
no-preview |
Preview is disabled for this pull request. |
preview-component-* |
The workflow selects a component and its runtime dependencies. |
preview-image-* |
The workflow selects a component's pull-request image. |
Watch the workflow:
PREVIEW_PR=123
gh pr checks "$PREVIEW_PR" --repo ScreamingFace/screamingface --watch
Read the maintained Preview comment:
gh pr view "$PREVIEW_PR" --repo ScreamingFace/screamingface --comments
The comment shows status, application links, Kubernetes commands, and SigNoz links.
Build dependencies
Component labels and image labels have different purposes.
- A
preview-component-*label selects a dependency-complete runtime profile. - A
preview-image-*label selects an image built from the pull request. - A selected component without an image label uses a reviewed
mainimage.
Each pull-request image uses this tag:
pr-<pull-request-number>-<first-seven-characters-of-head-sha>
The workflow records readiness against the full 40-character head SHA.
It adds preview only after all required pull-request image publications succeed.
The label is valid only for a branch inside ScreamingFace/screamingface.
flowchart LR
PR[Same-repository PR<br/>exact head SHA] --> PATHS[Detect changed component paths]
PATHS --> COMPONENTS[Set preview-component labels]
COMPONENTS --> CLOSURE[Resolve runtime dependencies]
PATHS --> IMAGES[Set preview-image labels]
IMAGES --> BUILD[Build changed images only]
BUILD -->|Engine image selected| BENCH[Build Engine benchmark<br/>FROM exact Engine image]
BUILD --> READY[Exact-revision readiness]
BENCH --> READY
CLOSURE --> READY
READY -->|slot available| LABEL[Add preview label]
READY -->|three active| QUEUE[Keep preview-queued]
LABEL --> ARGO[Argo discovers the PR]
Argo polls pull requests through a dedicated read-only GitHub App. The App is
installed only on ScreamingFace/screamingface. Its private key stays in Azure
Key Vault and reaches Argo through External Secrets Operator.
| Runtime image | Pull-request source | Reviewed dependency source |
|---|---|---|
screamingface-aigateway |
acropenminedpreview.azurecr.io, PR tag |
acropenmined.azurecr.io, full main-<40-character-sha> tag and digest |
screamingface-aigateway-ui |
acropenminedpreview.azurecr.io, PR tag |
acropenmined.azurecr.io, full main-<40-character-sha> tag and digest |
screamingface-scoreboard |
acropenminedpreview.azurecr.io, PR tag |
acropenmined.azurecr.io, full main-<40-character-sha> tag and digest |
screamingface-engine |
acropenminedpreview.azurecr.io, PR tag |
acropenmined.azurecr.io, full main-<40-character-sha> tag and digest |
screamingface-engine-benchmark |
Follows the Engine PR selection | Follows the Engine reviewed-main selection |
The reviewed GitHub Container Registry packages are private and returned 401.
Preview does not use them as a dependency-image source.
Path-to-build rules
The workflow follows the Docker inputs and deployed charts in the ScreamingFace repository. Tests and documentation alone do not build an application image.
| Component | Runtime-selection paths | Image-build inputs |
|---|---|---|
| AI Gateway | AI Gateway image inputs or apps/aigateway/charts/aigateway/ |
Dockerfile, pyproject.toml, uv.lock, and src/ |
| AI Gateway UI | UI image inputs or apps/aigateway-ui/charts/aigateway-ui/ |
Dockerfile, package locks, Next configuration, public/, and src/ |
| Scoreboard | Scoreboard image inputs or apps/scoreboard/charts/scoreboard/ |
Dockerfile, Python locks, src/, portal assets, and shipped benchmark artifacts |
| Engine | Engine image inputs or apps/screamingface-engine/deploy/helm/ |
both Dockerfiles, Python locks, url4.toml, src/, and packages/url4/ |
A root .dockerignore change selects all four components and all five images.
A file rename checks both the old path and the new path.
A deployed-chart-only change selects its component profile and builds no image.
For example, a change under packages/url4/ builds only Engine and benchmark.
Selected runtime profiles
The workflow can select more than one component. The final runtime is the union of each selected profile and its dependencies.
| Component label | Runtime services | Application routes |
|---|---|---|
preview-component-aigateway |
AI Gateway and its PostgreSQL database | aigw-pr-<number> |
preview-component-aigateway-ui |
AI Gateway, its database, and the admin console | aigw-pr-<number>, console-pr-<number> |
preview-component-scoreboard |
Scoreboard and its PostgreSQL database | leaderboard-pr-<number> |
preview-component-engine |
AI Gateway, its database, Engine, NATS, and Garage | aigw-pr-<number>, fusion-pr-<number> |
The kubeconfig route exists for every active profile. Image labels do not add runtime services. They select which runtime services use pull-request images.
preview-image-engine selects both the Engine and benchmark pull-request images.
Engine-only example
Assume a pull request changes only apps/screamingface-engine/.
The workflow sets preview-component-engine and preview-image-engine.
The Engine profile also selects AI Gateway because Engine needs it.
The workflow builds the Engine and benchmark images. The benchmark build uses the exact Engine pull-request image as its base.
AI Gateway uses its reviewed-main image from acropenmined.azurecr.io.
Its full tag and image digest prevent a floating dependency.
At runtime:
- Argo reads the Engine and AI Gateway charts from the pull request's full SHA.
- Argo deploys AI Gateway, its PostgreSQL database, Engine, NATS, and Garage.
- Argo does not deploy the AI Gateway console or Scoreboard.
- The Engine control plane uses the
url4-cloudServiceAccount. - Engine creates benchmark Jobs only inside the Preview namespace.
- Admission policy restricts each Job's image, token, volumes, and node placement.
- Large results move through the Preview Garage service.
- Run blobs move through a second Preview Garage,
preview-garage-blobs, with its own bucket and key pair (url4-cloud-blobs). The tenant chart's own blob Garage and blob Secret are not rendered.
The automated Engine qualification uses the provider-free literal hello.
This test does not prove a real paid model call.
Add your own provider API key
Preview copies no provider key from dev, staging, or production. Each author can create a personal AI Gateway API-key profile through the authenticated API.
Warning: use only a disposable, spend-capped provider key. Selected pull-request code and the pull-request author share this Preview trust boundary.
Never use a production provider key. Pull-request code can make calls with a key stored in its Preview. An AI Gateway image change can also inspect that key.
The API uses this request:
PUT /v1/auth/{provider}/profiles/{name}/api-key
{"api_key":"<your-provider-api-key>"}
Use https://aigw-pr-<number>.preview.dev.screamingface.ai as the API origin.
Cloudflare Access supplies your identity before Envoy sends the request.
This example reads both secrets without placing them in process arguments:
AIGW_URL="https://aigw-pr-${PREVIEW_PR}.preview.dev.screamingface.ai"
AIGW_PROVIDER="openrouter"
AIGW_PROFILE="default"
CF_ACCESS_TOKEN="$(cloudflared access token -app="$AIGW_URL")"
read -r -s -p "Provider API key: " PROVIDER_API_KEY
printf '\n'
printf '%s' "$PROVIDER_API_KEY" | jq -Rs '{api_key: .}' | \
curl --fail --silent --show-error \
--config <(
printf 'header = "cf-access-token: %s"\n' "$CF_ACCESS_TOKEN"
printf 'header = "Content-Type: application/json"\n'
printf 'url = "%s/v1/auth/%s/profiles/%s/api-key"\n' \
"$AIGW_URL" "$AIGW_PROVIDER" "$AIGW_PROFILE"
) \
--request PUT \
--data-binary @-
unset CF_ACCESS_TOKEN PROVIDER_API_KEY
AI Gateway validates the key with the provider. It then encrypts the key in the disposable per-PR PostgreSQL database. The response never returns the raw key.
Warning: do not put a real provider key in shell history, a pull-request comment, or a Kubernetes object.
Your key and profile disappear with the Preview. The console is an admin console. Pull-request authors do not automatically receive console administrator access.
Paid-model qualification requires a disposable, capped fixture credential. The automated provider-free proof intentionally does not provide one.
Runtime dependencies
Argo CD is the deployment controller. Pull-request CI never receives a Kubernetes credential and cannot change an Argo Application.
flowchart TB
subgraph GitHub
PR[Open PR with preview label]
SHA[Exact tenant commit]
end
subgraph Supply[Image supply]
PRACR[Preview ACR<br/>selected PR images]
MAINACR[Platform ACR<br/>reviewed-main dependencies]
end
subgraph Argo[Argo CD in aks-dev]
APPSET[PR ApplicationSet]
PLATFORM[Trusted platform Application<br/>infrastructure main]
PROJECT[Per-PR AppProject]
CHILD[Restricted workload Application<br/>exact tenant SHA]
end
subgraph Kubernetes[Host AKS cluster]
ACCESS[Access namespace<br/>sf-preview-access-pr-N]
WORK[Workload namespace<br/>sf-preview-pr-N]
BOUNDARY[Quota, admission, RBAC,<br/>network policy, generated secrets]
STACK[Selected runtime profile<br/>plus required dependencies]
end
subgraph Edge[Request path]
CF[Cloudflare Access and tunnel]
ENVOY[Protected Envoy Gateway]
end
subgraph Observe[Observability]
SIGNOZ[SigNoz]
end
PR --> APPSET
SHA --> CHILD
PRACR --> STACK
MAINACR --> STACK
APPSET --> PLATFORM
PLATFORM --> ACCESS
PLATFORM --> WORK
PLATFORM --> BOUNDARY
PLATFORM --> PROJECT
PROJECT --> CHILD
CHILD --> STACK
CF --> ENVOY
ENVOY --> STACK
ENVOY --> ACCESS
STACK --> SIGNOZ
ENVOY --> SIGNOZ
Argo ownership
The ApplicationSet checks the public ScreamingFace repository every 180 seconds.
It selects open pull requests with the preview label.
The trusted platform Application reads the infrastructure main branch. It owns:
- both namespaces;
- RBAC, quotas, limits, and network policies;
- Preview-generated secrets;
- Cloudflare JWT SecurityPolicies and HTTPRoutes;
- the access server; and
- the exact per-PR AppProject and child Application.
The child Application reads the exact pull-request commit. Its AppProject permits approved namespaced workload kinds in one namespace only.
Argo waits for the child Application at sync wave 3. It creates HTTPRoutes at wave 4, after the child creates and readies their backend Services.
The child cannot create RBAC, Secrets, NetworkPolicies, routes, external secrets, or cluster resources.
Kubernetes isolation
Preview uses the host AKS API, not a virtual cluster.
Each Preview has two namespaces:
| Namespace | Contents |
|---|---|
sf-preview-pr-<number> |
All tenant workloads and disposable data. |
sf-preview-access-pr-<number> |
The trusted access server and its ServiceAccount. |
Every Preview pod selects the tainted Preview node pool. Admission rejects a pod that bypasses the scheduler or omits the required placement.
Both namespaces enforce the Kubernetes restricted Pod Security Standard. The real tenant chart render checks every container before the platform change merges.
Default-deny network policy blocks other namespaces, private networks, and the Azure metadata endpoint. There is no blanket same-namespace traffic rule.
Explicit rules permit these internal paths only:
- AI Gateway and its migration Job to the AI Gateway database;
- Scoreboard and its migration Job to the Scoreboard database;
- selected Engine, Runner, and console Pods to AI Gateway;
- selected Engine and Runner Pods to NATS and Garage;
- Engine to the Kubernetes API; and
- the qualification Job to the SigNoz collector.
The workload namespace has resource and storage limits. The active Preview limit is three. The Preview node pool scales from one to two nodes.
Secrets and disposable data
External Secrets Operator generates new credentials for each Preview.
Generation uses CreatedOnce, so reconciliation does not rotate a running Preview.
Generated credentials cover:
- AI Gateway authentication and encryption;
- AI Gateway PostgreSQL;
- Scoreboard PostgreSQL;
- Engine authentication;
- Garage RPC authentication;
- Garage S3 access; and
- the blob Garage's S3 key pair and RPC secret (
url4-cloud-blobs).
No credential is copied from dev, staging, or production. Preview starts with no model-provider key. An author can add a personal encrypted profile.
Selected PostgreSQL databases use 4 Gi standard SSD claims. Both Garage
services use ephemeral storage. Engine NATS uses a 2 GiB JetStream file store
on a 3 GiB ephemeral volume; the NATS container requests 1 GiB and is limited
to 3 GiB of ephemeral storage. The Engine reserves 1 GiB of the store for its
shared events stream, so the store must stay at least as large as the Engine's
events.maxBytes; scripts/ci/preview-validate.sh checks this on every
change. This store supports Engine streams without another claim.
Database data survives Pod rescheduling, but all Preview storage disappears
with namespace deletion.
The Preview developer credential can read, create, change, and delete all Secrets in its workload namespace. It cannot read another Preview or the trusted access namespace.
How local Kubernetes access works
There is one real access server for each Preview. It runs in the trusted access namespace. Pull-request code cannot select its ServiceAccount or call its ClusterIP Service directly.
AKS exposes an authenticated public API endpoint. The developer needs no Azure login or VPN. The downloaded kubeconfig supplies the scoped credential and TLS data.
sequenceDiagram
participant D as PR author
participant C as Cloudflare Access
participant E as Envoy Gateway
participant S as Per-PR access server
participant G as GitHub API
participant K as AKS API
D->>C: Request kube-pr-N URL with Access token
C->>E: Forward verified Access assertion
E->>E: Validate issuer, audience, signature, and identity claims
E->>S: GET /kubeconfig with verified Access identity and GitHub token
S->>G: GET /user with GitHub token
G-->>S: Authenticated login
S->>S: Match GitHub login to PR author
S-->>D: Fixed-namespace kubeconfig and one-hour token
D->>K: kubectl with TLS verification and Preview token
K-->>D: Namespace-scoped response through RBAC
The access token:
- expires after no more than 3,600 seconds;
- binds to the current access-server pod;
- uses the workload namespace as its fixed context;
- permits logs, exec, port-forward, and workload changes;
- permits read-only inspection of RBAC, routes, policies, external-secret declarations, quota, and ServiceAccounts;
- cannot change those boundary objects; and
- cannot read another namespace or a cluster-scoped resource.
Download the kubeconfig
Set the local file path:
PREVIEW_KUBECONFIG="/tmp/sf-preview-pr-${PREVIEW_PR}.kubeconfig"
PREVIEW_KUBECONFIG_URL="https://kube-pr-${PREVIEW_PR}.preview.dev.screamingface.ai/kubeconfig"
Fetch the kubeconfig. The GitHub token must identify the pull-request author. The command keeps both bearer tokens out of process arguments.
CF_ACCESS_TOKEN="$(cloudflared access token -app="$PREVIEW_KUBECONFIG_URL")"
GITHUB_TOKEN="$(gh auth token)"
{
printf 'header = "cf-access-token: %s"\n' "$CF_ACCESS_TOKEN"
printf 'header = "X-Preview-Access-Token: %s"\n' "$CF_ACCESS_TOKEN"
printf 'header = "X-GitHub-Token: %s"\n' "$GITHUB_TOKEN"
printf 'url = "%s"\n' "$PREVIEW_KUBECONFIG_URL"
} | curl --fail --silent --show-error --config - > "$PREVIEW_KUBECONFIG"
unset CF_ACCESS_TOKEN GITHUB_TOKEN
chmod 600 "$PREVIEW_KUBECONFIG"
export KUBECONFIG="$PREVIEW_KUBECONFIG"
Use normal Kubernetes tools:
kubectl get pods
kubectl get events --sort-by=.lastTimestamp
kubectl logs POD_NAME --all-containers --tail=200
kubectl exec -it POD_NAME -- sh
kubectl port-forward service/SERVICE_NAME LOCAL_PORT:SERVICE_PORT
Check the namespace boundary:
kubectl auth can-i get pods
kubectl auth can-i get pods -n sf-fusion
kubectl auth can-i get nodes
The expected answers are yes, no, and no.
Download a new kubeconfig after one hour. A replacement access-server pod also invalidates the old token.
Edge requests
Cloudflare owns wildcard DNS, TLS, the Access application, and the tunnel. The tunnel sends traffic to the internal protected Envoy Gateway.
Envoy verifies the Cloudflare Access JSON Web Token before routing. It checks:
- issuer
https://openmined.cloudflareaccess.com; - the Preview Access audience; and
- keys from the Cloudflare Access JWKS endpoint.
The shared Gateway removes any client-supplied identity headers before JWT validation. Envoy then writes verified identity claims into new headers.
Each selected application route sends traffic to its matching workload Service.
The access route sends only /kubeconfig to the trusted access server. The
server verifies the Cloudflare signature, issuer, audience, lifetime, and human
identity again. It then verifies that the GitHub login matches the pull-request author.
No Preview automation changes Cloudflare configuration.
Logs and traces
The pull-request comment links to SigNoz Logs Explorer and Traces Explorer.
Both links start with a filter for sf-preview-pr-<number>.
The node collector sends container logs without a Preview application credential.
You can read the same workload logs with kubectl logs.
Preview telemetry has two distinct proofs:
- A trusted access-namespace Job sends synthetic logs, metrics, and traces.
- Envoy sends real spans for requests to active Preview routes.
The limits are important:
- A SigNoz filter is a view, not an authorization boundary.
- A user can change the filter and see other telemetry their SigNoz role permits.
- Synthetic telemetry does not prove application instrumentation.
- Envoy spans show edge requests only.
- Internal application spans do not exist until each application emits them.
- Kubernetes access remains the namespace-scoped path when log isolation matters.
Lifecycle and deletion
- A same-repository pull request selects component and image labels.
- The workflow builds only the selected pull-request images.
- The workflow marks the request as building or queued.
- Exact-revision readiness permits the
previewlabel. - Argo creates the trusted platform Application.
- The platform creates namespaces and security boundaries.
- The platform creates the exact AppProject and child Application.
- The child deploys the selected dependency-complete runtime.
- The pull-request comment publishes URLs and access commands.
- A new commit completes its required builds before Argo changes the Preview.
- Close, merge, label removal, or expiry removes
preview. - Argo deletes the child workload Application first.
- Argo then deletes the project, policies, and both namespaces.
- The old kubeconfig stops working after access-namespace deletion.
- A protected hourly process later removes inactive registry tags.
An active Preview expires 72 hours after the latest preview label addition.
At most three Preview environments can run at once.
Remove the label to end a Preview early:
gh pr edit "$PREVIEW_PR" --repo ScreamingFace/screamingface --remove-label preview
Preview deletion removes generated secrets, databases, Garage data, workloads, and Kubernetes access. Registry cleanup has a two-hour safety period and runs separately from namespace deletion.
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
preview-building does not clear |
One required image build or publication failed. | Open the Preview workflow and find the failed image job. |
preview-queued remains |
Three active Preview slots exist. | Wait for a slot. Do not add preview manually. |
| No Preview labels appear | The PR is a fork, a draft, disabled, or changes no deployable path. | Check PR state, labels, and changed paths. |
| The comment shows an old SHA | A newer build has not completed. | Wait for exact-revision readiness. |
| A URL redirects to Access | Authentication is required. | Sign in with the approved Cloudflare identity. |
| Kubeconfig download returns 403 | The GitHub token is invalid or belongs to a different pull-request author. | Authenticate as the author and retry. |
| Kubeconfig download returns 502 | GitHub identity verification is unavailable. | Retry after GitHub API recovery. |
kubectl reports an expired token |
The one-hour token expired or its server pod changed. | Download a new kubeconfig. |
kubectl returns 403 outside the Preview |
Namespace RBAC blocked the request. | Use only the namespace in the kubeconfig context. |
A dependency image returns 401 |
The Preview selected a private GHCR dependency. | Report the issue. Dependencies must use the reviewed-main platform ACR image. |
| Pods stay Pending | Capacity, quota, placement, or an image pull blocks scheduling. | Read pod events and the maintained PR comment. |
| A route fails after Access login | The route, policy, backend, or workload is not ready. | Check pods, Services, and events. Contact the platform operator for route status. |
| Logs are empty | The workload has not emitted a log, or the filter is wrong. | Use kubectl logs, then compare the exact namespace filter. |
| Traces show only Envoy | Applications do not emit internal spans yet. | Treat edge spans as request-path proof only. |
| A real model call fails | Your personal provider profile is absent or invalid. | Add or replace it through the authenticated AI Gateway API. |
| The console denies admin access | Pull-request authors are not automatic console administrators. | Use the AI Gateway API or ask an administrator when console access is necessary. |
| Resources remain after Preview ends | Argo deletion is delayed or blocked. | Contact the platform operator. Do not change platform boundaries. |
Platform operators use the repository-only runbook at
docs/runbooks/preview-environments.md. That file contains protected cluster,
identity, cleanup, recovery, and qualification procedures. It is not published
on Pulse Hub.
Security summary
The important boundaries are:
- pull-request CI can push Preview images but has no Kubernetes credential;
- Argo deploys from protected infrastructure source and the exact tenant SHA;
- the untrusted child Application writes approved workload kinds into one namespace;
- admission and network policy enforce placement and traffic boundaries;
- the access server verifies both Cloudflare and GitHub identity;
- the Kubernetes token is short-lived, pod-bound, and namespace-scoped; and
- deletion removes the ServiceAccount, workload, generated credentials, and data.
Known access limits remain:
- Kubernetes audit identifies the shared per-Preview ServiceAccount;
- Cloudflare Access logs identify the person who downloaded the kubeconfig;
- the downloaded bearer token is not bound to one developer device;
- the access server accepts a GitHub bearer token and reads its login;
- personal provider keys are available to selected pull-request code within that Preview;
- SigNoz does not provide per-Preview authorization; and
- a compromised Preview can affect its own namespace and Preview nodes.
See ADR-0010 for image supply and ADR-0014 for the current namespace architecture.