Skip to content

Changing ScreamingFace on PULSE

ScreamingFace dev uses two repositories.

The tenant repository produces the application and Helm chart. Kargo advances routine immutable pins automatically under ADR-0008.

The infrastructure repository owns platform boundaries. Contract, permission, network, state, identity, and routing changes still require platform review.

flowchart LR
    C[Classify the change]
    TP[Tenant pull request]
    B[Merge-SHA image build]
    A[(ACR<br/>main-&lt;full-sha&gt;)]
    KG[Kargo promotion<br/>and render gate]
    IP[Two-line pin PR<br/>and required checks]
    AR[Argo reconciliation]
    P[Deployment, health,<br/>and functional proof]
    OK[Accept the release]
    RB[Revert to the previous<br/>known-good pin]

    C --> TP --> B --> A --> KG --> IP --> AR --> P
    P -->|all required proof passes| OK
    P -->|regression or uncertain result| RB
    RB --> AR

Start by classifying the change

Change First repository Platform review that follows Required proof
Application implementation OpenMined/screamingface Automatic Kargo promotion; no platform review unless the contract changes Source SHA, successful image build, ACR digest, running image digest, application-specific function
Compatible Helm chart change OpenMined/screamingface Automatic Kargo render, checks, and promotion Exact chart render, Argo reconciliation, positive and negative reachability
Shared package used by url4-cloud OpenMined/screamingface Advance url4-cloud; advance aigateway too only if its artifact or contract changed Each affected workflow and artifact is proven independently
RBAC, ServiceAccount, NetworkPolicy, edge, or identity OpenMined/infrastructure Review the platform-owned companion resources Boundary tests as well as successful traffic
Secret name, key, or credential behavior Usually both repositories Review the chart contract and the Key Vault/ESO mapping Metadata-only ESO status and the affected functional path
Database schema OpenMined/screamingface Review migration ordering, compatibility, and rollback before advancing Argo's retained migration result plus application behavior
Initial environment activation OpenMined/infrastructure Separate platform readiness review The target environment's full gate set
Routine release to an active environment Automated Kargo change No platform human review Exact Freight, checked pin change, Argo revision, and running digest

If one tenant merge changes both deployable applications, prove and pin both artifacts. If only one path-filtered workflow runs, do not invent an image for the unaffected application and do not advance its tag without evidence.

The normal dev change

  1. Open a ScreamingFace pull request and identify which deployable artifacts and chart contracts it changes.
  2. Merge only after the tenant tests and review pass. The merge commit—not the pull-request head—is the deployable source revision.
  3. Confirm each affected path-filtered workflow built that merge commit and published main-<full-sha> to ACR. A green workflow without a matching registry artifact is incomplete evidence.
  4. Kargo discovers the image and derives the chart revision from the same full-SHA tag.
  5. Kargo renders the exact tenant chart with platform values, opens the pin pull request, and waits for required checks.
  6. Kargo merges the pin pull request after the checks pass. No platform human approves a routine application-only promotion.
  7. After merge, require the relevant evidence classes: Argo source revisions, running image digests, migration evidence when applicable, continuous health, and release-specific functional acceptance.
  8. Update the evidence page and affected application page in the same change whenever their current facts or contracts moved.

If the tenant change alters a platform-owned contract, first check whether an active repository program governs it. Program work needs an approved work package and linked Asana task. Other work uses the normal Asana task and pull request flow. It does not need a work package.

The detailed artifact and chart transports are described in Delivery: repository to cluster.

What each green signal means

Signal It establishes It does not establish
Tenant tests passed The tested source behavior passed in CI An image exists in ACR or is deployed
Dev image workflow passed The build job exited successfully The cluster runs that digest
Infrastructure CI passed Platform manifests and repository checks passed Argo reconciled or the application works
Argo Synced/Healthy Desired resources reconciled and Kubernetes health assessment passed Runner, JetStream, identity, or provider behavior
Gatus health passed The named process, Service, and NetworkPolicy path answered A complete url4 run or model completion
Local-expression acceptance passed Token, route, WebSocket attach, Runner, JetStream, and terminal lifecycle worked together Provider credentials, profile assignment, spend control, or model output
Provider canary passed The approved profile reached the intended provider within its controls Unrelated routes, identities, or failure paths

Never summarize these as a single undifferentiated “green.”

Changes that need wider review

Escalate beyond a routine pin when the tenant change affects any of these contracts:

  • a resource kind permitted by the ScreamingFace AppProject;
  • a ServiceAccount or Kubernetes API call;
  • a pod or Service selector used by a NetworkPolicy;
  • an ingress/egress destination or port;
  • a Secret name, key, generation, or chart-side generation toggle;
  • AIGATEWAY_SECRET_KEY or encrypted provider credentials;
  • a migration or persistent-data format;
  • the /token, /ws, ai.url4.attach, Runner, or JetStream protocol;
  • public paths, timeouts, hostname, Cloudflare audience, or identity headers.

The platform companion resources may need to change in the same infrastructure pull request. Do not hide a wider permission or traffic-boundary change inside an otherwise ordinary chart bump.

Functional proof for url4-cloud

The release test is ordered:

  1. mint a fresh capability ticket;
  2. open the ticketed WebSocket;
  3. send an ai.url4.attach CloudEvent with from_sequence: 1;
  4. submit the local expression asynchronously with the same capability;
  5. require HTTP 202 and the complete terminal event sequence.

Opening the WebSocket is not an attach. Without the attach frame, the current subscriber gate can allow the request while the client receives heartbeats only. Capability tickets expire after 60 seconds, so each new test starts with a new ticket.

This local expression deliberately avoids a model provider. Provider acceptance is a separate gate requiring an approved profile, canonical provider-prefixed model ID, credential, and spend limit.

Platform operators execute the exact procedure from the internal docs/runbooks/verify-screamingface.md; privileged commands and client credentials are not published here.

Rollback

For an ordinary application or chart regression, promote the previous known-good Freight through the dev promotion runbook. The resulting pin change still passes required checks before Argo reconciles it.

Rollback is not always just an image change:

  • A database migration may make the old application incompatible. Define its backward-compatibility or recovery plan before merging the forward change.
  • A Secret rotation follows the credential-specific procedure; changing an image pin does not restore an old credential.
  • AIGATEWAY_SECRET_KEY has no in-place rotation or casual rollback path.
  • Renaming a persistent resource can cause Argo pruning. Preserve state unless a separately approved destructive operation explicitly retires it.
  • The retired trial stack is not a fallback. Dev rollback means restoring the previous tenant-owned revision.

If the evidence is ambiguous, stop the release and preserve the current state. Do not improvise a manual cluster mutation to make Argo appear green.

Do not

  • Do not run kubectl apply, edit live resources, or use rollout undo.
  • Do not deploy a branch name, latest, or another floating reference.
  • Do not advance a chart commit without its matching image artifact.
  • Do not put a token, key, password, private key, rendered Secret, or credential-bearing command in a pull request or task tracker.
  • Do not enable chart-generated Secrets under Argo.
  • Do not let a tenant chart silently widen platform-owned RBAC, networking, or edge policy.
  • Do not delete an ExternalSecret, Secret, Job history, or PVC as routine rollback.
  • Do not call a health endpoint or local arithmetic successful provider completion.

Before approving a platform-boundary change

  • The Asana task is linked. If an active program governs the change, its approved work package is also linked.
  • The affected tenant merge SHA and build run are recorded.
  • The ACR tag exists and its digest is recorded.
  • Each affected chart ticket and image tag resolve to the same merge commit.
  • The exact chart render was reviewed with platform values.
  • Contract changes have matching platform companion changes.
  • Migration and rollback consequences are written down.
  • CI passed, with any skipped or untested surface named.
  • The post-merge proof plan names which evidence classes are required.
  • The tenant evidence and application pages will remain accurate after merge.

See Credentials and identity before any credential-contract change, and Deployment decisions for the ownership rules behind this workflow.