Skip to content

ADR-0008 — Continuous tenant delivery: the platform is never the gate

  • Status: accepted (2026-08-05; live the same day) · amendment 2026-09-09 proposed (the Kargo UI is the only promotion door for people; see the section "Amendment 2026-09-09" — Status: proposed until the owner accepts) · amendment 2026-11-04 accepted (Slack is a second promotion door for url4-cloud; section at the end)
  • Deciders: platform engineering
  • Informed by: two independent external reviews (2026-08-05), both GO-WITH-CHANGES; their amendments are folded into what shipped

Context

Tenant delivery was a hand-written "repin PR": a human read a changed-file list, edited two fields (chartRef in the chart ticket, image.tag in the values file), and merged. Measured cost when the pin-staleness detector first ran: four days and ~24 builds of undeployed tenant work, invisible to every green dashboard. The "review" the latency paid for was not a code review — the tenant's own CI had already gated the merge.

Policy decision (2026-08-04, reaffirmed and extended 2026-08-05): the platform is never the gate on tenant deploys. Their CI gates; whatever lands in our registry deploys — dev now, and the same model for prod at M2 (their release process gates, the prod Stage promotes automatically; the platform's two-phase human gate remains for infrastructure changes only).

Two structural facts shaped the mechanism:

  1. The pin is two fields that must name one build, and the chart half is a 40-hex git commit — git refuses abbreviated-SHA fetches, registry tags originally carried 7. The tenant's full-SHA alias (their #508) made the registry self-sufficient; no cross-repo credential exists anywhere in the pipeline.
  2. The tenant repo is a path-filtered monorepo: each app's images form a disjoint SHA subsequence of main, so git+image join strategies are structurally dead and tags are unordered content hashes. Only ordering by the image's own build metadata works.

Decision

Kargo v1.11.0, installed headless (api.enabled: false, no external webhooks, Argo CD/Rollouts integrations off), one image-only Warehouse per app (NewestBuild over ^main-[0-9a-f]{40}$, cacheByTagPolicy: Forbid — tags here are immutable by convention only), and one reusable PromotionTask whose steps ARE the delivery contract:

  1. clone our repo and the tenant repo at the commit derived from the tag
  2. write both pin fields from that ONE tag — they cannot diverge by construction
  3. render their chart at that exact commit against our values (the render gate: a chart↔values mismatch fails the Promotion instead of surfacing as an Argo sync failure — this check runs where the read credential legitimately exists, which our CI structurally cannot)
  4. open a two-line PR as the pulse-bot App, labeled automated-pin
  5. git-merge-pr — the machine merges its own PR once GitHub reports it mergeable, which under the enforced branch protection means every required status check is green

The gates that made "no hands" safe were prerequisites, not afterthoughts (both reviews' condition), all live-verified before the flip:

  • Checks as law: required contexts (Secret scan, Terraform fmt & validate, validate) + enforce_admins on main — with an empty required list, git-merge-pr merges instantly; that world no longer exists on this repo.
  • A watched applier: the argo-health alarm (self-editing issue, unreadable-is-a-finding, GitOps-reviewed named acks) paired with the pin-staleness alarm — two independent watchers on opposite ends of the chain, deliberately outside Kargo.
  • A real tenant boundary: every companion (boundary) app migrated to the platform rail; NetworkPolicy + ExternalSecret stripped from the tenant whitelist; the ClusterSecretStore conditioned to named namespaces; PSA baseline enforced; ResourceQuotas with deliberate PVC caps.

Human promotion survives as rollback and break-glass only (scripts/ops/promote-dev.sh, the promote verb via the native dev-promoters Role — Azure RBAC cannot express custom verbs and kubectl auth can-i false-positives on them).

Consequences

  • A tenant merge is running in dev ~15–20 minutes later, no human anywhere, with a PR-shaped audit trail per delivery.
  • The tenant's CI is the quality gate. Deliberately: dev converges to their newest built revision; making the full-SHA tag test-gated is their improvement to make, recommended but never our precondition.
  • Freight records tag+digest at discovery → rollback = promoting an older Freight by name; "latest" can never silently change meaning.
  • Known sharp edges, documented in the runbook: manual promotion of non-candidate freight silently holds auto-promotion (the reliable escape is the autoPromotionEnabled toggle — a no-op re-promotion errors at the push step under PR-flow); first-mint GitHub App tokens can 404 private clones (edge replication; retry heals); kargo-* RBAC names in a Project namespace are reserved by Kargo's reconciler.
  • Registered residual: renderer divergence — CI's kustomize/helm render is not byte-identical to Argo's for at least one field class; tracked on the security register because it bounds what validate can promise.
  • Rejected alternatives: Renovate and both image-updaters (cannot write chartRef; cannot order content-hash tags), a bespoke poll-and-PR script (built, tested, then deleted at K3 — same tenant prerequisite, none of the Freight/Stage model), branch-protection bypass for the bot (strictly weaker than required-checks + machine merge).

Amendment 2026-09-09 — the Kargo UI is the only promotion door for people (STP-P1-87)

Status: proposed. Decider: platform owner. Drafted with STP-P1-35 (the Phase 1 pilot plan, docs/programs/staging-production/phase-1-pilot-plan.md, WS3 and A13). Accept after the first staging press through the UI (STP-P1-34) has produced its evidence; until then the sections above stand unamended.

What changes

  • "Installed headless" ends. The Decision above set api.enabled: false — no API server, no UI, no admin credential, no internet surface. Staging and production need people, and people need a door. The Kargo API server and UI are on (kubernetes/infrastructure/kargo/base/kustomization.yaml), served at kargo.pulse.dev.openmined.org on the admin edge (renamed from kargo.dev.openmined.org 2026-09-10 as the admin hostnames converge on <tool>.pulse.). The external webhooks server stays off; nothing dials in.
  • The UI is the only door for people. A person moves a build to staging or production by one press in the UI. No promote.sh, no promote workflow, no kubectl for a promoter — the owner fixed this on 2026-09-04. scripts/ops/promote-dev.sh survives as the owner's break-glass for dev only. The two owner break-glass procedures (approve a build that never passed staging; hand-write the two-line pin when Kargo has deleted the build record) are written in docs/runbooks/promote-production.md.
  • The lock is the promote Role. Kargo's admission webhook authorizes every Promotion by SubjectAccessReview on the custom verb promote against the named target Stage. staging-promoters and prod-promoters (kubernetes/apps/sf-kargo/base/rbac.yaml) name people and Stages one by one: one ServiceAccount per person, mapped to the person's Entra login by object id; resourceNames lists the exact staging-<app> and prod-<app> Stages. The owner is the only member of prod-promoters in Phase 1. The API server acts on the cluster with its own credentials and authorizes each call as the mapped ServiceAccounts, so a promoter holds no Azure RBAC grant on the cluster — the two-grant rule of the headless design applies to User subjects only.
  • Posture. Entra login only (authorization code with PKCE; a public client; no client secret to store or rotate), the built-in admin account off, Secret management off (the namespace holds the git-write credential; the UI cannot read or replace it), and the hostname behind the admin-edge Entra policy like every other admin hostname (owner decision 2026-09-09). Kargo also authenticates every request itself, so a browser user signs in twice and an expired edge cookie makes the UI's API calls redirect until the page is reloaded; the Kargo CLI, which is bearer-only, is not supported through this hostname and is not needed, because the UI is the only promotion door for people. What this buys: nothing of Kargo is reachable from the admin edge, in the namespace that holds the git-write credential.
  • Staging and production are not automatic in Phase 1. The Context above sketched "the same model for prod at M2 (their release process gates, the prod Stage promotes automatically)". Phase 1 keeps a named person's press on staging-<app> and prod-<app> (no autoPromotionEnabled for either; Kargo enforces order only — production takes builds from staging, staging from dev — and people verify health, STP-P1-31). Automatic production promotion is a Phase 2 question, not a rejected option.

What does not change

The tenant's CI stays the gate for dev; dev auto-promotion, the two-field pin derived from one registry tag, the render gate, the machine-merged pin PR under required checks, and the two independent watchers (argo-health, pin-staleness) all stand. The reserved kargo-* names, the auto-promotion hold caveat and the first-mint 404 stay documented where they were.

Options considered

  • A — the Kargo UI (chosen). One press, one Promotion object with the actor annotated (kargo.akuity.io/create-actor), the Role as the lock, no new code to own. Cost: the API server becomes a reachable surface; the membership list lives in two files (the Role subjects, and the chart's api.oidc.users.claims for the cluster-scoped project list).
  • B — a promote.sh script over kubectl. Rejected by the owner (2026-09-04): every promoter would need cluster access and an Azure RBAC grant for the standard verbs; the audit trail would be a shell history.
  • C — a GitHub workflow with environment approvals. Rejected: required reviewers on environments need GitHub Enterprise on private repos (the ci-identities README records the 422), and a workflow approval names a job, not a person and a Stage.

Consequences

  • Implementation: STP-P1-35 (values, hostname, RBAC, the Entra app registration as count-guarded Terraform in the trust-root stack, owner-applied), STP-P1-33 (the three runbooks), STP-P1-34 (the rehearsal).
  • Operating: the ledger row per press carries the build's two-word name, the pin PR and the running digest; a new promoter is one PR; the access audit reads rbac.yaml alongside Azure role assignments.
  • Risks kept: usernameClaim: email depends on Entra emitting the claim (guests by default, members via the optional claim); the soak window before a production press is not set yet.

Supersession and rollback

A later ADR that automates production promotion (Phase 2) or moves the door elsewhere supersedes this amendment. Turning api.enabled back to false is an implementation rollback: it removes the door, not this decision.

Amendment 2026-11-04 — Slack is a second promotion door (url4-cloud)

Status: accepted (owner decisions D1–D4, 2026-11-04). It amends "The UI is the only door for people" above for one app, url4-cloud. Plan: ~/infra-workdocs/promotion-release/part2-plan.md. Runbook: docs/runbooks/sf-slack-promote.md.

What changes

  • A second door. A promoter can move url4-cloud to staging, then to production, from Slack: a Promote button on the /pulse freight card (or /pulse promote url4-cloud <env>), a form that lists the last 10 builds the earlier stage ran (its current build picked) and shows the picked build, its PR, build run, CI and client, and one confirmation that promotes and releases the matching client (ADR-0022). A pick older than the build the stage runs is a rollback: it releases no client. After staging is live, the presser gets a [Promote to production] direct message. Production moves only on a person's press; PULSE never chains staging to production by itself. Kargo accepts only a build the earlier stage verified (its availability rule), so nothing skips a stage.
  • How PULSE reaches Kargo. The admin-edge Entra login on kargo.pulse.dev.openmined.org stays (2026-09-09 decision). The promote workflow signs in to Azure as gh-pulse-promote (federated credential for the main-only GitHub Environment kargo-promote; Azure grants only AKS Cluster User on aks-dev), port-forwards to the kargo-api Service, and calls Kargo's PromoteToStage with its own AKS token. Kargo 1.11 accepts a token the cluster accepts and acts with that identity's Kubernetes permissions, so Kargo still builds the Promotion from the Stage's template and its admission webhook still checks the promote verb.
  • The lock is still the promote Role. A Role in sf-kargo grants gh-pulse-promote (bound by object ID, as dev-promoters binds a person) promote on exactly staging-url4-cloud and prod-url4-cloud. Two more Roles let it port-forward in kargo and read the two url4-cloud Argo CD Applications. Nothing else.
  • Who may press. The same people as the Kargo promoter RoleBindings (today Stephen, Kevin, Irina), listed in platform/pulse/people.json; a CI check keeps the two lists equal. The presser is the Slack user of the signed request. Their Slack accounts sign in with two-factor (D4).
  • First press wins. Before it promotes, the workflow lists the Stage's Promotions; a pending, running or finished promotion of the same build, from either door, refuses the second press and names the first presser.
  • Kill switch. The repository variable SF_SLACK_PROMOTE: only on lets the workflow promote. It ships off. The runbook has the switch and the faster and harder stops.

What does not change

The Kargo UI stays a door, with the same Roles and the same notices. Kargo enforces order (production takes only builds staging ran). Dev auto-promotion, the pin steps, the render gate, the machine-merged pin PR and every client-release guard stand. No Kargo secret is stored anywhere.

Options considered

  • B — through the cluster (chosen). No edge change, no stored Kargo token, and the same identity can read Argo CD to say "live on staging". Cost: one trust-root identity and three Kubernetes Roles; port-forward is granted for the whole kargo namespace (Kubernetes cannot pin it to one Deployment's pods).
  • A — exempt the Kargo API from the edge login and store a Kargo API token. Rejected: it reverses the 2026-09-09 decision, puts the Kargo API on the internet, stores a long-lived secret, and gives no Argo read.

Consequences

  • The Slack account of a promoter becomes enough to promote url4-cloud; two-factor sign-in on Slack is the matching control.
  • A compromised PULSE Worker could start promotions in any listed person's name. Bound: only builds the earlier stage ran and Kargo verified, every press posted in #pulse-requests, and the kill switch.