ADR-0008 — Continuous tenant delivery: the platform is never the gate
- Status: accepted (2026-08-05; live the same day) · amendment 2026-09-09 proposed (the Kargo UI is the only promotion door for people; see the section "Amendment 2026-09-09" — Status: proposed until the owner accepts) · amendment 2026-11-04 accepted (Slack is a second promotion door for url4-cloud; section at the end)
- Deciders: platform engineering
- Informed by: two independent external reviews (2026-08-05), both GO-WITH-CHANGES; their amendments are folded into what shipped
Context
Tenant delivery was a hand-written "repin PR": a human read a changed-file
list, edited two fields (chartRef in the chart ticket, image.tag in the
values file), and merged. Measured cost when the pin-staleness detector first
ran: four days and ~24 builds of undeployed tenant work, invisible to
every green dashboard. The "review" the latency paid for was not a code
review — the tenant's own CI had already gated the merge.
Policy decision (2026-08-04, reaffirmed and extended 2026-08-05): the platform is never the gate on tenant deploys. Their CI gates; whatever lands in our registry deploys — dev now, and the same model for prod at M2 (their release process gates, the prod Stage promotes automatically; the platform's two-phase human gate remains for infrastructure changes only).
Two structural facts shaped the mechanism:
- The pin is two fields that must name one build, and the chart half is a 40-hex git commit — git refuses abbreviated-SHA fetches, registry tags originally carried 7. The tenant's full-SHA alias (their #508) made the registry self-sufficient; no cross-repo credential exists anywhere in the pipeline.
- The tenant repo is a path-filtered monorepo: each app's images form a disjoint SHA subsequence of main, so git+image join strategies are structurally dead and tags are unordered content hashes. Only ordering by the image's own build metadata works.
Decision
Kargo v1.11.0, installed headless (api.enabled: false, no external
webhooks, Argo CD/Rollouts integrations off), one image-only Warehouse per
app (NewestBuild over ^main-[0-9a-f]{40}$, cacheByTagPolicy: Forbid —
tags here are immutable by convention only), and one reusable
PromotionTask whose steps ARE the delivery contract:
- clone our repo and the tenant repo at the commit derived from the tag
- write both pin fields from that ONE tag — they cannot diverge by construction
- render their chart at that exact commit against our values (the render gate: a chart↔values mismatch fails the Promotion instead of surfacing as an Argo sync failure — this check runs where the read credential legitimately exists, which our CI structurally cannot)
- open a two-line PR as the pulse-bot App, labeled
automated-pin git-merge-pr— the machine merges its own PR once GitHub reports it mergeable, which under the enforced branch protection means every required status check is green
The gates that made "no hands" safe were prerequisites, not afterthoughts (both reviews' condition), all live-verified before the flip:
- Checks as law: required contexts (
Secret scan,Terraform fmt & validate,validate) +enforce_adminsonmain— with an empty required list,git-merge-prmerges instantly; that world no longer exists on this repo. - A watched applier: the argo-health alarm (self-editing issue, unreadable-is-a-finding, GitOps-reviewed named acks) paired with the pin-staleness alarm — two independent watchers on opposite ends of the chain, deliberately outside Kargo.
- A real tenant boundary: every companion (boundary) app migrated to the
platform rail;
NetworkPolicy+ExternalSecretstripped from the tenant whitelist; the ClusterSecretStore conditioned to named namespaces; PSA baseline enforced; ResourceQuotas with deliberate PVC caps.
Human promotion survives as rollback and break-glass only
(scripts/ops/promote-dev.sh, the promote verb via the native
dev-promoters Role — Azure RBAC cannot express custom verbs and
kubectl auth can-i false-positives on them).
Consequences
- A tenant merge is running in dev ~15–20 minutes later, no human anywhere, with a PR-shaped audit trail per delivery.
- The tenant's CI is the quality gate. Deliberately: dev converges to their newest built revision; making the full-SHA tag test-gated is their improvement to make, recommended but never our precondition.
- Freight records tag+digest at discovery → rollback = promoting an older Freight by name; "latest" can never silently change meaning.
- Known sharp edges, documented in the runbook: manual promotion of
non-candidate freight silently holds auto-promotion (the reliable escape
is the
autoPromotionEnabledtoggle — a no-op re-promotion errors at the push step under PR-flow); first-mint GitHub App tokens can 404 private clones (edge replication; retry heals);kargo-*RBAC names in a Project namespace are reserved by Kargo's reconciler. - Registered residual: renderer divergence — CI's kustomize/helm render
is not byte-identical to Argo's for at least one field class; tracked on
the security register because it bounds what
validatecan promise. - Rejected alternatives: Renovate and both image-updaters (cannot write
chartRef; cannot order content-hash tags), a bespoke poll-and-PR script (built, tested, then deleted at K3 — same tenant prerequisite, none of the Freight/Stage model), branch-protection bypass for the bot (strictly weaker than required-checks + machine merge).
Amendment 2026-09-09 — the Kargo UI is the only promotion door for people (STP-P1-87)
Status: proposed. Decider: platform owner. Drafted with STP-P1-35 (the
Phase 1 pilot plan, docs/programs/staging-production/phase-1-pilot-plan.md,
WS3 and A13). Accept after the first staging press through the UI (STP-P1-34)
has produced its evidence; until then the sections above stand unamended.
What changes
- "Installed headless" ends. The Decision above set
api.enabled: false— no API server, no UI, no admin credential, no internet surface. Staging and production need people, and people need a door. The Kargo API server and UI are on (kubernetes/infrastructure/kargo/base/kustomization.yaml), served atkargo.pulse.dev.openmined.orgon the admin edge (renamed fromkargo.dev.openmined.org2026-09-10 as the admin hostnames converge on<tool>.pulse.). The external webhooks server stays off; nothing dials in. - The UI is the only door for people. A person moves a build to staging
or production by one press in the UI. No
promote.sh, no promote workflow, no kubectl for a promoter — the owner fixed this on 2026-09-04.scripts/ops/promote-dev.shsurvives as the owner's break-glass for dev only. The two owner break-glass procedures (approve a build that never passed staging; hand-write the two-line pin when Kargo has deleted the build record) are written indocs/runbooks/promote-production.md. - The lock is the
promoteRole. Kargo's admission webhook authorizes every Promotion by SubjectAccessReview on the custom verbpromoteagainst the named target Stage.staging-promotersandprod-promoters(kubernetes/apps/sf-kargo/base/rbac.yaml) name people and Stages one by one: one ServiceAccount per person, mapped to the person's Entra login by object id;resourceNameslists the exactstaging-<app>andprod-<app>Stages. The owner is the only member ofprod-promotersin Phase 1. The API server acts on the cluster with its own credentials and authorizes each call as the mapped ServiceAccounts, so a promoter holds no Azure RBAC grant on the cluster — the two-grant rule of the headless design applies toUsersubjects only. - Posture. Entra login only (authorization code with PKCE; a public client; no client secret to store or rotate), the built-in admin account off, Secret management off (the namespace holds the git-write credential; the UI cannot read or replace it), and the hostname behind the admin-edge Entra policy like every other admin hostname (owner decision 2026-09-09). Kargo also authenticates every request itself, so a browser user signs in twice and an expired edge cookie makes the UI's API calls redirect until the page is reloaded; the Kargo CLI, which is bearer-only, is not supported through this hostname and is not needed, because the UI is the only promotion door for people. What this buys: nothing of Kargo is reachable from the admin edge, in the namespace that holds the git-write credential.
- Staging and production are not automatic in Phase 1. The Context above
sketched "the same model for prod at M2 (their release process gates, the
prod Stage promotes automatically)". Phase 1 keeps a named person's press
on
staging-<app>andprod-<app>(noautoPromotionEnabledfor either; Kargo enforces order only — production takes builds from staging, staging from dev — and people verify health, STP-P1-31). Automatic production promotion is a Phase 2 question, not a rejected option.
What does not change
The tenant's CI stays the gate for dev; dev auto-promotion, the two-field pin
derived from one registry tag, the render gate, the machine-merged pin PR
under required checks, and the two independent watchers (argo-health,
pin-staleness) all stand. The reserved kargo-* names, the auto-promotion
hold caveat and the first-mint 404 stay documented where they were.
Options considered
- A — the Kargo UI (chosen). One press, one Promotion object with the
actor annotated (
kargo.akuity.io/create-actor), the Role as the lock, no new code to own. Cost: the API server becomes a reachable surface; the membership list lives in two files (the Role subjects, and the chart'sapi.oidc.users.claimsfor the cluster-scoped project list). - B — a
promote.shscript over kubectl. Rejected by the owner (2026-09-04): every promoter would need cluster access and an Azure RBAC grant for the standard verbs; the audit trail would be a shell history. - C — a GitHub workflow with environment approvals. Rejected: required reviewers on environments need GitHub Enterprise on private repos (the ci-identities README records the 422), and a workflow approval names a job, not a person and a Stage.
Consequences
- Implementation: STP-P1-35 (values, hostname, RBAC, the Entra app registration as count-guarded Terraform in the trust-root stack, owner-applied), STP-P1-33 (the three runbooks), STP-P1-34 (the rehearsal).
- Operating: the ledger row per press carries the build's two-word name, the
pin PR and the running digest; a new promoter is one PR; the access audit
reads
rbac.yamlalongside Azure role assignments. - Risks kept:
usernameClaim: emaildepends on Entra emitting the claim (guests by default, members via the optional claim); the soak window before a production press is not set yet.
Supersession and rollback
A later ADR that automates production promotion (Phase 2) or moves the door
elsewhere supersedes this amendment. Turning
api.enabled back to false is an implementation rollback: it removes the
door, not this decision.
Amendment 2026-11-04 — Slack is a second promotion door (url4-cloud)
Status: accepted (owner decisions D1–D4, 2026-11-04). It amends "The UI
is the only door for people" above for one app, url4-cloud. Plan:
~/infra-workdocs/promotion-release/part2-plan.md. Runbook:
docs/runbooks/sf-slack-promote.md.
What changes
- A second door. A promoter can move url4-cloud to staging, then to
production, from Slack: a Promote button on the
/pulse freightcard (or/pulse promote url4-cloud <env>), a form that lists the last 10 builds the earlier stage ran (its current build picked) and shows the picked build, its PR, build run, CI and client, and one confirmation that promotes and releases the matching client (ADR-0022). A pick older than the build the stage runs is a rollback: it releases no client. After staging is live, the presser gets a [Promote to production] direct message. Production moves only on a person's press; PULSE never chains staging to production by itself. Kargo accepts only a build the earlier stage verified (its availability rule), so nothing skips a stage. - How PULSE reaches Kargo. The admin-edge Entra login on
kargo.pulse.dev.openmined.orgstays (2026-09-09 decision). The promote workflow signs in to Azure asgh-pulse-promote(federated credential for the main-only GitHub Environmentkargo-promote; Azure grants only AKS Cluster User on aks-dev), port-forwards to thekargo-apiService, and calls Kargo'sPromoteToStagewith its own AKS token. Kargo 1.11 accepts a token the cluster accepts and acts with that identity's Kubernetes permissions, so Kargo still builds the Promotion from the Stage's template and its admission webhook still checks thepromoteverb. - The lock is still the
promoteRole. A Role insf-kargograntsgh-pulse-promote(bound by object ID, asdev-promotersbinds a person)promoteon exactlystaging-url4-cloudandprod-url4-cloud. Two more Roles let it port-forward inkargoand read the two url4-cloud Argo CD Applications. Nothing else. - Who may press. The same people as the Kargo promoter RoleBindings
(today Stephen, Kevin, Irina), listed in
platform/pulse/people.json; a CI check keeps the two lists equal. The presser is the Slack user of the signed request. Their Slack accounts sign in with two-factor (D4). - First press wins. Before it promotes, the workflow lists the Stage's Promotions; a pending, running or finished promotion of the same build, from either door, refuses the second press and names the first presser.
- Kill switch. The repository variable
SF_SLACK_PROMOTE: onlyonlets the workflow promote. It ships off. The runbook has the switch and the faster and harder stops.
What does not change
The Kargo UI stays a door, with the same Roles and the same notices. Kargo enforces order (production takes only builds staging ran). Dev auto-promotion, the pin steps, the render gate, the machine-merged pin PR and every client-release guard stand. No Kargo secret is stored anywhere.
Options considered
- B — through the cluster (chosen). No edge change, no stored Kargo
token, and the same identity can read Argo CD to say "live on staging".
Cost: one trust-root identity and three Kubernetes Roles; port-forward is
granted for the whole
kargonamespace (Kubernetes cannot pin it to one Deployment's pods). - A — exempt the Kargo API from the edge login and store a Kargo API token. Rejected: it reverses the 2026-09-09 decision, puts the Kargo API on the internet, stores a long-lived secret, and gives no Argo read.
Consequences
- The Slack account of a promoter becomes enough to promote url4-cloud; two-factor sign-in on Slack is the matching control.
- A compromised PULSE Worker could start promotions in any listed person's
name. Bound: only builds the earlier stage ran and Kargo verified, every
press posted in
#pulse-requests, and the kill switch.