ADR-0006: Tenant workload ownership — platform owns the trust boundary, the chart owns the workload
Date: 2026-07-31 Status: accepted Amended: 2026-08-13 — §3 preview build identity (see the amendment at the end) Amends: ADR-0004 §1 (table row 3 — the openings on top of the default-deny baseline are platform-authored, not "the tenant's own policies"; every other primitive in 0004 stands, as amended by 0005) Extends: ADR-0002 §1 (the delivery plane) — fixes the two credential paths a private tenant repo needs, which 0002 did not have to answer for first-party repos
Context
ADR-0004 defines a tenant as an AppProject + a namespace set + a vault path + an edge, and ADR-0002 makes Argo CD the only thing that writes to a cluster. Neither says who authors which object inside a tenant namespace. That was answerable in the abstract right up until the first real tenant workloads were staged, at which point the tenant's own Helm chart — which the sent proposal promises stays their deployment interface — turned out to create, alongside its Deployments and Services: its own ServiceAccount, its own Role, its own Secrets (generated at render time), its own NetworkPolicy, and its own database.
Every one of those is a statement about what the workload may reach or read, and every one of them is a file on the tenant's release cadence that we do not review. Meanwhile the mechanics of getting their chart and their images — trivial for a first-party repo — turned out to be constrained by hard platform facts that eliminated most of the obvious options (Consequences).
This ADR fixes the ownership line and the two transports it depends on. It is written from a first tenant already standing on it, with the first workload delivered this way and running.
Decision
1. The line: trust boundary vs workload
| Object class | Owner | Why the line falls here |
|---|---|---|
| Secret material — Key Vault records, ExternalSecrets, the Secrets they materialize | Platform | Credentials are minted under break-glass and never exist in a repo. A chart that generates them cannot be reasoned about under GitOps (§5). |
| Persistent storage and dev-only datastores | Platform | Storage class is a cluster fact, not an app fact; a chart default that assumes another distribution simply hangs. |
| ServiceAccount, and its token automount setting | Platform | Whether a pod carries an API token is a blast-radius decision. Set explicitly in both directions — silence reads as an oversight. |
| Role / RoleBinding — the exact rules | Platform | The fence cannot see inside a Role (Consequences, first item). |
| NetworkPolicy / CiliumNetworkPolicy — both the default-deny baseline and the openings | Platform | The openings are the isolation claim; they are also written against the chart's labels, so they are reviewed whenever the chart moves. |
| Deployment / StatefulSet / Service / Jobs, app configuration, images, resources, replica counts | Tenant chart | This is the tenant's product. Their release cadence governs it, unchanged. |
| Route, hostname and edge auth policy | Platform (unchanged from ADR-0005 §5) | Tenants supply values; they cannot weaken a tier guarantee. |
Two rules make the split operable:
- Where the chart can create a platform-owned kind, its toggle goes off in
our values (
serviceAccount.create,rbac.create, secret-generation toggles), and the platform satisfies the contract the chart expects on the other side — the named ServiceAccount, theexistingSecretkeys. - Ownership moves; privilege does not. A platform-authored Role starts as a byte-for-byte copy of what the chart granted itself at the pinned commit. This is not a tightening exercise, and framing it as one invites arguments the decision does not need. Tightening, if any, is a later reviewed diff.
2. Private chart source: a GitHub App, over HTTPS
Argo CD reads the tenant's private repository with a GitHub App installation
credential — a platform-owned App, installed on the tenant's repository by
their admins once — delivered Key Vault → ESO → a labeled repository Secret in
the argocd namespace. Two non-negotiable details:
- The repository URL must be the HTTPS form — App auth is HTTPS-only — and
it must match the tenant AppProject's
sourceReposentry byte-for-byte, or the fence rejects the source with an error that reads like a credential failure and is not one. - App ID and installation ID are not secrets and stay as literals in the manifest; only the private key comes from the vault.
Deploy keys are not an option for a repository the platform does not
administer: adding one requires admin, which a platform team onboarding
someone else's repository generally does not hold, and the API answers 404 —
GitHub's spelling of "insufficient permission" on a repository you can
otherwise read. The App, once installed, needs nothing further from the
tenant's admins.
3. Images: the tenant's CI pushes to the platform registry over OIDC
Their build authenticates to Azure with GitHub OIDC — a federated CI
identity holding AcrPush on the platform registry and nothing else, trusted
only from the tenant repository's default branch — and pushes the image it has
just built. GHCR remains their public artifact home, unchanged.
No registry credential is stored on either side. Inside Actions the built-in first-party token already carries the package access, so the image never has to be read from outside GitHub at all — which is why this is the better design and not merely the surviving one.
Current deviation — recorded 2026-07-31; the decision above is the target and is not being amended. The first identity standing on this section is configured more broadly than it describes, in two ways. It carries a second federated subject for pull-request builds, created with the identity ahead of the preview work it was intended for, so the trust is not restricted to the default branch as stated. And its push role is granted at registry scope rather than over the repositories that tenant publishes, so the same token could write another tenant's repository in the shared registry. No workflow requests a token on the extra subject, so nothing has assumed it and no environment depended on it — the gap is between what is configured and what is used, which is precisely the kind of gap a successful rollout leaves undisturbed. Remediation, required before this path carries anything pre-production: remove the unused pull-request federation, or move preview builds onto a separately constrained identity so preview trust never rides on the identity that publishes default-branch images; and narrow the push role off registry scope. Until both land, read the paragraph above as the intended invariant, not as a description of the live configuration. The concrete subjects, scope and actions are in the repo-internal rollout record.
A registry mirror is not a general fallback, and the Consequences below say why: a mirror needs a credential that can read the source, and for this organization's private GHCR packages no such credential exists. It works only where the source is actually readable — public packages, or a registry in another organization whose policy permits a token — and it is never the steady-state path. Instrumenting the tenant's build is what fills the registry; images built before instrumentation are re-pushed by a build, not mirrored.
4. Two transports, on purpose
Charts travel by Git + App credential; images travel by push + OIDC. The asymmetry is forced by the registry facts in Consequences, and it should not later be "simplified" by trying to make one credential serve both. Anyone attempting that will spend a day proving the same negative again.
5. Secrets are minted, versioned, and never chart-generated
Credentials are minted into Key Vault under a break-glass window, delivered by
ESO, and consumed through the chart's existingSecret contract with the
chart's own generation switched off. Record names carry a generation suffix,
so a new stack can be given fresh credentials while an existing stack keeps
reading the old ones. Keys that cannot be rotated in place — an application's
master encryption key over stored credentials — are minted once, recorded as
one-time, and never reused across generations without an application-supported
migration.
Consequences
- An AppProject constrains kinds, not rules — this is the whole reason RBAC
moved. The fence can refuse a
ClusterRoleoutright, but it cannot inspect the verbs inside aRoleit admits. A chart bump could quietly add Secret reads in a namespace holding provider credentials, and every gate we operate would still pass green. The cost is real and accepted: a chart that legitimately needs a new permission is now blocked on a platform PR. That is the point — a permission change becomes a reviewed diff in our repo rather than a line in someone else's release. - No personal access token of any kind can read this org's private GHCR, so
pull-based image delivery was never on the table. Classic PATs are forbidden
org-wide; ghcr.io does not support fine-grained PATs; and App installation
tokens are rejected by design — a container package's access list has
entry types for users, teams and repositories, and none for an app
installation, so the installation resolves to zero package grants. Actions'
GITHUB_TOKENworks only because it belongs to GitHub's own first-party Actions App: an allowlist of one, not a pattern we can join. The symptom to recognise, because every layer looks correct while it fails:docker loginsucceeds (that check is unscoped), then the scoped token exchange returns a token with an emptyaccessclaim and the manifest request 403s. Recorded with reproduction steps in the repo-internal GitHub App registration runbook. - Chart-generated secrets regenerate under Argo. Charts that mint their own
credentials with
lookup+randAlphaNumdepend on reading the live cluster; Argo renders withhelm templateand no cluster access, so the lookup finds nothing and every sync mints new values — credentials rotating under a running application indefinitely, presenting as permanent drift rather than as an outage. Any chart with that pattern must have its create toggle off and itsexistingSecretcontract satisfied from outside. deletionPolicy: Retaindoes not defeat ownerReference garbage collection.creationPolicy: Ownerputs an ownerReference on the materialized Secret;Retaingoverns only whether the provider-side value survives. Deleting an ExternalSecret to free a vault name therefore garbage-collects the live Secret and can take a running stack down with it. Generational record names, never deletions, are how a second stack is stood up beside a live one.- The platform now owns objects that live on the tenant's release cadence. A chart that renames its ServiceAccount, changes its expected Secret keys, or moves its selector labels breaks a platform-owned object rather than its own. Mitigations are structural, not vigilance: the chart ref is pinned to a specific commit, and label-keyed policies plus the mirrored Role are re-read at every bump.
- Onboarding grows a per-workload cost, not just a per-tenant one. ADR-0004 promised the second tenant is "a prefix, a project, a vault path, and an edge"; that still holds for the tenant construct, but each workload now also needs its trust-boundary set authored. Cheap to copy, impossible to skip.
- Two ownership seams mean two review surfaces, and a change that spans them lands in two repositories in order — platform foundation first, workload second. Sequencing is now a property of the design, not a preference.
- The OIDC push path has one sharp edge worth naming: a tenant's workflow
reads the Azure identifiers as repository variables, and a workflow reading
vars.*cannot see repository Secrets — supplying them as Secrets evaluates to empty and fails at login with values reported missing, in a way that looks like a broken federation rather than a wrong storage class. The onboarding runbook carries the working names and this failure mode; onboard from the runbook, not from memory. - Status honesty: every part of this decision has now been exercised on
aks-devwith the first tenant. Images: their CI pushed to the platform registry over OIDC. Charts: Argo's repo-server fetched a chart from their private repository with the App credential — the gate this ADR was accepted ahead of, now met rather than assumed. Ownership: one workload runs from their chart, on the image digest its chart ref was pinned to, with the platform's Secrets, storage, ServiceAccount, RBAC and network policy around it. What is still unproven is the application's behaviour under this arrangement — a second tenant workload and functional acceptance are both outstanding — not the arrangement itself. Exercised is not the same as narrowed: the push identity is currently trusted and scoped more widely than §3 describes, with the remediation recorded in the deviation note there. - The split is only as good as the values file that enforces it. The create-toggles and the labels our values stamp are load-bearing: the first rollout found the chart's server pod carrying no component label, so a policy written against the obvious selector also matched a migration hook. Render the pinned chart and read the labels it actually emits; do not write policy against the labels you expect. Expect the same class of gap on the hook objects a chart creates (Job ServiceAccounts especially) — they are easy to miss because they exist only briefly.
Amendment — 2026-08-13: preview builds ride a separate identity
Routed by the preview-environments program
(PVW-D02, work package
PVW-M0-W02).
The owner decided DECIDE PVW-D02: OPTION A on 2026-08-13. Section 3's
invariant for the main image identity is unchanged; this amendment adds the
separately constrained preview path its deviation note demanded.
Decision. Preview images are built and pushed by a new dedicated service
principal in the trust root (working name sp-screamingface-preview-ci),
following the existing tenant-SP pattern:
- One federated credential. Issuer
https://token.actions.githubusercontent.com, subject exactlyrepo:OpenMined/screamingface:pull_request(the underscore form GitHub issues — some Microsoft examples show a hyphen form that fails silently), audienceapi://AzureADTokenExchange. - One role:
AcrPushon the ADR-0010 preview registry only, granted beside the identity in the trust root via a by-name lookup. The registry holds nothing except previews, so registry scope is acceptable there. - No stored secret exists; nothing rotates. Trust review is platform-owned.
- The identity joins the control-drift expected state in the same change that mints it.
Trust statement, recorded honestly. A same-repo pull request executes its own workflow content, so anyone who can open a pull request in the tenant repository can mint a token on this subject. That population is exactly the preview population, and the write surface is the disposable preview registry. Fork pull requests receive no token by default; the implementation verifies the org setting and decodes one real token before first use.
Remediation routing. With preview trust carried here, removing the dormant pull-request credential from the main identity (STP-M0-W10/W20) is a pure deletion with nothing depending on it. Narrowing the main identity's push scope stays open under STP-D19. Implementation of this amendment is PVW-M1 work behind its own gates (break-glass apply, after the preview registry stack applies).
Amendment — 2026-08-24: preserve preview trust after repository transfer
The ScreamingFace repository moved after the 2026-08-13 decision. GitHub now
issues the immutable subject
repo:ScreamingFace@318002732/screamingface@1177944246:pull_request.
This subject replaces only the former OpenMined/screamingface subject.
ADR-0013 suspended the PVW delivery workflow. It did not change the separate Preview identity or registry boundary. The current Terraform source defines the identity. The Preview architecture guide defines its use. STP-M0-W10 and STP-M0-W20 still own removal of dormant trust from the main identity. STP-D19 still owns its push-scope reduction.