ADR-0019: Environment and subscription ownership for staging and production
Date: 2026-09-08
Status: accepted (2026-09-30 — implemented as decided: the production subscription openmined-production under the MCA billing account, aks-prod and its managed Postgres in it, state in the existing account, Postgres on the public endpoint behind the firewall; see the ledger rows of 2026-09-19 to 2026-09-30)
Decider: platform owner
Amended: 2026-09-18 — decision 1, the production subscription positions
Context
ADR-0002 fixed two planes and three clusters. It did not fix which Azure subscription owns each environment, where each Terraform root keeps its state, which identity applies each root, or who grants roles across subscriptions. The staging and production program recorded the gap as work package STP-M0-W02 and decision STP-D10, with STP-D14 (database and secret bootstrap) and STP-D19 (shared registry) adjacent (https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/decisions.md). ADR-0013 suspended the program workflow. The Phase 1 pilot plan (https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/phase-1-pilot-plan.md) is the working plan and routes this decision as item STP-P1-80.
The owner fixed three positions on 2026-09-02 (plan A3.1 to A3.3): production workloads live in a separate Azure subscription and are built there from day one; staging and production use managed Postgres from day one; clusters hold no durable state. This ADR records those positions and the positions that depend on them. A position the owner has not yet taken is marked Decision pending with its options and its default. Until the owner answers, the default is a proposal, not a fact.
Verified constraints (origin/main and live reads, 2026-09-08):
- One subscription exists today,
Microsoft Azure Sponsorship. It holds dev, the shared registry, the DNS zone, the Terraform state account, and the CI identities. - Every root keeps state in the storage account
sttfstateopenmined(resource grouprg-tfstate-prod-eastus): containertfstatefor non-production roots, containertfstate-prodfor production roots. Entra RBAC plus OIDC, no access keys. Keyazure/<system>/<env>.tfstate. The two containers are two RBAC scopes. - The dev root reads the platform registry
acropenmined, the preview registryacropenminedpreview, and the zonedev.openmined.orgby name through data sources. No root usesterraform_remote_state. - Pull requests carry no cloud credential. The apply identity
sp-infra-ci-applyholds a federated credential for workflow runs onrefs/heads/main. On push tomainthe generic Terraform workflow plans and applies every root whose directory is not namedprod(the Firecall roots are excepted; they have their own workflows). Production roots run only throughworkflow_dispatchin two phases (plan, then apply of the saved plan). The saved plan is not yet bound to the stack, commit, lock file, and checksum manifest (STP-M0-W24 and W25; plan items P1-14 and P1-15). - The
ci-identitiesstack (production root) is the trust root. It grants the apply identityContributorandRole Based Access Control Administratoron the current subscription, the latter under an ABAC condition that limits assignable roles to the whitelistci_assignable_roles(AcrPull,DNS Zone Contributor,Storage Blob Data Contributor,AcrPush,AcrDelete,Key Vault Secrets User, and the Firecall roles). The trust root applies locally through its break-glass runbook, never from CI. - No staging root, no production root, and no Postgres module exist yet
(
terraform/stacks/azure/aks-platform/holdsdev/only;terraform/modules/azure/holdsaksonly). No production subscription exists yet (plan item P1-10).
Decision
- Production lives in its own Azure subscription, production-only
(owner, 2026-09-02). Production resources are created there from the first
apply (plan item P1-16). Nothing but production runs in it. Every resource
carries the
owner,project, andexpirytags. The subscription carries one budget alert. - Decided 2026-09-18 (owner, by merging the change that adds
terraform/stacks/azure/subscription): the display name and alias areopenmined-production. This ADR names the subscription; it never prints the id. The id reaches Terraform as the repository variableTF_VAR_PROD_SUBSCRIPTION_ID, mapped to the environment variableTF_VAR_prod_subscription_idin the generic Terraform workflow (plan item P1-11); the subscription root publishes it on apply. - Decided 2026-09-18: the billing path is a second subscription under the
tenant's existing Microsoft Customer Agreement billing account, on the
same billing profile and invoice section as the Sponsorship subscription
(verified live 2026-09-18: the operator holds Billing account owner, the
profile carries an active payment method, and the account's second
profile holds only a deleted subscription). The subscription is billed,
not sponsored. Terraform creates it (
azurerm_subscriptionwithprevent_destroy, cancellation refused at the provider) from a human-applied root beside the trust root, never from CI: creating a subscription needs a billing role no CI identity holds. - Decided 2026-09-18: the budget is $1,100 per month, created with
aks-platform/prod(plan item P1-16) beside the resources it measures, as staging's is — not in the subscription root, where it would race the new subscription's provider registration. - Access on the new subscription follows ADR-0020 from its first day:
azure-ownersholds Owner,azure-readersholds Reader, no human holds Contributor, and the CI apply identity holds Contributor and the ABAC-constrained RBAC Administrator (plan item P1-11). - Staging lives in the current subscription, beside dev. Staging and dev share the subscription, the state account, the registry, and the DNS zone. They share nothing else: separate cluster, resource group, Key Vault, identities, and Postgres server.
- Decision pending: staging placement. Options: (a) the current subscription; (b) its own subscription. Default: (a), the plan's proposal (A5; the positions under the P1-10 row). Option (b) costs a second trust-root grant and a second state scope for disposable data.
- Terraform state stays in
sttfstateopenmined. Staging roots write to containertfstate; production roots write to containertfstate-prod. One state per stack per environment; the key mirrors the directory path (azure/aks-platform/staging.tfstate,azure/aks-platform/prod.tfstate). - Decision pending: state placement. Options: (a) the existing account and containers; (b) a new state account inside the production subscription (a second bootstrap root and a second break-glass ceremony). Default: (a). A move of production state into the production subscription is a Phase 2 decision, not a Phase 1 task.
- One apply identity, two routes.
sp-infra-ci-applyapplies staging on merge tomainthrough the generic Terraform workflow, the same route as dev. It applies production only through the owner's two-phase saved-plan dispatch, and only after plan items P1-14 and P1-15 have bound the saved plan and proven the binding live. Pull requests stay credentialless. On the production subscription the apply identity receivesContributorandRole Based Access Control Administratorwith the same ABAC condition it holds today (plan item P1-11, a trust-root change). - Decision pending: staging auto-apply on merge. Options: (1) auto-apply
like dev; the generic workflow already applies every non-
prodroot, so no workflow change is needed; (2) a dispatch path for staging, which needs a workflow change because the workflow keys on the directory nameprod. Default: (1), the plan's proposal (P1-13). - The shared registry and DNS zone stay production-owned in the current
subscription and are read by name. The staging root reads
acropenminedby name as the dev root does. The production root declares a provider alias for the current subscription and readsacropenminedthrough a data source. Neither root reads a DNS zone: staging and production run no external-dns and no cert-manager (plan items P1-16, P1-23, P1-25). No root reads another root's state. - Decision pending: whether the registry and zone stay production-owned and read by name until Phase 2. Options: yes, and STP-D19 is decided in Phase 2 on the pilot's evidence; or a production registry now (new spend, a new root, and a second image push path in Phase 1). Default: yes.
- Cross-subscription grants have one authority and one ceremony. A role
assignment whose role is on the
ci_assignable_roleswhitelist is created by the root that needs it and is not an ask: for exampleAcrPullfor the production kubelet identity onacropenmined, created in the current subscription by the production root through its provider alias. Any role outside the whitelist, any federated credential, and any change to the whitelist itself is a trust-root change: the owner approves it, and theci-identitiesbreak-glass runbook applies it.Azure Kubernetes Service RBAC Cluster Adminis not on the whitelist, so the cluster-admin grants for the hub identity, the apply identity, and the owner on each new cluster (plan item P1-11b) follow the same ceremony. - Managed Postgres, Phase 1 shape (owner, 2026-09-02: no database inside
a staging or production cluster; dev keeps its disposable in-cluster
Postgres). Staging and production each run one Azure Database for
PostgreSQL Flexible Server from a new module
terraform/modules/azure/postgres-flexible(plan item P1-12). Connectivity in Phase 1: a public endpoint with a firewall rule limited to that cluster's egress IP, andrequire_secure_transporton. The AKS module takes no subnet today, so private access is a Phase 2 item (plan B4). Backups: 7 days on staging; 35 days, geo-redundant, on production. Databases and roles are one per application (plan items P1-17a, P1-17b). - Decision pending: server tiers (spend). Options as in plan A5 and A11:
staging burstable single zone (
B2s, about $53 per month) and production General Purpose zone-redundant HA (D2ds_v5, about $267 per month); or other tiers the owner names. Default: the plan's tiers. The spend is approved with P1-13 and P1-16; this ADR records the tier only. - Decision pending: how a root sources the administrator credential.
Options: (a) Entra-only authentication: no password exists, the owner is
the Entra administrator, applications authenticate with tokens, and the
module has no
administrator_passwordinput; (b) the root reads a Key Vault record the owner writes in a break-glass window, and the apply identity holdsKey Vault Secrets Useron that vault (on the whitelist; the value enters remote state); (c) the root generates the password and writes it to the vault, and the apply identity holdsKey Vault Secrets Officer(a standing secret-write right on protectedmain). Default: (b), because the plan text names it; the WS1 specification asks the owner to weigh (a). No secret enters Terraform source in any option. - Secrets per environment. Each environment has its own Key Vault and
its own External Secrets identity, in that environment's subscription,
named by the dev root's pattern. Credentials are written in one
break-glass window per environment (plan items P1-17a, P1-17b). Nobody
holds standing read on a vault; proof of a record is the consuming
ExternalSecret's
Readycondition.
Out of scope, recorded elsewhere: the network and API-access model (STP-D11, Phase 2); where the hub (Argo CD and Kargo) runs (STP-D12; the plan proposes to accept the hub on dev for the pilot, A13); the identity cluster of ADR-0002 §3 (not built in Phase 1); production on-call (STP-D18).
Ownership, trust, data, interface, delivery, and operating effects: the table in Consequences.
Options considered
Option A — one subscription for dev, staging, and production
- Benefit: no new trust-root grant; one budget; one quota pool.
- Cost or risk: one blast radius for quota, RBAC, and billing; a dev mistake can consume production quota. The owner rejected it on 2026-09-02.
- Rollback or migration effect: nothing to roll back; production would need a subscription move later, and Azure resource moves do not cover AKS clusters.
Option B — one subscription per environment (three)
- Benefit: full isolation of staging from dev.
- Cost or risk: a second trust-root grant and a second state scope for staging; more Phase 1 time for no pilot benefit; staging data is disposable.
- Rollback or migration effect: staging would be rebuilt, not moved.
Option C — production separate, staging beside dev (proposed default)
- Benefit: production isolation from day one; staging costs one root and no trust change; state, registry, and DNS stay where they are.
- Cost or risk: staging shares dev's quota and RBAC scope during the pilot; the production subscription needs resource-provider registration, a budget, and a quota request before the first apply (plan item P1-10).
- Rollback or migration effect: staging can move to its own subscription in Phase 2 by rebuild; production never moves.
State placement — the existing account (proposed default) or a new account in the production subscription
- Existing account: no new bootstrap root;
tfstate-prodis already a separate RBAC scope; the legacy plan identity's role on the state account is being removed (plan items P1-50, P1-51). - New account: production state would sit in the production subscription, at the cost of a second bootstrap root and a second break-glass ceremony in Phase 1. Deferred to Phase 2.
Postgres connectivity — public endpoint with a firewall (proposed for Phase 1) or private access
- Public endpoint, one firewall rule per cluster egress IP, TLS required: works with the AKS module as it is (no subnet input); the endpoint is reachable from one IP on the internet and answers nobody else.
- Private access (VNet integration or a private endpoint): needs a network model, subnets, and private DNS that Phase 1 does not build (STP-D11). Phase 2.
Consequences
| Effect | What changes |
|---|---|
| Ownership | Production resources: the production subscription. Staging: the current subscription. Shared registry and DNS zone: production-owned, in the current subscription. |
| Trust | One grant pair on the production subscription for the apply identity (P1-11). Azure Kubernetes Service RBAC Cluster Admin on each new cluster for the hub identity, the apply identity, and the owner (P1-11b). Both are trust-root changes through the break-glass runbook. |
| Data | One managed Postgres server per environment; no database in a staging or production cluster; connectivity public-with-firewall until Phase 2; backups 7 and 35 days. |
| Interface | Every root has one backend.tf, one state key, one apply identity. Production roots declare a provider alias for the current subscription. Every variable has a CI-visible value (a checked-in default or a TF_VAR_* repository variable). |
| Delivery | Staging applies on merge to main (pending decision 4). Production applies only by the saved-plan dispatch after P1-14 and P1-15. |
| Operating | Two budget alerts; two Key Vaults; two break-glass windows for database credentials (P1-17a, P1-17b). |
- Required implementation work: plan items P1-10, P1-11, P1-11b, P1-12, P1-13, P1-16, P1-17a, P1-17b.
- Tests and operating work: a no-change plan on the dev root after the AKS module change; the read-only plan dispatch that proves binding (P1-15); the staging apply's plan output kept as evidence.
- Remaining risks: staging state sits in the non-production container that the legacy plan identity can still write until P1-51 applies (plan A13); the hub on dev manages production during the pilot (plan A13, STP-D12); the production apply path is not exact-bound until P1-14 lands and P1-15 proves it.
Supersession and rollback
A later ADR can move staging to its own subscription, move production state into the production subscription, give production its own registry (STP-D19), or make Postgres private (STP-D11). Each is a migration with its own plan; none is a rollback of this decision. Deleting the production subscription is an owner deletion decision, never an implementation rollback.
Program links
- Program decision: STP-M0-W02 (closed by this ADR once accepted), STP-D10 (closed), STP-D14 (Phase 1 connectivity, backup, and credential shape recorded; rotation and restore stay open), STP-D19 (deferred to Phase 2) — https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/decisions.md.
- Working plan: Phase 1 pilot plan, items P1-10, P1-11, P1-11b, P1-12, P1-13, P1-16, P1-17a, P1-17b, P1-80 — https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/phase-1-pilot-plan.md.
- Dependent work packages: the program workflow is suspended (ADR-0013); the plan's A7 tables replace packages.
Note 2026-09-09: PostgreSQL region
The current subscription is restricted from provisioning PostgreSQL Flexible Server in eastus (Azure capabilities API: no versions offered; a support request would be needed). Staging's server runs in eastus2, the nearest unrestricted region, which also offers the burstable and General Purpose SKUs and zone-redundant HA. Decision pending: whether production's server also goes to eastus2 or the production subscription is checked and a support exception is requested for eastus. Default: eastus2, decided when P1-16 is built.