Milestones — what shipped, with its evidence, and what comes next
The program at milestone altitude. The ledger is the per-PR truth; the roadmap is the phase checklist; THIS page is the narrative: each milestone, the work under it, and the proof it happened. Dates are 2026. Asana (⚙️Platform Engineering) owns current task status.
Staging and production: the Phase 1 pilot plan owns scope, gates, and order (the program milestone sequence is suspended, ADR-0013). Linked ADRs own architecture decisions. The ledger remains the merged-work history.
✅ M0 — Foundation (July, greenfield rebuild)
The monorepo rebuilt from zero, layered by toolchain (ADR-0001), with the trust chain as code.
- Repo layout:
terraform/modules+stacks ·kubernetes/bootstrap / infrastructure / apps / clusters / tenants ·platform/·observability/·finops/ - Trust root
ci-identities: OIDC-federated plan/apply identities, zero stored cloud secrets, ABAC-whitelisted RBAC admin, branch protection as code, break-glass runbook (the one local-apply exception) aks-dev(AAD-only, workload identity, Cilium enforced), ACR, DNS, Argo CD delivering itself (ADR-0002/0003)- Evidence: phases 1–3a ledger rows; Argo was the first product through its own pipeline
✅ M1 — First tenant live (Jul 18–31)
ScreamingFace on the platform, from tenancy model to public cutover.
- Tenant construct: AppProject fence, namespace registry, two rails
(ADR-0004/0006); tenant edge on
cf-protected— ONE shared listener, CF Access JWT verified at Envoy, identity stamped, tier-wide inbound strip with a sentinel witness (ADR-0005) - The chart rail: THEIR chart at a pinned merge SHA + OUR refusal-values (multi-source), secrets via ESO/Key Vault only
- Real-chart cutover:
fusion.dev.screamingface.aiserves url4-cloud from their chart (#167–#174); the trial retired with receipts (#175) - Evidence: cutover + retirement ledger rows; forged-header probe transcript; Gatus tenant board green
✅ M1.5 — Observability (Aug 1–3)
- SigNoz + ClickHouse (ADR-0007): logs first (the tenant's actual ask), traces armed, root-user fail-closed provisioning
- ClickHouse self-instrumentation defeat: 98% of the datastore was its own profiling; 11 GiB → sane, ~1.4 cores recovered (#201/#202)
- A 26-hour silent ingestion outage found and owned → the ingestion canary is on the future list, not forgotten
- Tenant apps #2/#3: aigateway (the credential custodian) + admin console, admin allowlist armed, edge split to its own Access app (#198)
✅ M1.7 — Continuous tenant delivery (Aug 4–5) — ADR-0008
The big one: from a 4-day manual repin latency to merge-to-dev in ~15–20 minutes with no platform human anywhere, every gate demanded by two independent external reviews met and live-verified.
Policy (the decision everything serves): the platform is never the gate on tenant deploys — their CI gates; whatever lands in ACR deploys. Dev now; the same model for prod at M2.
- K0 — the alarm before the mechanism: pin-staleness detector + issue alarm (#207/#208); later digest-keyed with REWRITTEN/BEYOND/MISSING verdicts and the attestation-manifest filter (#213)
- The tenant's one-liner: full-SHA ACR alias + self-triggering workflows
- OCI labels (their #508) — the registry became self-sufficient; no cross-repo credential exists in the pipeline
- K1 — Kargo v1.11 headless (#214): minimal surface, workload identity (verified in the pod spec), resources pinned, webhook drift pre-solved
- K2 — promotion objects + acceptance (#216/#219/#221): 3 image-only
Warehouses, PromotionTask with the render gate, PR-flow tail; the
promotecustom verb (native Role — Azure RBAC can't express it,can-ifalse-positives); acceptance 3/3 apps promoted end-to-end, every run digest-verified byte-for-byte (#223/#224/#225) - Operator tooling:
promote-dev.sh(#228) — freight resolution, deployed-state diff, rollback flagging; runbooktenant-dev-promotion.md(#221/#226) - The K3 sprint (Aug 5) — three gates, then the flip:
- Gate 1, checks as law:
validatealways-reports (in-job path guard), trust-root required contexts ×3 +enforce_admins, applied via the break-glass ceremony;allow_auto_mergefor the bot lanes (#231/#232) - Gate 2, the watched applier: argo-health alarm, GitOps ConfigMap acks
(a snooze is a reviewed PR with a name); en route root-caused the
eternal signoz OutOfSync — SSA empty-field drift (normalized rail-wide)
- a genuine renderer divergence (acknowledged, registered) (#234/#235)
- Gate 3, the boundary made real: all three companions rail-migrated one
at a time (guinea pig first;
preserveResourcesOnDeletionbecame standing policy after the controller reinstated hand-stripped finalizers — else the credential store's Postgres+PVC would cascade; the tunnel cut over with ZERO dropped probes; the Postgres pod survived untouched), whitelist strip (NetworkPolicy+ExternalSecret off the fence), store conditions, PSA baseline enforced ×5 (live-audited first), quotas with deliberate PVC caps + LimitRanges (#236–#242) - The flip (#243):
autoPromotionEnabled×3 +git-merge-pr— the machine merges its own two-line PR once the three required checks pass - Evidence: the 2026-08-05 ledger rows, ADR-0008, and the accepted repository controls. Private review notes are historical input, not program authority.
Future milestones
M1.8 — Security hardening register (historical snapshot)
Some items below have since landed. Some remain open.
Use the active program and Asana for current scope and task status.
From the two 2026-08-05 reviews' register, minus what the K3 sprint already landed. Asana: "Security hardening — consultant P0/P1 register".
- Argo CD + SigNoz off the public IP and behind
cf-protected+ Access; delete ingress-nginx (retired upstream) — the only stranger-exploitable finding; 1-hour LB stopgap available any day - Registry integrity: digest pinning in the promotion commit, ACR soft-delete/retention, split the dormant tenant PR fedcred; plan the Basic-ABAC migration window
- Protection-drift detector (declared==live, quarterly) · terraform.yml dispatch injection · url4 Runner Job RBAC narrowing · subscription Owner sprawl → PIM · SigNoz init-binary pinning · AKS audit-log export
- Read-only CI identity at the next break-glass ceremony (argo-health + pin-staleness least-privilege debts, documented in-file)
- Renderer-divergence item: CI's render is not byte-identical to Argo's —
bound what
validatecan promise
▶ M1.9 — Delivery observability (design exists)
- Freight-lifecycle → Slack (arrived / checking / deployed): a second subscriber on the existing alarm scripts; needs the Slack-webhook secret decision (one spine: SigNoz-native alerts vs a dedicated watcher)
- Deployment receipts: source SHA → digest → Freight → pin commit → Argo health → smoke, as one queryable record
- Ingestion canary (the 26-hour-outage lesson) + pod-currency leg on pin-staleness
▶ M1.10 — PR previews (implementation in review)
The current implementation replaces the deferred draft. It uses the shared dev AKS cluster and the dedicated Preview node pool.
- A trusted ApplicationSet watches labelled ScreamingFace pull requests.
- Each Preview gets separate workload and access namespaces.
- Component labels select the smallest complete runtime and image set.
- Cloudflare Access protects application routes and kubeconfig download.
- The pull-request author gets a one-hour, pod-bound Kubernetes credential.
- The credential permits workload inspection and changes only in that Preview.
- SigNoz links select logs and traces for the Preview namespace.
- Argo CD deletes the Preview when the label or pull request disappears.
▶ M1.11 — Leaderboard (tenant app #4)
leaderboard.dev.screamingface.ai (user, 2026-08-05). Chart ticket +
refusal-values + platform edge route on the shared listener. Open question
for the tenant: public or behind Access. Their T1 (release lane) rides
along.
▶ M1.12 — Tenant contract, batch 2 (their timeline, never our gate)
Test-gated full-SHA tag ("dev converges to the newest VERIFIED revision") · cosign keyless in their lanes, verified at our promotion gate · chart digest support · Q15 root-logger (re-framed as a governance requirement) · Q16 checksum/config.
▶ M2 — Production
The canonical program now splits this outcome into staged gates:
- Blockers before staging.
- Shared foundations.
- Staging build and qualification.
- Dark production build.
- Operations readiness.
- Cutover and stabilization.
Start at the program entry page.
Routine tenant releases follow ADR-0008. Production infrastructure remains human-gated through the two-phase saved-plan dispatch. STP-M0 must close the current plan-manifest binding gap before the first workload-production apply.
▶ M2.x — The AI-workload strategy (from the reviews' §7, adoptable slices)
- Make finops real BEFORE any GPU exists (OpenCost or Standard-tier cost analysis; the tenant label is already the join key)
- Confidential-VM pool pilot for the credential custodian + Trusted Launch as module default; attested key release needs Premium KV/HSM
- Publish first: the multi-tenant AI isolation reference architecture — "hosting someone else's AI product without trusting their code" (~70% already written across ADRs 0004/0005/0006 + the gateway-patterns pages)
- WORM audit export; GPU pools only after measured demand