Skip to content

Milestones — what shipped, with its evidence, and what comes next

The program at milestone altitude. The ledger is the per-PR truth; the roadmap is the phase checklist; THIS page is the narrative: each milestone, the work under it, and the proof it happened. Dates are 2026. Asana (⚙️Platform Engineering) owns current task status.

Staging and production: the Phase 1 pilot plan owns scope, gates, and order (the program milestone sequence is suspended, ADR-0013). Linked ADRs own architecture decisions. The ledger remains the merged-work history.


✅ M0 — Foundation (July, greenfield rebuild)

The monorepo rebuilt from zero, layered by toolchain (ADR-0001), with the trust chain as code.

  • Repo layout: terraform/ modules+stacks · kubernetes/ bootstrap / infrastructure / apps / clusters / tenants · platform/ · observability/ · finops/
  • Trust root ci-identities: OIDC-federated plan/apply identities, zero stored cloud secrets, ABAC-whitelisted RBAC admin, branch protection as code, break-glass runbook (the one local-apply exception)
  • aks-dev (AAD-only, workload identity, Cilium enforced), ACR, DNS, Argo CD delivering itself (ADR-0002/0003)
  • Evidence: phases 1–3a ledger rows; Argo was the first product through its own pipeline

✅ M1 — First tenant live (Jul 18–31)

ScreamingFace on the platform, from tenancy model to public cutover.

  • Tenant construct: AppProject fence, namespace registry, two rails (ADR-0004/0006); tenant edge on cf-protected — ONE shared listener, CF Access JWT verified at Envoy, identity stamped, tier-wide inbound strip with a sentinel witness (ADR-0005)
  • The chart rail: THEIR chart at a pinned merge SHA + OUR refusal-values (multi-source), secrets via ESO/Key Vault only
  • Real-chart cutover: fusion.dev.screamingface.ai serves url4-cloud from their chart (#167–#174); the trial retired with receipts (#175)
  • Evidence: cutover + retirement ledger rows; forged-header probe transcript; Gatus tenant board green

✅ M1.5 — Observability (Aug 1–3)

  • SigNoz + ClickHouse (ADR-0007): logs first (the tenant's actual ask), traces armed, root-user fail-closed provisioning
  • ClickHouse self-instrumentation defeat: 98% of the datastore was its own profiling; 11 GiB → sane, ~1.4 cores recovered (#201/#202)
  • A 26-hour silent ingestion outage found and owned → the ingestion canary is on the future list, not forgotten
  • Tenant apps #2/#3: aigateway (the credential custodian) + admin console, admin allowlist armed, edge split to its own Access app (#198)

✅ M1.7 — Continuous tenant delivery (Aug 4–5) — ADR-0008

The big one: from a 4-day manual repin latency to merge-to-dev in ~15–20 minutes with no platform human anywhere, every gate demanded by two independent external reviews met and live-verified.

Policy (the decision everything serves): the platform is never the gate on tenant deploys — their CI gates; whatever lands in ACR deploys. Dev now; the same model for prod at M2.

  • K0 — the alarm before the mechanism: pin-staleness detector + issue alarm (#207/#208); later digest-keyed with REWRITTEN/BEYOND/MISSING verdicts and the attestation-manifest filter (#213)
  • The tenant's one-liner: full-SHA ACR alias + self-triggering workflows
  • OCI labels (their #508) — the registry became self-sufficient; no cross-repo credential exists in the pipeline
  • K1 — Kargo v1.11 headless (#214): minimal surface, workload identity (verified in the pod spec), resources pinned, webhook drift pre-solved
  • K2 — promotion objects + acceptance (#216/#219/#221): 3 image-only Warehouses, PromotionTask with the render gate, PR-flow tail; the promote custom verb (native Role — Azure RBAC can't express it, can-i false-positives); acceptance 3/3 apps promoted end-to-end, every run digest-verified byte-for-byte (#223/#224/#225)
  • Operator tooling: promote-dev.sh (#228) — freight resolution, deployed-state diff, rollback flagging; runbook tenant-dev-promotion.md (#221/#226)
  • The K3 sprint (Aug 5) — three gates, then the flip:
  • Gate 1, checks as law: validate always-reports (in-job path guard), trust-root required contexts ×3 + enforce_admins, applied via the break-glass ceremony; allow_auto_merge for the bot lanes (#231/#232)
  • Gate 2, the watched applier: argo-health alarm, GitOps ConfigMap acks (a snooze is a reviewed PR with a name); en route root-caused the eternal signoz OutOfSync — SSA empty-field drift (normalized rail-wide)
    • a genuine renderer divergence (acknowledged, registered) (#234/#235)
  • Gate 3, the boundary made real: all three companions rail-migrated one at a time (guinea pig first; preserveResourcesOnDeletion became standing policy after the controller reinstated hand-stripped finalizers — else the credential store's Postgres+PVC would cascade; the tunnel cut over with ZERO dropped probes; the Postgres pod survived untouched), whitelist strip (NetworkPolicy+ExternalSecret off the fence), store conditions, PSA baseline enforced ×5 (live-audited first), quotas with deliberate PVC caps + LimitRanges (#236–#242)
  • The flip (#243): autoPromotionEnabled ×3 + git-merge-pr — the machine merges its own two-line PR once the three required checks pass
  • Evidence: the 2026-08-05 ledger rows, ADR-0008, and the accepted repository controls. Private review notes are historical input, not program authority.

Future milestones

M1.8 — Security hardening register (historical snapshot)

Some items below have since landed. Some remain open.

Use the active program and Asana for current scope and task status.

From the two 2026-08-05 reviews' register, minus what the K3 sprint already landed. Asana: "Security hardening — consultant P0/P1 register".

  • Argo CD + SigNoz off the public IP and behind cf-protected + Access; delete ingress-nginx (retired upstream) — the only stranger-exploitable finding; 1-hour LB stopgap available any day
  • Registry integrity: digest pinning in the promotion commit, ACR soft-delete/retention, split the dormant tenant PR fedcred; plan the Basic-ABAC migration window
  • Protection-drift detector (declared==live, quarterly) · terraform.yml dispatch injection · url4 Runner Job RBAC narrowing · subscription Owner sprawl → PIM · SigNoz init-binary pinning · AKS audit-log export
  • Read-only CI identity at the next break-glass ceremony (argo-health + pin-staleness least-privilege debts, documented in-file)
  • Renderer-divergence item: CI's render is not byte-identical to Argo's — bound what validate can promise

▶ M1.9 — Delivery observability (design exists)

  • Freight-lifecycle → Slack (arrived / checking / deployed): a second subscriber on the existing alarm scripts; needs the Slack-webhook secret decision (one spine: SigNoz-native alerts vs a dedicated watcher)
  • Deployment receipts: source SHA → digest → Freight → pin commit → Argo health → smoke, as one queryable record
  • Ingestion canary (the 26-hour-outage lesson) + pod-currency leg on pin-staleness

▶ M1.10 — PR previews (implementation in review)

The current implementation replaces the deferred draft. It uses the shared dev AKS cluster and the dedicated Preview node pool.

  • A trusted ApplicationSet watches labelled ScreamingFace pull requests.
  • Each Preview gets separate workload and access namespaces.
  • Component labels select the smallest complete runtime and image set.
  • Cloudflare Access protects application routes and kubeconfig download.
  • The pull-request author gets a one-hour, pod-bound Kubernetes credential.
  • The credential permits workload inspection and changes only in that Preview.
  • SigNoz links select logs and traces for the Preview namespace.
  • Argo CD deletes the Preview when the label or pull request disappears.

▶ M1.11 — Leaderboard (tenant app #4)

leaderboard.dev.screamingface.ai (user, 2026-08-05). Chart ticket + refusal-values + platform edge route on the shared listener. Open question for the tenant: public or behind Access. Their T1 (release lane) rides along.

▶ M1.12 — Tenant contract, batch 2 (their timeline, never our gate)

Test-gated full-SHA tag ("dev converges to the newest VERIFIED revision") · cosign keyless in their lanes, verified at our promotion gate · chart digest support · Q15 root-logger (re-framed as a governance requirement) · Q16 checksum/config.

▶ M2 — Production

The canonical program now splits this outcome into staged gates:

  • Blockers before staging.
  • Shared foundations.
  • Staging build and qualification.
  • Dark production build.
  • Operations readiness.
  • Cutover and stabilization.

Start at the program entry page.

Routine tenant releases follow ADR-0008. Production infrastructure remains human-gated through the two-phase saved-plan dispatch. STP-M0 must close the current plan-manifest binding gap before the first workload-production apply.

▶ M2.x — The AI-workload strategy (from the reviews' §7, adoptable slices)

  • Make finops real BEFORE any GPU exists (OpenCost or Standard-tier cost analysis; the tenant label is already the join key)
  • Confidential-VM pool pilot for the credential custodian + Trusted Launch as module default; attested key release needs Premium KV/HSM
  • Publish first: the multi-tenant AI isolation reference architecture — "hosting someone else's AI product without trusting their code" (~70% already written across ADRs 0004/0005/0006 + the gateway-patterns pages)
  • WORM audit export; GPU pools only after measured demand