Skip to content

ADR-0007 — Observability backend: SigNoz, logs-first, on the OTEL spine

Status: accepted (2026-07-31) · Owner: platform · Supersedes: the kube-prometheus-stack-first MVP in platform-delivery-plan.md §Observability (that section's own text deferred this exact choice "to the observability stage"; this is that stage) · Amendment proposed: 2026-09-08 (Phase 1: one SigNoz on the hub serves dev, staging, and production through one ingest door; collectors are the contract; SigNoz Cloud is the Phase 2 port — Status: proposed until the owner accepts; see the amendment section).

Context

The platform committed early to two things that survive unchanged: the observability leasehold (a tenant declares what they want — a small reviewed record — and scoped observability appears; the fifth tenant primitive), and OpenTelemetry as the collection layer, precisely so the backend behind it could be swapped without touching a tenant. What was left open was the backend: the Grafana assembly (kube-prometheus-stack + Loki + Tempo, three stores, three query languages) versus an OTEL-native all-in-one.

Two facts changed the priority order since that text was written. The first tenant is now live and serving on its own deployable, and what that tenant actually needs first is logs — their application logs have been read on their behalf with kubectl logs throughout the delivery program. And the platform's own delivery work now produces enough operational surface (Argo, gateway, ESO, collectors) that self-observability is due.

Decision

  1. SigNoz (MIT core; ClickHouse, Apache-2.0) is the platform observability backend: logs, traces and metrics in one store with one UI, replacing the planned three-store Grafana assembly. Deployed as two independent add-ons on the existing infrastructure rail: kubernetes/infrastructure/signoz/ (store + query/UI + OTLP ingest) and kubernetes/infrastructure/signoz-k8s-infra/ (collection plane), enabled per-cluster like every other add-on.
  2. Logs ship first. Collection is the k8s-infra DaemonSet's filelog receiver reading container logs off the node — zero tenant-side change, no SDK, no netpol (nothing in a tenant namespace opens a connection; the agent reads files as the node). Kubernetes attributes stamp k8s.namespace.name and the openmined.org/tenant label at collection.
  3. The traces lane is built into the same spine, dormant. The OTLP ingest endpoints (4317/4318) exist from day one; tenant traces flow when tenant code emits them (their engineering, their timing) plus a per-namespace egress-allow NetworkPolicy (tenant namespaces are default-deny). Identity on telemetry is stamped from the source namespace at the collector, never trusted from the app — the observability restatement of the gateway's strip-and-re-stamp invariant (ADR-0005 §6/§9).
  4. The platform observes itself in the same stack — collection is cluster-wide (minus kube-system) from the start.
  5. The UI is served on the platform host rail (signoz.pulse.dev.openmined.org, nginx + DNS-01 + external-dns), with SigNoz-native auth (admin-registered, invite-only) until Keycloak + oauth2-proxy — the same posture as Argo CD.

The visibility limitation, stated plainly

SigNoz Community Edition has no per-user or per-team data scoping: every invited user (Admin/Editor/Viewer) sees all collected data. Fine-grained RBAC and SSO beyond Google OAuth are paid features. Consequently:

  • Inviting the tenant while platform telemetry is collected would expose platform internals to the tenant. Before the first tenant invite goes out, one of these must be chosen explicitly (tracked as the O2 decision): narrow collection to the tenant's namespaces for that window, accept the dev-only disclosure deliberately, or defer direct access.
  • before-2nd-tenant gate (hard): a second tenant makes shared global visibility indefensible. Options then: paid scoping, a scoping proxy in front of the query API, or per-tenant instances. This mirrors the committed Option-A→Option-B multi-tenancy path the delivery plan already records for the metrics design.

Staging

One shared stack for everything — never a stack per app. Apps get scoped slices of it, eventually by declaration (O4). Each stage ships alone.

Stage Delivers Gate / trigger
O1 — stack + logs Both add-ons live; tenant (sf-*) and platform container logs readable in the UI, zero tenant-side change; infra metrics (host/kubelet/cluster) flowing from the same collectors; UI on the platform rail + Gatus check; retention set deliberately this ADR's PR
O2 — access The CE-visibility decision made explicitly (narrow collection to the tenant's namespaces vs. accept the dev-only disclosure vs. defer); then tenant invites (needs only their email addresses) + saved per-namespace views — the hand-made version of what O4 automates first tenant invite
O3 — traces lane Gateway spans via EnvoyProxy.telemetry OTLP (no tenant code); per-namespace egress-allow netpols; tenant SDK spans flow whenever their code emits tenant asks, or first cross-service mystery
O4 — metrics content + the leasehold App metrics (prometheus-annotation scrape + OTLP); SigNoz dashboards/alerts as code in observability/, alerting into the existing notification spine; the lease record + generator so an app's scoped slice is one declared file second consumer, or first alert need
O5 — hardening Real multi-tenant scoping (paid tier / scoping proxy / per-tenant instance), SSO via Keycloak + oauth2-proxy, retention/quota tiers, HA + the prod profile before-2nd-tenant (hard) / before prod

Accepted dev exception — the ClickHouse chart credential

The bundled ClickHouse ships a published default admin password: a constant in the chart's values.yaml, byte-identical for every install in the world (verified — two renders after purging the chart cache produce the same value; there is no generator function in the template). Its default admin/networks/ip admits the whole RFC1918 range, which covers our pod CIDR.

The chart offers no secret-backed path for it — no existingSecret, no secretKeyRef; that exists only in externalClickhouse mode, i.e. only if we own and operate ClickHouse ourselves. Overriding the value in-chart would put a real credential in git, which this repo forbids outright.

So the decision is explicit: the credential is not the control — the NetworkPolicy is, exactly as it is for aigateway in cloudflare_headers mode. The signoz namespace runs default-deny with four reviewed crossings, and allowedNetworkIps is narrowed from the chart's six RFC1918 ranges to this cluster's pod CIDRs as defence in depth.

Owner: platform. Trigger to resolve: before prod, or before a second tenant — whichever comes first. Resolution options, in preference order: external ClickHouse with existingSecret; sync-time value injection from a platform secret; or accepting it permanently only if ClickHouse is provably unreachable from any non-observability workload.

Amendment 2026-09-08 — Phase 1: one SigNoz on the hub serves every environment

Status: proposed. Decider: platform owner. This section records the owner's 2026-09-04 observability decision (Phase 1 pilot plan, A3.6; https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/phase-1-pilot-plan.md) and the positions that depend on it. It resolves STP-D15 for Phase 1 when accepted. A position the owner has not yet taken is marked Decision pending with its options and its default. Acceptance condition: the ingest door (plan item P1-60) exists and its smoke test has passed; an ADR must not cite a door that does not exist. As of 2026-09-08 the door does not exist: kubernetes/infrastructure/signoz/overlays/dev/ holds the HTTPRoute for the SigNoz UI on the admin edge but no ingest (OTLP door) HTTPRoute and no matching SecurityPolicy, and the signoz NetworkPolicy admits the cf-protected proxies on 4317 only.

Decision (owner, 2026-09-04). In Phase 1 the store — ClickHouse, the SigNoz service and UI, and the central signoz-otel-collector (kubernetes/infrastructure/signoz/) — runs only on the hub (aks-dev). Staging and production run collectors only: kubernetes/infrastructure/signoz-k8s-infra/, one overlay per environment (plan item P1-62). Collectors are the contract: a tenant or a cluster sees only a collector endpoint. The Phase 2 port to SigNoz Cloud, if the owner takes it on the P1-61 cost data (plan B4), changes the exporter endpoint and its credential and nothing else. No SigNoz Cloud account exists in Phase 1.

The ingest door (plan item P1-60). Each remote collector exports OTLP/HTTP to one hostname on the hub's Cloudflare-tunnel edge (pulse-edge, the shared cf-protected listener, host-routed like every tenant hostname; no second listener). The door is: one Cloudflare Access application with one Service Auth policy and no policy for people; one Access service token per environment, minted when that environment's Key Vault exists and stored there by break-glass, consumed by the collector through an ExternalSecret; one route-targeted Envoy Gateway SecurityPolicy that validates the Access JWT against the application's AUD (the AUD is a public verifier input and lives in git); one HTTPRoute that forwards only the /v1/ OTLP paths to signoz-otel-collector:4318; and one new crossing in the signoz NetworkPolicy, the cf-protected proxy fleet on 4318 (the same fleet is already admitted on 4317 for preview spans). A request without a token is refused by Cloudflare before it reaches the cluster; a token for another application fails the audience check with 401. Every collector stamps k8s.cluster.name with its cluster: aks-dev already does through the chart's global.clusterName; staging and production stamp aks-staging and aks-prod. One store shows three clusters by name.

  • Decision pending: the ingest hostname. Options: (A) otlp.openmined.tools, a platform zone already in the account, where one exact-host DNS record and one exact-host Access application take precedence over the Firecall wildcards; (B) otlp.openmined.app, the same shape on the untrusted-preview zone; (C) a new platform zone (spend and trust). Default: (A).
  • Decision pending: service-token lifetime. Options: (A) Cloudflare's default duration, rotated every 90 days on the P1-41 cadence and recorded in the rotation runbook; (B) a 90-day duration, so an unrotated token expires by itself and the absent-telemetry alert (P1-63) fires. Default: (A).

Reconciliation with the Staging table. O5 ("HA + the prod profile", gate "before prod") is not a Phase 1 condition. Production telemetry flows into the dev-hosted SigNoz during the pilot; O5 moves to Phase 2 together with the SigNoz Cloud decision. O1 to O4 are unchanged. The before-2nd-tenant gate is unchanged. The O2 access decision is unchanged: no tenant invite exists, and O2 still gates the first one.

Reconciliation with the ClickHouse credential exception. Its trigger reads "before prod, or before a second tenant — whichever comes first". Production telemetry arriving on the hub is not production running the store; the store still runs on dev only. The ingest door ends at the collector, not at ClickHouse, and no workload on staging or production can reach ClickHouse.

  • Decision pending: the resolution of the exception for Phase 1. Options: (a) defer to Phase 2; the control stays the NetworkPolicy (the signoz namespace is default-deny, and the door adds one crossing that ends at the collector); (b) resolve now with an external ClickHouse and existingSecret (new spend and a data migration on the hub); (c) sync-time value injection from a platform secret. Default: (a), matching the plan's rule to build nothing that Phase 2 will undo.

Reconciliation with Decisions 2 and 3. "Zero tenant-side change" and "identity stamped at the collector" hold per cluster: the collectors on staging and production are platform add-ons, and tenant namespaces on those clusters open nothing. The service-token headers are added by the collector's exporter, never by tenant code.

Interim risks the owner is asked to accept (Phase 1).

  1. One store for three environments on the dev cluster. A hub outage blinds all three. Mitigations: the external uptime check on staging and production health endpoints is independent of the hub and pages the owner (P1-65); an absent-telemetry alert per cluster (P1-63), the first alert to land after the 2026-09-03 to 2026-09-08 incident, when a node restart wedged the single-replica ClickHouse replication queue and ingest stopped for five days with no alarm; a disk-usage alert at 80% (P1-61); a clean shutdown and PodDisruptionBudgets for ClickHouse and ZooKeeper (P1-61). Collectors buffer briefly and then drop.
  2. Decision pending: telemetry loss during a hub outage is accepted for the pilot, and STP-D16 (the drop budget and replay contract) stays open to Phase 2. Options: yes, or a replay contract in Phase 1. Default: yes.
  3. Production telemetry crosses the internet to the hub. The path is TLS at the Cloudflare edge, a per-environment Access service token, the tunnel, and the platform Envoy fleet. Compromise of one token exposes one environment's ingest path, not the store; rotation is per environment.
  4. SigNoz Community Edition global visibility (the limitation above) now covers three environments. Every invited user sees dev, staging, and production telemetry.

Sizing. Retention and the ClickHouse disk on the hub are sized for three environments from measured dev ingest (P1-61; observed 2026-09-04: 1.0 to 1.6 million log rows per day, 242 to 320 MiB of raw log body per day, 15% of a 25 GiB disk used).

  • Decision pending: retention. Options: (A) one setting for all environments, logs and traces 7 days and metrics 30 days (the dev target); SigNoz CE sets retention per signal, not per cluster; (B) logs 15 days for the pilot, about twice the log space. Default: (A).
  • Decision pending: disk. Options: (A) keep 25 GiB and re-derive 14 days after P1-62 lands; (B) grow to 64 GiB now (spend). Default: (A).

Where this is enforced. The staging and production cluster trees enable signoz-k8s-infra and no signoz store; a CI check rejects a store ticket outside kubernetes/clusters/aks-dev/ (P1-62). The ingest route carries a route-targeted JWT policy, so the anonymous-route tripwire in scripts/ci/k8s-validate.sh needs no allowlist line for it. A Gatus posture check proves that an anonymous request to the ingest hostname is refused (P1-60). The hostname's row in the gateway posture table lands with the door.

Rejected for Phase 1. A SigNoz store per cluster: three stores, three UIs, three retention settings, and three ClickHouse credential exceptions for a pilot. SigNoz Cloud now: the decision is Phase 2's, on cost data that does not exist yet. An Access application per collector: one application per door with one service token per environment is the boundary, and per-token audiences add nothing on a machine-only route.

Program links. STP-D15 (resolved for Phase 1 when this amendment is accepted) and STP-D16 (open) — https://github.com/OpenMined/infrastructure/blob/main/docs/programs/staging-production/decisions.md. Plan items P1-60 to P1-65 and P1-81.

Consequences

  • The ADR-0003 line "Gatus grows into kube-prometheus + Grafana" is amended by this ADR: Gatus stays (cheap, continuous, honest), and grows into SigNoz.
  • observability/ remains the content tree; dashboard/alert content becomes SigNoz-shaped (its dashboards are JSON; alerting integrates with the existing notification spine at the alerts stage).
  • ClickHouse is a real operational surface on a single-node dev cluster: explicit resource limits are mandatory (the chart ships requests-only and ClickHouse sizes caches off node RAM), ZooKeeper is required even single-node (no supported keeper mode in the chart), retention is a UI/TTL setting that must be set deliberately, and ClickHouse's internal system logs need TTLs or they grow unbounded.
  • The leasehold mechanism (appsets/observability.yaml + lease records) is unchanged in shape and lands at the leasehold stage; what a lease provisions becomes SigNoz-scoped content instead of Grafana folders.