gerodimos.dev

Architecture area

Observability

Every service is observable from its first deploy, in every environment, with no per-service setup. Two labels make that work.

The shape

Every cluster's Prometheus scrapes its services and records platform series, then remote-writes only series labelled with component and environment through a write-only gateway into Mimir on the management cluster. Grafana and Backstage read Mimir. EVERY CLUSTER · dev · prod · management Service /metrics · component + environment scrape Prometheus ServiceMonitor from the scaffold records platform:* rules MANAGEMENT CLUSTER remote write only labelled Gateway mimir.gerodimos.dev write-only · Basic Auth tenant pinned Mimir one store, every environment query Grafana one dashboard, every service Backstage Metrics tab · fixed queries developers & guests
Scroll sideways to see all of it. Highlighted arrows are the only path metrics take out of a cluster.

How it works

  1. Identity comes from the scaffold

    The golang-service template exposes Go runtime, HTTP request count and latency (by matched route, so cardinality stays bounded) and database pool metrics. It also ships a ServiceMonitor that turns the workload's platform labels into component and environment. A new service is scraped the moment it runs.

  2. Platform rules in every cluster

    Recording rules join cAdvisor and kube-state-metrics to the pod labels and produce platform:* series (CPU and memory against limits, restarts, readiness, replicas) carrying the same two labels. The labels are the membership test: no namespace or pod-name filters anywhere.

  3. Only the contract leaves the cluster

    Each Prometheus remote-writes series that carry both labels, and nothing else. Raw Kubernetes metrics stay local.

    writeRelabelConfigs:
  - sourceLabels: [component, environment]
    regex: ".+;.+"
    action: keep

  4. One store, two readers

    Grafana's Service Overview dashboard and Backstage's Metrics tab both query Mimir inside the cluster. Pick a component and an environment; every panel is a plain {component, environment} selector.

Design choices

Measure before shipping

Sending everything would have been ~268,000 series and ~9,000 samples/s, over Mimir's default limits. About 88% of it was API-server histograms nothing on the platform reads, and identical series from two clusters would have collided. The label filter sends ~100 series per environment.

Labels, not an allowlist

A newly onboarded service is included automatically, and a new dashboard panel needs no remote-write change in every environment.

Trade-off: a handful of series per service are sent that nothing visualizes yet.

Write-only from the internet

The public endpoint accepts only authenticated pushes. Each environment has its own credential, generated by Terraform and delivered through Secrets Manager, and the gateway overwrites the tenant header so a writer can't choose one. Reads never leave the management cluster.

Guests never write PromQL

Backstage's Metrics tab is public, so its backend runs a fixed set of queries. The browser supplies a component and an environment, nothing more.

A deliberately small Mimir

Single replicas and no object storage: 48 hours of metrics in memory-backed volumes. That fits clusters rebuilt every day.

Trade-off: an ingester restart loses all metrics. Long-term storage in S3 is the upgrade path.

Not built yet

  • Alerting. The rules exist; there is no Alertmanager or routing.
  • Workload health panels. Restarts, readiness and OOM kills are recorded but not on the dashboard yet.
  • Logs. Services log structured JSON ready for Loki; nothing collects it centrally yet.

Code

platform-scaffolds

Metrics and the ServiceMonitor every service starts with.

helm-charts

platform:* recording rules and the Service Overview dashboard.

backstage

The Metrics tab and its fixed-query backend.