gerodimos.dev

Architecture area

Hub and spoke

One management cluster runs the whole platform; dev and prod only run services. Every environment is deployed, inspected and observed from one place.

The shape

The management cluster (the hub) runs Argo CD, Backstage, Mimir and Grafana, and the operators with every platform resource. Argo CD syncs from Git and deploys to the dev and prod workload clusters (the spokes); Backstage reads them read-only; their Prometheus pushes metrics into Mimir. The operators commit back to Git. Git what runs where MANAGEMENT CLUSTER · THE HUB Argo CD one instance, five ApplicationSets Backstage desired and running state on one page Mimir + Grafana every environment's metrics Operators · Crossplane every Component, Release, Database and schema, for every environment sync commit WORKLOAD CLUSTERS · dev · prod Services only what Git deploys Thin infra ingress · DNS · certificates External Secrets · Atlas Operator Prometheus read-only role for Backstage deploy read only metrics
Scroll sideways to see all of it. The arrows show which side starts each connection. Only Argo CD can change a workload cluster.

How it works

  1. The hub runs the platform

    Argo CD, Backstage, Mimir and Grafana, the platform's operators and Crossplane all run on the management cluster, and every platform resource (Component, Release, Database, DatabaseSchema) lives there, whichever environment it targets. Dev and prod run services plus the small, identical set of infrastructure every cluster needs, such as the Atlas Operator that applies database schemas.

  2. A cluster joins through Git

    Each workload cluster's Terraform publishes its endpoint and CA to Secrets Manager. On the hub, External Secrets turns them into an Argo CD cluster entry labelled environment: dev and into Backstage's connection files. Neither side ever calls the other to set this up. Which clusters take part is two short lists in Git:

    # Argo CD delivers to
registeredClusters: [dev, prod]

# Backstage reads from
environments: [dev, prod]

  3. Identities, not credentials

    Argo CD and Backstage each have their own AWS IAM role, presented through EKS Pod Identity as short-lived tokens. Each workload cluster decides what a role may do: Argo CD gets admin; Backstage gets a group bound to a read-only ClusterRole (get, list and watch, plus pod logs), which Argo CD itself delivers from Git.

  4. Three directions

    Argo CD deploys out to the clusters whose label matches each file's environment. Backstage reads out: desired state from the hub, running state from the spoke, on one page. Each spoke's Prometheus pushes in to Mimir, so Grafana shows every environment side by side. See Observability.

Design choices

One Argo CD for everything

A single place to see and change every environment, instead of an Argo CD per cluster drifting apart.

Trade-off: the hub is a single point of failure for change. If it's down, nothing deploys, though services already running keep running.

Thin, identical spokes

Dev and prod are built from the same Terraform, differing only in the environment name, and run the same infrastructure set. A workload cluster holds nothing that can change the platform.

Matched by label, never by URL

ApplicationSets target clusters by their environment label. No cluster address appears anywhere in Git, so a rebuilt cluster with a new endpoint needs no change.

The spoke decides who gets in

Access is granted to an identity by the cluster being entered, not handed out as a credential. There is nothing to leak or rotate, and read-only is enforced by the spoke, not trusted to Backstage.

Producer publishes, consumer picks up

A cluster publishes its connection details and the hub turns them into what it needs. This is the same pattern a database uses to reach a service.

Rebuild in any order

All three clusters are created and destroyed daily. The IAM roles live outside them, and access entries bind to a role's internal ID, so the roles are never recreated and connections re-form on their own.

Not built yet

  • A highly available hub. One management cluster in one region; an outage pauses delivery and the central view.
  • Metrics history across hub rebuilds. Mimir keeps 48 hours without object storage, so a rebuilt hub starts empty.

Code

argocd

The ApplicationSets and their cluster selectors.

application-repositories

Which clusters are registered, and what each one runs.

helm-charts

Cluster registration templates and Backstage's read-only role.