Grafana

Observability and dashboarding platform with auto-provisioned datasources, 34 auto-enabled dashboard types, and multi-stack support that creates isolated dashboard folders per stack with automatic cluster label injection.

Architecture

Grafana Server - Helm-deployed visualization platform (:3000)
Datasources - Auto-provisioned from links: Prometheus, Loki, ClickHouse
Dashboard Folders - One folder per stack, dashboards mounted as ConfigMaps
Dashboard Engine - 34 types, auto-enabled based on what each Prometheus monitors

Attributes

Attribute Example Description
namespace REQ grafana Kubernetes namespace for all Grafana resources
admin_password REQ secret123 Admin password (stored in grafana.env, SOPS encrypted)
service_type ClusterIP Kubernetes Service type (ClusterIP, NodePort, LoadBalancer)
cpu_request 250m CPU request
mem_request 256Mi Memory request
cpu_limit 500m CPU limit
mem_limit 512Mi Memory limit
chart_version_grafana 11.2.0 Grafana Helm chart version

Links

Link Type Direction What It Automates
gateway-dataviz Inbound External access via APISIX/Kong gateway route to Grafana :3000 (route generated on gateway side)
prometheus-grafana Inbound Enables ServiceMonitor to scrape Grafana's own metrics on /metrics
grafana-prometheus Outbound Prometheus datasource in datasources.yaml
URL: http://prometheus-operated.{ns}.svc:9090
Dashboard auto-enable + multi-stack detection (see below)
grafana-loki Outbound Loki datasource in datasources.yaml
URL: http://{name}-gateway.{ns}.svc:80
Optional BasicAuth when Loki's enable_auth is set
grafana-clickhouse Outbound ClickHouse datasource per database sub-component in datasources.yaml
Installs grafana-clickhouse-datasource plugin
Internal: clickhouse-{name}.{ns}.svc:9000 (native, no TLS)
External (cross-stack): {public_host}:443 (native, TLS)
Link attribute (grafana-clickhouse): public_host — required for cross-stack ClickHouse connections (TLS endpoint)

Multi-Stack Dashboard Automation

When Grafana connects to a Prometheus that receives remote-write from other stacks, it automatically creates isolated dashboards for each stack:

How it works:
1. Grafana detects all Prometheus instances — the local one (grafana-prometheus link) plus any remote stacks that send metrics to it via prometheus-prometheus federation links.
2. A separate dashboard folder is created per stack (e.g., dashboard-stack-1/, dashboard-stack-2/).
3. For each remote stack, a hidden $cluster variable is set to that stack's system_name, and all PromQL queries in those dashboards get cluster="{stack_name}" injected — so each folder only shows metrics from that specific stack.
4. The local stack dashboards are left as-is (no cluster injection needed — it's the default).
5. Dashboard UIDs and titles are made unique per stack (UID gets a stack suffix, max 40 chars).
6. Only dashboards matching what each Prometheus actually monitors are included (disabled ones are removed).
Example: If your local Prometheus (stack-3) receives remote-write from stack-1 and stack-2, Grafana generates 3 dashboard folders. The stack-1 and stack-2 folders have cluster="stack-1" and cluster="stack-2" injected into every PromQL query, while stack-3 uses direct queries.

Dashboard Auto-Enable (34 Types)

Dashboards are automatically enabled per stack based on what that stack's Prometheus monitors. If Prometheus has a monitoring link to a component, the matching dashboard is included. Disabled dashboards are removed from output.

Always Enabled (3)

Dashboard File
Kubernetes kubernetes.json
JVM jvm.json
Kube State Metrics kube-state-metrics.json

Auto-Enabled by Prometheus Monitor Link (31)

Dashboard File Trigger (Prometheus link type)
PostgreSQL postgresql.json prometheus-postgresql, prometheus-keycloak, prometheus-jupyterhub
Kong kong.json prometheus-kong
Keycloak keycloak.json prometheus-keycloak
Keycloak Operator keycloak-operator.json prometheus-keycloak_operator
Node.js nodejs.json prometheus-nodejs
RabbitMQ rabbitmq.json prometheus-rabbitmq
MongoDB mongodb.json prometheus-mongodb
MongoDB Advanced mongo_advance.json prometheus-mongodb
k6 k6.json prometheus-k6
Python python.json prometheus-python
Kafka Broker kafka-broker.json prometheus-kafka
Kafka KRaft kafka-kraft.json prometheus-kafka
Kafka Exporter kafka-exporter.json prometheus-kafka
Kafka Connect kafka-connect.json prometheus-kafka
Kafka Bridge kafka-bridge.json prometheus-kafka
Kafka Operators kafka-operators.json prometheus-kafka
CNPG cnpg.json prometheus-cnpg
Cassandra cassandra.json prometheus-cassandra
MinIO minio.json prometheus-minio
OPA opa.json prometheus-opa
Elasticsearch elasticsearch.json prometheus-elasticsearch
Loki loki.json prometheus-loki
Neo4j neo4j.json prometheus-neo4j
APISIX apisix.json prometheus-apisix
ImmuDB immudb.json prometheus-immudb
Alloy alloy.json prometheus-alloy
FastAPI fastapi.json prometheus-fastapi
Cert Manager cert_manager.json prometheus-cert_manager
SeaweedFS seaweedfs.json prometheus-seaweedfs
vLLM vllm.json prometheus-vllm
Qdrant qdrant.json prometheus-qdrant
Note: PostgreSQL dashboard also triggers on prometheus-keycloak and prometheus-jupyterhub (they use PostgreSQL internally). All 6 Kafka dashboards share the same trigger (prometheus-kafka).

Generated Files

File Condition Contains
k8s/deploy/base/namespace.yaml Always Namespace
k8s/deploy/base/kustomization.yaml Always Resources, secretGenerator, configMapGenerator (one per dashboard per stack), labels, patches
k8s/deploy/base/patch/resource.yaml Always CPU/memory requests and limits patch
k8s/deploy/base/config/datasources.yaml Always SOPS Prometheus + Loki + ClickHouse datasource definitions (content depends on outbound links)
k8s/deploy/base/config/dashboards.yaml Always Dashboard file provider config — one provider entry per stack folder
k8s/deploy/base/dashboard-{stack_name}/*.json Per Prometheus link Dashboard JSON files per stack. Remote stacks get cluster label injection. Disabled dashboards removed. Each mounted as ConfigMap.
k8s/deploy/base/secret/grafana.env Always SOPS Admin password (admin-password={admin_password})
k8s/deploy/base/secret/registry.json Always SOPS Docker registry credentials (.dockerconfigjson)
helm/helm-values.yaml Always Grafana Helm values: service, serviceMonitor, plugins, extraConfigmapMounts (dashboard mounts per stack), extraSecretMounts (datasources + dashboards config)
helm/generate-yaml.sh Always Helm template render script → outputs grafana.yaml