OpenSearch

OpenSearch Cluster - Data nodes (StatefulSet) for indexing and search
OpenSearch Dashboards - Visualization UI (Deployment)
SIEM Bootstrap Job - ArgoCD PostSync hook that imports index templates, dashboards, alerting monitors, and ISM policies
Prometheus Exporter - Metrics exporter for cluster health monitoring

SIEM Data Flow

post_gen cleanup: If siem is not "true" or no SIEM source links exist, the entire k8s/deploy/base/siem/ directory and siem-bootstrap-job.yaml are removed. Per-source files (index-template, alerting, dashboard) are removed individually when their corresponding link doesn't exist.

pre_gen_project.py detects each inbound SIEM link:

if link.component.is_referenced:
    # External source from another stack
    stack_name = link.component.name.replace('-falco', '')
    # e.g., "prod-falco" → stack_name = "prod"
    __siem_stacks[stack_name].append("falco")
else:
    # Local source (same stack)
    __siem_local_sources.append("falco")

No collision: Each stack's dashboards have unique IDs (prefixed with stack slug). The ?overwrite=true on import only overwrites the same stack's dashboards, not others. The global PCI-360 Overview has fixed IDs and is always safely overwritten.

Internal (same stack/cluster):
  endpoint: https://opensearch-cluster-master.{namespace}.svc.cluster.local:9200
  tls: skip verify
  auth: admin / {admin_password}

External (cross-stack/cross-cluster):
  endpoint: https://{remote_host}:443
  tls: skip verify
  auth: admin / {admin_password}

Detection: link.component.is_external (true = external)

remote_host setup: Typically exposed via an APISIX route on the SRE/central cluster. The remote_host attribute is the FQDN that resolves to this route (e.g., opensearch.sre.stack-3.source-lab.io). Port 443 is used for HTTPS through the ingress controller.

1. Edit base ndjson file(s) in templates/opensearch/.../siem/dashboards-*.ndjson
2. Regenerate stack (Stacktic build)
3. Push to git / ArgoCD sync
4. Bootstrap job runs automatically — cleans old objects, imports new ones

When dashboard IDs change between template versions, ?overwrite=true only updates objects with the same ID — old IDs persist forever in .kibana_1. The bootstrap job now extracts all IDs from the ndjson files it will import (plus a hardcoded list of legacy IDs from older versions) and deletes only those exact objects before importing. Customer-created dashboards are never touched.

# 1. Remove ISM policy from all SIEM indices
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/falco-*" ...
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/opa*" ...
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/apisix-*" ...
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/audit-*" ...
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/trivy-*" ...

# 2. Delete all SIEM indices
curl -sk -X DELETE "https://localhost:9200/falco-*" ...
curl -sk -X DELETE "https://localhost:9200/opa*" ...
curl -sk -X DELETE "https://localhost:9200/apisix-*" ...
curl -sk -X DELETE "https://localhost:9200/audit-*" ...
curl -sk -X DELETE "https://localhost:9200/trivy-*" ...

# 3. ArgoCD sync → bootstrap job recreates everything

# Remove then re-add ISM policy (per source)
curl -sk -X POST "https://localhost:9200/_plugins/_ism/remove/falco-*" ...
curl -sk -X POST "https://localhost:9200/_plugins/_ism/add/falco-*" -d '{"policy_id": "pci-dss-retention"}'

# Verify ISM state
curl -sk "https://localhost:9200/_plugins/_ism/explain/*?pretty" ...

ISM Lifecycle: hot (0–7d, rollover at 1d/30GB) → warm (7–90d, 1 replica, force-merged) → cold (90–365d, read-only, 0 replicas) → delete (365d+). New indices automatically pick up the ISM policy via ism_template patterns in the policy.