Trading the Grafana stack for VictoriaMetrics on OpenShift

A few hours after I moved Loki/Mimir/Tempo onto plain NFS, Mimir was still unhappy — a single monolithic ingester trying to hold ~2 million active series in 8 GB, OOM-looping on every WAL replay. The honest conclusion was that a monolith on a homelab box is the wrong tool for that series count. So I installed the VictoriaMetrics Operator and rebuilt the metrics half of the stack around it: VMSingle for storage, a 4-shard VMAgent for scraping, and VLSingle + VLAgent for logs. This is the story of the migration, including the four separate walls I walked into.

The target shape

#

• VMSingle — single-node metrics TSDB, data on the NFS telemetry export (victoriametrics/vmsingle), pinned to the two nodes with 10 Gbps NICs.
• VMAgent, shardCount: 4 — four agents, one per host (anti-affinity), splitting all scrape targets by consistent hashing. No duplicate samples, even for the dozen-plus static external exporters.
• VLSingle / VLAgent — VictoriaLogs storage + a per-node log collector.
• Repoint OpenShift’s platform Prometheus remoteWrite from Mimir to VMSingle, swap Grafana’s datasources, then delete Mimir and Tempo.

The nice surprise: the operator auto-converts existing Prometheus-Operator ServiceMonitor/PodMonitor/Probe resources into its own scrape configs. ~80 of my jobs lit up for free. I only had to translate Alloy’s bespoke jobs — the prometheus.io/scrape annotation discovery, the service-endpoint discovery, and the static exporters (ipmi multi-target, nut, traefik, databases, …) — into VMScrapeConfig CRs.

Wall #1: the config-reloader image that didn’t exist

#

VMAgent pods went straight to ImagePullBackOff on an init container:

Failed to pull image "victoriametrics/operator:config-reloader-v0.68.5":
  registry.connect.redhat.com/victoriametrics/operator: name unknown

The operator defaults its config-reloader to an unqualified image name, which OpenShift’s registry search helpfully resolved to the Red Hat certified registry — which carries the operator image but not the config-reloader-* tag. The operator binary itself runs from quay, so the fix was to pin the reloader there too, via the Subscription so OLM keeps it:

# Subscription victoriametrics-operator
spec:
  config:
    env:
      - name: VM_CONFIG_RELOADER_IMAGE
        value: quay.io/victoriametrics/operator:config-reloader-v0.68.5

(VMSingle and VLSingle came up fine — only VMAgent uses the reloader sidecar.)

Wall #2: metricsPath vs path

#

After the agents started, most of my static exporters were simply absent — no up series at all. Some worked (unpoller, the nvidia exporters); the rest didn’t. The pattern: every scrape I’d given a metricsPath: was being silently dropped by the operator. The field on VMScrapeConfig is path, not metricsPath. The CRD has x-kubernetes-preserve-unknown-fields, so the wrong key applied cleanly and then evaporated at config-generation time. One sed and they all appeared. (Their job label is scrapeConfig/<ns>/<name> — worth knowing when you query.)

While here: the platform ServiceMonitors that use a tls-client-certificate-auth scrape class throw operator errors and get skipped — which is fine, because the platform Prometheus scrapes those natively and now remoteWrites them to VMSingle. So I deleted my redundant kubelet/cadvisor scrapes entirely.

Wall #3: the platform repoint that fought Argo

#

Editing cluster-monitoring-config to point at VMSingle, then oc apply-ing it, did nothing — Argo’s self-heal reverted it, because I hadn’t committed the change yet, so Argo’s source of truth still said Mimir. Commit, push, re-apply. Then the Prometheus CR updated but the running pods kept the old config (secret mount sync lag), so a delete pod prometheus-k8s-0/1 forced the reload. After that: node_cpu_seconds_total, kube_pod_info, apiserver_request_total all flowing, 0 failed sends — versus the two billion failed samples that had piled up against the dying Mimir.

Wall #4: VLAgent collects nothing (still open)

#

The metrics side is done and verified. Logs are not. VLAgent’s brand-new k8sCollector (VictoriaLogs v1.50) runs, reads /var/log/pods fine, has the RBAC, writes a checkpoints file — of []. Zero rows, zero errors, and even a freshly-created pod’s logs never get tailed. I went deep:

• restricted-v2 forbids the persistent-queue emptyDir for the container UID → moved tmpDataPath to /tmp.
• Host /var/log/pods is unreadable without the spc_t SELinux type → except the operator drops seLinuxOptions from the CR, so I had to patch it onto the DaemonSet directly. After that the container reads the logs by hand…
• …and the collector still tracks nothing.

That looks like a bug in a very new feature. Rather than rabbit-hole it, I left Loki + Alloy running (logs keep flowing there) and shipped everything else. The next move is either pinning a different VictoriaLogs build or dropping a proven Vector/Fluent-bit DaemonSet onto VictoriaLogs’ Loki-compatible insert endpoint — then Alloy and Loki can finally go.

Where it landed

#

Metrics: fully on VictoriaMetrics — 4 VMAgents, ~135k series, platform + app + external-exporter coverage, NFS-backed. Mimir and Tempo are gone; Grafana defaults to the VictoriaMetrics datasource. Logs: a known follow-up. Sometimes a migration ends with one honest TODO instead of a fake green checkmark.