You need to upgrade an old Kubernetes monitoring installation without losing the alerts that warn your team about failures. First identify what is installed. Prometheus collects and stores numerical measurements. Prometheus Operator manages Prometheus-related deployments from Kubernetes resources. kube-prometheus-stack is a Helm chart that packages the operator and other monitoring components together.

The chart previously named prometheus-operator became kube-prometheus-stack; the operator itself remains a component. Use the stack's upgrade notes if that is the chart you run. A standalone server, manually installed operator or vendor bundle needs its own instructions. After the upgrade, success means the expected services are still monitored and alerts still reach people, not just that a Pod is running.

Record the metrics, alerts and data you must keep

Record release name, namespace, chart/app versions, values, custom resources, Prometheus storage, retention, alert routing and recording rules. Identify which ServiceMonitors and PodMonitors are expected to be selected. Save secrets securely rather than pasting their values into a public issue or review.

  1. Read every relevant breaking-change section between the current and target chart versions.
  2. Determine how CRDs are owned and upgraded. Installing a chart does not prove existing CRDs match the controller.
  3. Render the proposed change and inspect selectors, labels, names, PVCs and security settings.
  4. Rehearse on a representative monitoring instance with known targets and alerts.
  5. Check rule evaluation, target discovery, dashboards, notification routing and retained data before accepting the upgrade.

Prove that alerts still work

A green Prometheus Pod is not sufficient. Fire a harmless test alert through the intended routing path and confirm its resolution message. Compare expected scrape targets and rule counts. A selector change can silently remove monitoring while the stack remains healthy.

Recovery considerations

Retain the old configuration and a tested data recovery path. CRD changes and application storage formats may make a simple Helm rollback insufficient. Avoid running two alerting stacks against the same receivers without deduplication planning. Upgrade notes must be read for the precise version jump.

This page is a migration review plan, not an executed chart-upgrade transcript. Keep the operator identity distinct from the bundle and the Prometheus server when selecting documentation.

Sources & further reading

  1. kube-prometheus-stack upgrade and rename notes
  2. Prometheus Operator installation
  3. Helm CRD lifecycle caveats

Spotted something that needs another look?

Help improve this page →