Grafana datasource errors sent to only some contact points


A Grafana alarm (Alarm1) was set up with error as failure and a label foo=bar. Notification policy one was set up to match on foo=bar and forward to contact point A. Notification policy two was set up to match on foo=bar and alertname=Alarm1, and to forward to contact point B. Data source errors for Alarm1 arrived at contact point A, but not B.

Solution 1

On data source errors Grafana sets the alertname label to DatasourceError and the rulename label to the name of the alarm (e.g. Alarm1). Because of this the second notification policy would fail to match the alertname condition. Added 2024-01-17