Grafana datasource errors sent to only some contact points
Problem
A Grafana alarm (Alarm1) was set up with error as failure and a label foo=bar.
Notification policy one was set up to match on foo=bar and forward to contact
point A. Notification policy two was set up to match on foo=bar and
alertname=Alarm1, and to forward to contact point B. Data source errors for
Alarm1 arrived at contact point A, but not B.
Solution 1
On data source errors Grafana sets the alertname label to DatasourceError and
the rulename label to the name of the alarm (e.g. Alarm1). Because of this the
second notification policy would fail to match the alertname condition.
Added 2024-01-17