Better slack notifications for Grafana
While the introduction of Grafana’s unified alerting system is an improvement, everything isn’t perfect. Slack notifications sent by Grafana whenever an alert is fired or resolved got significantly less readable.
Let’s improve them!
Out of the box, Slack notifications look like this:
With a single alert, the amount of information thrown at us already feels overwhelming. Even more so with real alerts for which the values field can contain a lot of unreadable data.
Fortunately, Grafana supports defining custom message templates to be used in contact points.
The following template generates notifications like this one:
To use it, create a new template with the following content (its name is irrelevant):
1{{ define "custom_alert.title" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}: {{ .Alerts.Firing | len }}{{ if gt (.Alerts.Resolved | len) 0 }}, RESOLVED: {{ .Alerts.Resolved | len }}{{ end }}{{ end }}]{{ if gt (len .GroupLabels) 0 }} Grouped by: {{ range .CommonLabels.SortedPairs }}{{ .Name }}: {{ .Value }}{{ end }}{{ end }}{{ end }}
2{{ define "__text_alert_name" }}{{ range .Labels.SortedPairs }}{{ if eq .Name "alertname" }}{{ .Value }}{{ end }}{{ end }}{{ end }}
3{{ define "__text_alert_summary" }}{{ range .Annotations.SortedPairs }}{{ if eq .Name "summary" }}{{ .Value }}
4{{ end }}{{ end }}{{ end }}
5{{ define "__text_alert_description" }}{{ range .Annotations.SortedPairs }}{{ if eq .Name "description" }}{{ .Value }}{{ end }}{{ end }}{{ end }}
6{{ define "__text_alert_runbook_url" }}{{ range .Annotations.SortedPairs }}{{ if eq .Name "runbook_url" }}
7:bookmark_tabs: <{{ .Value }}|Playbook>{{ end }}{{ end }}{{ end }}
8{{ define "__text_alert_firing_item" }}:bell: {{ template "__text_alert_name" . }}
9{{ template "__text_alert_summary" . }}{{ template "__text_alert_description" . }}
10
11Labels: {{ range .Labels.SortedPairs }}
12{{- if ne .Name "alertname" }}
13{{- if ne .Name "ref_id" }}
14{{- if ne .Name "datasource_uid" }}
15{{- if ne .Name "rule_uid" }}
16- {{ .Name }} = {{ .Value }}
17{{- end }}
18{{- end }}
19{{- end }}
20{{- end }}
21{{- end }}
22
23Actions:
24{{ if gt (len .DashboardURL) 0 }}:grafana: <{{ .DashboardURL }}|Go to dashboard>{{ end }}
25{{ if gt (len .PanelURL) 0 }}:chart_with_upwards_trend: <{{ .PanelURL }}|Go to panel>{{ end }}
26{{ if gt (len .GeneratorURL) 0 }}:arrow_right: <{{ .GeneratorURL }}|Go to alert>{{ end }}
27{{ if gt (len .SilenceURL) 0 }}:mute: <{{ .SilenceURL }}|Silence alert>{{ end }}{{ template "__text_alert_runbook_url" . }}{{ end }}
28{{ define "__text_alert_resolved_item" }}:large_green_circle: {{ template "__text_alert_name" . }}
29
30Labels: {{ range .Labels.SortedPairs }}
31{{- if ne .Name "alertname" }}
32{{- if ne .Name "ref_id" }}
33{{- if ne .Name "datasource_uid" }}
34{{- if ne .Name "rule_uid" }}
35- {{ .Name }} = {{ .Value }}
36{{- end }}
37{{- end }}
38{{- end }}
39{{- end }}
40{{- end }}
41
42Actions:
43{{ if gt (len .DashboardURL) 0 }}:grafana: <{{ .DashboardURL }}|Go to dashboard>{{ end }}
44{{ if gt (len .PanelURL) 0 }}:chart_with_upwards_trend: <{{ .PanelURL }}|Go to panel>{{ end }}
45{{ if gt (len .GeneratorURL) 0 }}:arrow_right: <{{ .GeneratorURL }}|Go to alert>{{ end }}{{ end }}
46
47{{ define "__text_alert_list_firing" }}{{ range . }}
48
49{{ template "__text_alert_firing_item" . }}{{ end }}{{ end }}
50
51{{ define "__text_alert_list_resolved" }}{{ range . }}
52
53{{ template "__text_alert_resolved_item" . }}{{ end }}{{ end }}
54{{ define "custom_alert.message" }}
55{{ if gt (len .Alerts.Firing) 0 }}{{ .Alerts.Firing | len }} Firing{{ template "__text_alert_list_firing" .Alerts.Firing }}{{ end }}
56
57{{ if gt (len .Alerts.Resolved) 0 }}{{ .Alerts.Resolved | len }} Resolved{{ template "__text_alert_list_resolved" .Alerts.Resolved }}{{ end }}{{ end }}
Then configure your contact points to use it:
- set the “Title” setting to
{{ template "custom_alert.title" . }} - set the “Text body” setting to
{{ template "custom_alert.message" . }}
Both options live under the “Optional Slack settings” section.
That’s it! You can now enjoy better Slack alerts from your Grafana :)