Reducing noise: Configuring Alert Processing with Terraform
.png)
With increasing numbers of alerts, keeping focus on the important and most critical alerts proves to be more and more of a challenge. A reduction of alert noise, meaning the prevention of too many created alerts and any kind of user notifications, is needed to ensure efficient alert response. While a detailed explanation of this topic is given in this blog post, a flexible and automated setup for your relevant resources can be achieved with Terraform using the ilert Terraform provider.
Resource creation via Terraform
For this example, we will create a Grafana alert source while referencing another, already created escalation policy with a data source.
We can route incoming events, depending on their title, to different escalation policies by defining a routing template to set a routing key.
Setting up support hours helps to prevent notifications outside of working hours while motivating actions during those hours by keeping high alert priority.
Assign these support hours to the alert source and set this alert priority rule.
You can further create a mapping for the resulting alert priority by defining a template and mappings depending on event keys, such as “state.”
A flexible and efficient way to filter out incoming events is the Event filter. It allows for complex conditions and checks in many event fields and properties.
Another simple but effective way to reduce noise is alert grouping, either via time-based or AI-based grouping.
Lastly, an auto-resolution timer may be set to prevent “dead” alerts (alerts in a “pending” or “accepted” state over a long period of time).
The full script with all resources from above can be found in our public terraform playground.
