Skip to main content

Alert Notification Channels

Introduction​

Alert notification channels are the pathways through which Prometheus and Alertmanager deliver alert notifications to end-users or systems. When an alert is triggered, Alertmanager processes it and sends notifications through configured channels to ensure the right people or systems are informed about the issue.

In this guide, we'll explore different notification channels available in Alertmanager, how to configure them, and best practices for implementing an effective notification strategy.

Understanding Alert Notification Flow​

Before diving into specific channels, let's understand how alerts flow through the Prometheus ecosystem:

  1. Prometheus evaluates alert rules against collected metrics
  2. When alert conditions are met, Prometheus fires an alert to Alertmanager
  3. Alertmanager processes the alert according to its configuration
  4. Alertmanager routes the alert to appropriate notification channels
  5. The notification is delivered to end-users or systems

Configuring Alertmanager​

Alertmanager configuration is done through a YAML file. Here's a basic structure that includes notification channels:

global:
  resolve_timeout: 5m
  # Default notification channel configurations

route:
  group_by: ['alertname', 'job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default-receiver'
  
  routes:
    - matchers:
        - severity="critical"
      receiver: 'team-critical'
      continue: true

receivers:
  - name: 'default-receiver'
    email_configs:
      - to: '[email protected]'
  
  - name: 'team-critical'
    slack_configs:
      - channel: '#critical-alerts'
        api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'

Common Notification Channels​

Email Notifications​

Email is one of the most basic notification channels. It's easy to set up and widely accessible.

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.example.com:587'
        auth_username: 'alertmanager'
        auth_password: 'password'
        auth_identity: '[email protected]'
        auth_secret: 'password'
        send_resolved: true
        html: '{{ template "email.default.html" . }}'
        headers:
          subject: '{{ template "email.default.subject" . }}'

Slack Integration​

Slack is popular for team-based alert notifications due to its real-time communication capabilities.

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - channel: '#alerts'
        api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'
        send_resolved: true
        icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'
        actions:
          - type: 'button'
            text: 'Runbook 📚'
            url: '{{ (index .Alerts 0).Annotations.runbook }}'

PagerDuty​

PagerDuty is ideal for critical alerts that require immediate attention and on-call management.

receivers:
  - name: 'pagerduty-notifications'
    pagerduty_configs:
      - service_key: 'your_pagerduty_service_key'
        send_resolved: true
        description: '{{ template "pagerduty.default.description" . }}'
        details:
          severity: '{{ .CommonLabels.severity }}'
          summary: '{{ .CommonAnnotations.summary }}'
          runbook_url: '{{ .CommonAnnotations.runbook }}'

Webhook​

Webhooks allow for custom integrations with other systems or services.

receivers:
  - name: 'webhook-notifications'
    webhook_configs:
      - url: 'http://example.com/webhook'
        send_resolved: true
        http_config:
          bearer_token: 'token'

OpsGenie​

OpsGenie provides advanced alert management with escalation policies and on-call scheduling.

receivers:
  - name: 'opsgenie-notifications'
    opsgenie_configs:
      - api_key: 'your_opsgenie_api_key'
        message: '{{ template "opsgenie.default.message" . }}'
        description: '{{ template "opsgenie.default.description" . }}'
        source: 'prometheus'
        tags: '{{ range .CommonLabels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}'
        note: 'Alert triggered by Prometheus'

Advanced Configuration​

Templating Notifications​

Alertmanager supports templating to customize the content of notifications:

templates:
  - '/etc/alertmanager/templates/*.tmpl'

Example template file (/etc/alertmanager/templates/custom.tmpl):

{{ define "slack.custom.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.custom.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
*Started:* {{ .StartsAt | since }}
{{ end }}
{{ end }}

Usage in configuration:

receivers:
  - name: 'slack-custom'
    slack_configs:
      - channel: '#alerts'
        api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'
        title: '{{ template "slack.custom.title" . }}'
        text: '{{ template "slack.custom.text" . }}'

Routing and Grouping​

Routing ensures alerts reach the right teams, while grouping prevents notification storms:

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # Wait time to buffer alerts of same group
  group_interval: 5m     # Wait time before sending a new alert for group
  repeat_interval: 4h    # Minimum time between re-sending 
  receiver: 'default-receiver'
  
  routes:
    - matchers:
        - severity="critical"
      receiver: 'team-sre'
      routes:
        - matchers:
            - service="database"
          receiver: 'database-team'
    
    - matchers:
        - severity="warning"
      receiver: 'team-devs'

Implementing Notification Channels: Step-by-Step Example​

Let's walk through setting up a complete notification system:

  1. First, create an Alertmanager configuration file:
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'password'
  slack_api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'

templates:
  - '/etc/alertmanager/templates/*.tmpl'

route:
  group_by: ['alertname', 'job', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'email-team'
  
  routes:
    - matchers:
        - severity="critical"
      receiver: 'slack-sre'
      continue: true
    
    - matchers:
        - service=~"api|frontend"
      receiver: 'team-app'
    
    - matchers:
        - service="database"
      receiver: 'team-db'

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'cluster', 'service']

receivers:
  - name: 'email-team'
    email_configs:
      - to: '[email protected]'
        send_resolved: true
  
  - name: 'slack-sre'
    slack_configs:
      - channel: '#sre-alerts'
        send_resolved: true
        title: '{{ template "slack.default.title" . }}'
        text: '{{ template "slack.default.text" . }}'
  
  - name: 'team-app'
    slack_configs:
      - channel: '#app-alerts'
        send_resolved: true
    email_configs:
      - to: '[email protected]'
        send_resolved: true
  
  - name: 'team-db'
    pagerduty_configs:
      - service_key: 'db_team_pagerduty_key'
        send_resolved: true
    email_configs:
      - to: '[email protected]'
        send_resolved: true
  1. Create a template file for customized notifications:
# /etc/alertmanager/templates/custom.tmpl

{{ define "slack.custom.title" }}
[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.custom.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Started:* {{ .StartsAt.Format "2006-01-02 15:04:05" }}
*Metrics:* `{{ .GeneratorURL }}`
{{ end }}
{{ end }}
  1. Start Alertmanager with this configuration:
alertmanager --config.file=/etc/alertmanager/alertmanager.yml

Real-World Examples​

Notification Channel for Different Environments​

route:
  receiver: 'default'
  routes:
    - matchers:
        - environment="production"
      receiver: 'production-alerts'
    - matchers:
        - environment="staging"
      receiver: 'staging-alerts'
    - matchers:
        - environment="development"
      receiver: 'development-alerts'

receivers:
  - name: 'default'
    email_configs:
      - to: '[email protected]'
  
  - name: 'production-alerts'
    pagerduty_configs:
      - service_key: 'prod_pagerduty_key'
    slack_configs:
      - channel: '#prod-alerts'
        api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'
  
  - name: 'staging-alerts'
    slack_configs:
      - channel: '#staging-alerts'
        api_url: 'https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZZ'
  
  - name: 'development-alerts'
    email_configs:
      - to: '[email protected]'

Escalation-Based Notification System​

route:
  receiver: 'default'
  routes:
    - matchers:
        - severity="critical"
      receiver: 'critical-alerts'
      continue: true
      routes:
        - matchers:
            - service="database"
          receiver: 'database-critical'
    
    - matchers:
        - severity="warning"
      receiver: 'warning-alerts'

receivers:
  - name: 'default'
    email_configs:
      - to: '[email protected]'
  
  - name: 'critical-alerts'
    pagerduty_configs:
      - service_key: 'critical_pagerduty_key'
    slack_configs:
      - channel: '#critical-alerts'
  
  - name: 'warning-alerts'
    slack_configs:
      - channel: '#warning-alerts'
  
  - name: 'database-critical'
    pagerduty_configs:
      - service_key: 'db_pagerduty_key'
    slack_configs:
      - channel: '#db-alerts'
    email_configs:
      - to: '[email protected]'

Best Practices​

  1. Use Multiple Channels: Combine different notification methods for critical alerts to ensure they don't go unnoticed.

  2. Implement Proper Grouping: Group related alerts to prevent notification storms during large-scale incidents.

  3. Set Appropriate Timing:

    • group_wait: Initial wait time to buffer alerts (30s-1m is common)
    • group_interval: Time before sending updated notifications (5m-10m)
    • repeat_interval: Time before resending an unresolved alert (1h-4h)

  4. Provide Context: Include relevant information in alerts such as severity, affected service, and runbook links.

  5. Use Templates: Customize notifications for different channels to include the most relevant information.

  6. Design Escalation Paths: Create routes that escalate alerts based on severity or response time.

  7. Test Your Configuration: Regularly test your notification channels to ensure they work as expected.

  8. Document Your Setup: Keep documentation on your alert channels, routes, and expected behaviors.

Summary​

Alert notification channels are a crucial component of an effective monitoring system. They ensure that the right people are notified about issues at the right time, through the right medium.

In this guide, we've covered:

  • The alert notification flow in Prometheus and Alertmanager
  • Common notification channels like Email, Slack, PagerDuty, and Webhooks
  • How to configure Alertmanager for routing and grouping alerts
  • Templates for customizing notification content
  • Real-world examples of notification configurations
  • Best practices for implementing notification strategies

By properly configuring notification channels, you can build a robust alerting system that balances prompt notification with preventing alert fatigue.

Additional Resources​

Exercises​

  1. Set up a basic Alertmanager configuration with email and Slack notification channels.
  2. Create a template that customizes alert notifications for different severity levels.
  3. Implement a routing configuration that sends different types of alerts to different teams.
  4. Configure a notification channel that integrates with your current ticketing system using webhooks.
  5. Design an escalation-based notification system for critical production alerts.

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)