Elastalert

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch.

ElastAlert works with all versions of Elasticsearch.

At Yelp, we use Elasticsearch, Logstash, and Kibana for managing our ever-increasing amount of data and logs. Kibana is great for visualizing and querying data, but we quickly realized that it needed a companion tool for alerting on inconsistencies in our data. Out of this need, ElastAlert was created.

If you have data being written into Elasticsearch in near real time and want to be alerted when that data matches certain patterns, ElastAlert is the tool for you. If you can see it in Kibana, ElastAlert can alert on it.

Requirements

• Elasticsearch
• ISO8601 or Unix timestamped data
• Python 2.7
• pip, see requirements.txt
• Packages on Ubuntu 14.x: python-pip python-dev libffi-dev libssl-dev

Overview

It works by combining Elasticsearch with two types of components, rule types and alerts. Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match.

• This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts.
• Several rule types with common monitoring paradigms are included with ElastAlert:
• Match where there are at least X events in Y time” (frequency type)
• Match when the rate of events increases or decreases” (spike type)
• Match when there are less than X events in Y time” (flatline type)
• Match when a certain field matches a blacklist/whitelist” (blacklist and whitelist type)
• Match on any event matching a given filter” (any type)
• Match when a field has two different values within some time” (change type)
• Match when a never before seen term appears in a field” (new_term type)
• Match when the number of unique values for a field is above or below a threshold (cardinality type)

Currently, we have built-in support for the following alert types:

• Email
• JIRA
• OpsGenie
• Commands
• HipChat
• MS Teams
• Slack
• Telegram
• AWS SNS
• VictorOps
• PagerDuty
• Exotel
• Twilio
• Gitter

Additional rule types and alerts can be easily imported or written.

In addition to this basic usage, there are many other features that make alerts more useful:

• Alerts link to Kibana dashboards
• Aggregate counts for arbitrary fields
• Combine alerts into periodic reports
• Separate alerts by using a unique key field
• Intercept and enhance match data