jertel / elastalert2 Goto Github PK

ElastAlert 2 is a continuation of the original yelp/elastalert project. Pull requests are appreciated!

Home Page: https://elastalert2.readthedocs.org

License: Apache License 2.0

Python 99.55% Makefile 0.06% Dockerfile 0.09% Mustache 0.29%

elastalert2's Introduction

ElastAlert 2

ElastAlert 2 is a standalone software tool for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch and OpenSearch.

ElastAlert 2 is backwards compatible with the original ElastAlert rules.

Docker and Kubernetes

ElastAlert 2 is well-suited to being run as a microservice, and is available as an image on Docker Hub and on GitHub Container Registry. For more instructions on how to configure and run ElastAlert 2 using Docker, see here.

A Helm chart is also included for easy configuration as a Kubernetes deployment.

Documentation

Documentation, including an FAQ, for ElastAlert 2 can be found on readthedocs.com. This is the place to start if you're not familiar with ElastAlert 2 at all.

Elasticsearch 8 support is documented in the FAQ.

The full list of platforms that ElastAlert 2 can fire alerts into can be found in the documentation.

Contributing

Please see our contributing guidelines.

Security

See our security policy for reporting urgent vulnerabilities.

License

ElastAlert 2 is licensed under the Apache License, Version 2.0.

elastalert2's People

Contributors

Stargazers

Watchers

Forkers

iciouss savval dilaverdemirel amarjitghuman mostafahussein marcoacarvalho rpisani5c drakaru nsano-rururu vaibhavmehtaa nashrah31 fberrez archfz beirtipol mrfroggg ruff4 lazydk vbisserie genothomas metanovii ferozsalam niksteff stefan-caraiman markus-nclose rpotnuru daiwei233 lepouletsuisse jeffashton brightspace konstantin-kornienko pandvan randolph-esnet barisgece christophshyper gscsdlz antoineblaud shashank-peritus neuro-hsoc strikesmash amolde perceptron01 jeffkwasha openevan commjoen gygygyy bmbbms k0walskl thican jonyjalfon2204 asyncnetwork thumbtack williamwon9 white-knight-it yinhejianke nickbabkin kautik-aivid novaksam liuxingjun cr7258 maxboone nbrownus saiyoofan xavigpich mdavyt92 devopsdotmy wvru sczlog vpedosyuk fwalloe proxxious navokus ns-bwhitaker hugefarsen buratinopy linhthieuza dequis zainularfeen ns-sbhesaniya notify-events roman-tasi hasecuritysolutions-docker bradleyjkemp tjnh05 leonid-granulate cyberdefense-repo ktlcove unravelin sirudog linuxwind paulswith kyle-larry thib12 th12380923 jmorets21sec maduggan oschvr flaunderg omgitsroy tankmek chrisfraun

elastalert2's Issues

WARNING:apscheduler.scheduler:Execution of job "Rule: Alert on any error (trigger: interval[0:01:00], next run at: 2021-10-08 11:01:06 UTC)" skipped: maximum number of running instances reached (1)

WARNING:apscheduler.scheduler:Execution of job "Rule: Alert on any error (trigger: interval[0:01:00], next run at: 2021-10-08 11:01:06 UTC)" skipped: maximum number of running instances reached (1)
i am getting following. can anyone please help me out as i have not found anything related to this error.

WARNING:py.warnings:/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host "<elastic-host>"

The elastalert supports the "ssl_show_warn" flag to suppress warnings when it sends alerts to various targets but , as far as I can understand the elastalert doesn't support this flag when it polls the elasticsearch .

In many cases the elastalert2 is deployed ont he same cluster with ELK and polls the elastic via kubernetes internal names, therefore can't verify the TLS certificates. As result the log is blown with InsecureRequestWarning warnings.

Jinja template is not updated when using an alert-specific `alert_text`

To reproduce, in rule.yml add the following parameters:

alert_text_type: alert_text_jinja
alert_text: "Test"
alert:
  - slack:
      alert_text_type: alert_text_jinja
      alert_text: |
        An abnormal number ({{spike_count}}) of events occurred.

This is done as described in the documentation
You will then notice that your alert on Slack will not use the relevant Jinja template when using the FileRulesLoader.

The most likely cause is that the Jinja template is not recompiled as in https://github.com/jertel/elastalert2/blob/master/elastalert/loaders.py#L469, and therefore a wrong template (the one using the alert_text at the root of the config) is used.

how to get the number of matches from the rule file and sent to the alert？

How to remove _id, _index, _type, num_matches, num_hits and @timestamp from elastialert rules

Hi All,

Below is the content which im getting in my gmail inbox. I want to

remove _id, _index, _type, num_matches, num_hits and @timestamp

all this only i want message to be include in my email triggering everytime.

Below is getting into gmail inbox

@timestamp: 2021-05-24T11:16:06Z
_id: 2S0WnnkBz7SOxaiw1TZk
_index: logstash-2021.05.24
_type: _doc
message: <30>May 24 11:16:06 fx-prod-1 prod_fx-control-plane.1.knel5yam 2021-05-24 11:16:06.926 INFO 1 --- [nio-8080-exec-9] com.fxlabs.fxt.rest.run.RunController : Find Latest by job id [8a8089ba777311370177734530902ec8] org [8a8081066e02d6a2016e04eacd2005c7] principal [8a808155647d283a01647d7c5e0d07ba]
num_hits: 8
num_matches: 1

and

Here is the rules.yaml file which i used to trigger email for issue

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
# es_host: elasticsearch.example.com

es_host: elasticsearch.test.com
# (Optional)
# Elasticsearch port
#es_port: 14900
es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
es_username: testelastic
es_password: xxxx


# (Required)
# Rule name, must be unique
name: Exception Alert

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
#type: blacklist
type: any
include: ["message"]

# (Required)
# Index to search, wildcard supported
#index: logstash*
index: filebeat*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
#num_events: 1

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
#  hours: 1
#  minutes: 1
  seconds: 1

#compare_key: "message"

#blacklist:

#- "error"

realert:
  minutes: 5

 # This is send out all matches in one email
# aggregation:
#  minutes: 1

#  seconds: 0
#filter:
#- term:
#    message: "[error]"

#filter:
# - match:
 #   message: "job"

filter:
 - query:
      query_string:
 #       query: "message: exception AND  message: control"
        query: "message: job"
#filter:
#- query:
#   query_string:
#    query: "message: error"

alert:

- "slack"
- "email"


#- slack
slack_webhook_url: "https://hooks.slack.com/services/xxxxxxxx"
#slack_username_override: "ElastAlert"
slack_username_override: "Mohd Rashid"

#- email
email: ["[email protected]"]
smtp_host: "smtp.gmail.com"
smtp_port: "587"
from_addr: "[email protected]"

How to do it, Please do let me ASAP

Fatal Python error: Cannot recover from stack overflow.

It happens here:

    try:
        if rule.get('scroll_id') and self.thread_data.num_hits < self.thread_data.total_hits and should_scrolling_continue(rule):
            if not self.run_query(rule, start, end, scroll=True):
                return False
    except RuntimeError:
        # It's possible to scroll far enough to hit max recursive depth
        pass

This seems like whomever coded this was aware that this could happen, but i don't believe any kind of exception handling can deal with a stack overflow.

I don't have a reproduction unfortunately. This has happened once on an otherwise static setup. Of note is that my elastic instance can be quite slow at times.

Elasticsearch host error

Hi, I have an elk setup on GKE and I'm trying to use elastalert for sending notifications on error logs. I'm new to elastalert, I just followed your README doc on this URL https://github.com/jertel/elastalert2/tree/master/chart/elastalert2. after deploying the elastalert2 helm chart, elastalert pod is failing to connect with the existing elasticsearch deployment.

I did not change any of the values from values.yaml except for rules, because the existing elasticsearch deployment's service name is elasticsearch itself and the port is 9200 which are same as values.yaml .

the error log from elastalert pod is:

Traceback (most recent call last):
File "/usr/local/bin/elastalert-create-index", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/elastalert/create_index.py", line 218, in main
host = args.host if args.host else input('Enter Elasticsearch host: ')
EOFError: EOF when reading a line
Enter Elasticsearch host:

Could you tell me what I'm doing wrong here?

thanks in advance.

2.2.0 Crash on Startup

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 2142, in main
0 rules loaded
    client = ElastAlerter(args)
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 170, in __init__
    self.pretty_ts_format = self.conf['custom_pretty_ts_format']
KeyError: 'custom_pretty_ts_format'

Looks like this is something with #407

Elastalert2 High Available

Hi:

I may be use it in production env, but I didn't find any high availability solution in the documentation,relying on a single instance of ElastAlert is not really good enough in my opinion.

Thanks and kind regards,
Vincent.

elastalert-create-index: Mapper for [@timestamp] conflicts with existing mapper

$ elastalert-create-index 
Elastic Version: 7.14.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
Traceback (most recent call last):
  File "/opt/elastalert2/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
  File "/opt/elastalert2/lib/python3.9/site-packages/elastalert/create_index.py", line 273, in main
    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
  File "/opt/elastalert2/lib/python3.9/site-packages/elastalert/create_index.py", line 63, in create_index_mappings
    es_client.indices.put_mapping(index=ea_index, doc_type='_doc',
  File "/opt/elastalert2/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/opt/elastalert2/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 320, in put_mapping
    return self.transport.perform_request(
  File "/opt/elastalert2/lib/python3.9/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/opt/elastalert2/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 91, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/opt/elastalert2/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 131, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Mapper for [@timestamp] conflicts with existing mapper:\n\tCannot update parameter [format] from [strict_date_optional_time||epoch_millis] to [dateOptionalTime]')

Fixed index mappings like this:

    "@timestamp": {
      "type": "date"
    }

deleted options:

      "format": "dateOptionalTime"

Send recovery alerts

when the problem that triggered the alert has recovered,Whether to support sending a "resolve" message ?

Support count event per query_key

Dears,

Good news with this new elastalert2

Is it possible to have the count of event per query_key ( compound_key too )
There is the num_events but not the exact the count ( 20 in my example ) and sometimes not enough

here is my rules with alerta (work fine )
I tried with top_cout_keys but it didn't work with coumpound_key

# Index to search, wildcard supported
index: production_logs

# rule name, must be unique
name: ProductionLogs | Erreur java Exception Majeur

# Type of alert
type: frequency
timeframe:
 minutes: 5
num_events: 20

# A list of elasticsearch filters used to find events
filter:
 - query:
     query_string:
           query: tags:as AND javaexception:"java.sql.SQLException"

query_key: ["host","application","applog","doctype","javaclass","javaexception","req_alerta"]

# Anti spam
realert:
  minutes: 5

# Alert is sent when a match is found
alert:
- alerta
alerta_api_url: "http://XXXXXXXXXXXXXXXXXXX/api/alert"
alerta_customer: "%(application)s"
alerta_environment: "%(application)s"
alerta_service: ["%(application)s"]
alerta_resource: "%(applog)s %(doctype)s %(javaexception)s"
alerta_event: "%(req_alerta)s major"
alerta_text: "%(applog)s %(doctype)s %(javaclass)s %(req_alerta)s"
alerta_severity: "major"
alerta_tags: ["%(javaexception)s","exploitation","logs"]
alerta_value: "%(host)s %(javaclass)s %(javaexception)s"
alerta_api_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx'
alerta_timeout: 259200
alerta_use_match_timestamp: true

w-HA

alertmanager alert can't work.

configuration:
`
alert:

"alertmanager"
alertmanager_hosts:
"http://alertmanager:9093"
alertmanager_alertname: "Title"
alertmanager_annotations:
severity: "error"
alertmanager_labels:
source: "elastalert"
alertmanager_fields:
msg: "message"
log: "@log_name"
`

TheHive Configuration

I'm getting issue to get data field value when using elastalert docker version jertel/elastalert2:2.2.2
I try other version but still getting string like '{rule[name]}' instead of rule name value.
'{rule[name]}'
{match[somefield]}
or {match[_index]}?id={match[_id]}

anyone know how to solve this??

Discussed in #465

^{Originally posted by Harris-Chan September 15, 2021}
HI

I am creating thehive alert in elastalert. And I want to change the type config in hive_alert_config from 'external' to some match field like index or rule name. However, I found that it can only return the string "index" in the type. How can I do it on type while Yelp/elastalert can use match[index].

I tried {_index} or _index on that type field but its only return me the string

hive_alert_config:
  customFields:
    - name: example
      type: string
      value: example
  follow: True
  severity: 2
  status: 'New'
  source: 'elastalert'
  description: 'Sample description'
  tags: ['tag1', 'tag2']
  title: 'Title'
  tlp: 3
  type: 'external'   <------------ Change it to match field

hive_observable_data_mapping:
  - domain: agent.hostname
  - domain: response.domain
  - ip: client.ip
```</div>

USING python elastalert/elastalert.py to run this project,but got a error!

Traceback (most recent call last):
File "elastalert/elastalert.py", line 32, in
from elastalert import kibana
File "/data/elastalert2/elastalert2-master/elastalert/elastalert.py", line 32, in
from elastalert import kibana
ImportError: cannot import name 'kibana'

how can i work normally by this way

elastalert-test-rule fails with alert_text_jinja

Elastalert2 2.0.4
Python 3.9

I'm testing alert_text using jinja templates and it fails using the elastalert-test-rule:

/opt/elastalert $ elastalert-test-rule --config config.yaml --alert toto.yaml 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/elastalert/test_rule.py", line 445, in main
    test_instance.run_rule_test()
  File "/usr/local/lib/python3.9/site-packages/elastalert/test_rule.py", line 420, in run_rule_test
    hits = self.test_file(copy.deepcopy(rule_yaml), args)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.9/copyreg.py", line 95, in __newobj__
    return cls.__new__(cls, *args)
TypeError: __new__() missing 1 required positional argument: 'source'

But it doesn't crash when using elastalert --config config.yaml --rule toto.yaml --verbose.

I might not use it correctly, as I did not find the way refer to the rule parameter using jinja, as it doesn't put values in some of my field, but this might be another issue, as I do not see self.rule passed to the jinja renderer.

Part of the rule:

name: toto
index: [...]

type: metric_aggregation

query_key:
  - customer
  - customer.servername

buffer_time:
  minutes: 10

metric_agg_key: cpu.utilization
min_doc_count: 2
metric_agg_type: max
max_threshold: 20

alert_text: >+
   CPU Utilization with more than {{min_doc_count}} occurences ({{num_matches}}) in {{buffer_time}}
   and more than {{max_threshold}}% on {{_data['customer.servername']}}.

   Rule name: {{name}}

alert_text_type: alert_text_jinja
alert_subject: "{0} - CPU Utilization {1} occurences of more than {2}%"
alert_subject_args:
  - customer.servername
  - min_doc_count
  - max_threshold

Output from elastalert --config config.yaml --rule toto.yaml --verbose:

TESTSERVER - CPU Utilization 2 occurences of more than 20%

CPU Utilization with more than  occurrences (8) in  and more than % on TESTSERVER.
Rule name:

Support opensearch v1.0

AWS has recently released their version of ES as OpenSearch v1.0. This breaks the version detection being used in elastalert2, forcing pre ES v5 mode to be enabled, causing nothing to work.

AWS does have one way to resolve this, enable "compatibility mode", which must be done when you create the domain. There does not appear to be a way to change it after the fact.

phillbaker/terraform-provider-elasticsearch#218

Also, thank you for keeping this project alive.

Add alert history in elasticsearch

how to query alert history in elasticsearch ?

Test Rule question

I am running my test rule command and am getting this:

elastalert_status - {'rule_name': 'Lockout Alerting', 'endtime': datetime.datetime(2021, 9, 28, 18, 52, 47, 137084, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 9, 28, 17, 52, 11, 137084, tzinfo=tzutc()), 'matches': 76742, 'hits': 76742, '@timestamp': datetime.datetime(2021, 9, 28, 18, 54, 48, 709983, tzinfo=tzutc()), 'time_taken': 118.85222268104553}

Basically I don't know why my matches is equal to my hits. Also it isn't logical for me to have 76742 matches. Also the sample rule command is printing this out repeatedly:

silence - {'exponent': 0, 'rule_name': 'Lockout Alerting.10.100.10.83', '@timestamp': datetime.datetime(2021, 9, 28, 18, 54, 46, 435723, tzinfo=tzutc()), 'until': datetime.datetime(2021, 9, 28, 18, 55, 46, 435708, tzinfo=tzutc())}

but obviously with different IPs and timestamps, but all have the silence tag.

Can anyone help?

How can I pass elasticsearch host address

host = args.host if args.host else input('Enter Elasticsearch host: ')

I am getting this error while running elastalert on container.
Even I copied the config file on right place.
Can someone please help me to get out of this.

Error posting to ms teams: 400 Client Error - Bad Request

Hi all,
we got 400 bad request when we use the ms_teams alerter in conjunction with the prop aggregator.
The error in the logs is:

ERROR:elastalert:Error while running alert ms_teams: Error posting to ms teams: 400 Client Error: Bad Request 
for url: https://ms-team-webhook-url

The elastalert2 config is:

...
runIntervalMins: 1
bufferTimeMins: 60
alertRetryLimitMins: 2880
realertIntervalMins: 0
...
aggregation:
      minutes: 2
summary_table_fields:
  - "kubernetes.labels.app"
  - "log"
filter:
  - match:
      log: "error"
alert:
  - ms_teams
ms_teams_webhook_url: "https://ms-team-webhook-url"
ms_teams_alert_summary: "bla bla bla"
ms_teams_alert_fixed_width: False

Has anyone faced the same error?

Support Elasticsearch 8

Elasticsearch v8 no longer supports doc_type. There is likely going to be some effort need to update this project to deal with this.

query_key value is a string instead of a dict

Behavior

Method check_matches in ruletypes.py appends the query key as a string.
So match is a dict composed of simple type like string or integer.

Rules types impacted:

metric
percentage

On another side, all other rules create match from documents content. The method involved is add_data(self, data)

Example

Suppose that we store documents like this one in the index:
{'@timestamp': value, metadata': { 'ip': value} }

If we use a frequency rule on this index , the match generated will have this template :

[{**'metadata': {'ip**': "10.0.0.1"}, '@timestamp': datetime.datetime(20...o=tzutc()), '@version': '1', '_id': 'tR46i3oBTsznpu6_lwkh', '_index': 'index', '_type': '_doc'}]
If we use a metric rule, the match generated will have this template :

[{**'metadata.ip**': '10.0.0.1' , '@timestamp': datetime.datetime(20...o=tzutc())'}]

Possible Solution

Fix check_matches by converting query_key value="string1.string2.stringN" to ["string1']["string2]['stringN"]

Http Post Basic Auth credentials stored on environment variables

Hello

Since some credentials can be passed through environment variables (e.g. ElasticSearch) or files (e.g. jira creds), would it be possible to use the same for basic authorization with http posts?

From a CI/CD perspective it would be more practical to read the creds from environment variables injected into the containers. This has been mentioned on other github issues but the solutions were hacky.

Thanks

elastalert-test-rule运行规则生效，而使用python3 -m elastalert.elastalert --config config.yaml --verbose --rule rules/test_frequency.yaml命令不生效

使用elastalert-test-rule可以用


如上图使用elastalert命令不运行规则
config文件如下

rule文件如下

Helm Chart: missing "elastalert2/templates/config.yaml"

greate job so far ^^ i found a issue while testing Helm

The latest Helm chart crash with error

`helm upgrade --install elastalert2 jertel/elastalert2

Release "elastalert2" does not exist. Installing it now.
Error: template: elastalert2/templates/deployment.yaml:20:28: executing "elastalert2/templates/deployment.yaml" at <include (print $.Template.BasePath "/config.yaml") .>: error calling include: template: no template "elastalert2/templates/config.yaml" associated with template "gotpl"`

The Template File for "elastalert2/templates/config.yaml" is missing in Helm Chart also in GIT.

Spike Rule does not work properly after a restart of Elastalert

Suppose the following scenario:

You have defined a spike rule with the following config (strongly inspired from the examples):

# (Required)
# Rule name, must be unique
name: Event spike

# (Required)
# Type of alert.
# the spike rule type compares the number of events within two sliding windows to each other
type: spike

# (Required one of _cur or _ref, spike specific)
# The minimum number of events that will trigger an alert
# For example, if there are only 2 events between 12:00 and 2:00, and 20 between 2:00 and 4:00
# _ref is 2 and _cur is 20, and the alert WILL fire because 20 is greater than threshold_cur and (_ref * spike_height)
threshold_cur: 5
#threshold_ref: 5

# (Required, spike specific)
# The size of the window used to determine average event frequency
# We use two sliding windows each of size timeframe
# To measure the 'reference' rate and the current rate
timeframe:
  hours: 2

# (Required, spike specific)
# The spike rule matches when the current window contains spike_height times more
# events than the reference window
spike_height: 3

# (Required, spike specific)
# The direction of the spike
# 'up' matches only spikes, 'down' matches only troughs
# 'both' matches both spikes and troughs
spike_type: "up"

# If true, when ElastAlert 2 starts, it will always start querying at the current time
# minus the timeframe. timeframe must exist in the rule.
scan_entire_timeframe: true

Before 4 AM there were no events.
Between 4 and 5.50 AM, there were 10 events.
Between 5.50 AM and 6 AM, there were no events.
Between 6 AM and 7:50 AM there were 29 events.
Elastalert2 stops at 7.50 AM and is restarted immediately (no --start option is set).
At 7.55 AM there is one more event.

At startup, Elastalert will query the whole timeframe (= 2h), from 5.50 to 7.50 AM and fill in the current window. However, the reference window will be empty. This means that an alert will not triggered at 7.55 AM...

Do you agree that such an issue might happen?

A way should be found to fill in both the reference window and the current window when Elastalert is restarted. This could be done by adapting the definition of the SpikeRule.timeframe to be equal to the half of the rule["timeframe"]

percentage_format_string on Percentage Match rule doesn't apply to match_body

'percentage_format_string' option only applies to 'message' part of alert body. I believe this also should apply to 'percentage' value of 'match_body' so that you can use the value with 'alert_subject_args' and 'alert_text_args'

For example, one of my Percentage Match rule with 'percentage_format_string: "%.2f"' has following default alert_text:

PercentageMatch_CustomErrorHigh_XXXXXXX

Percentage violation, value: 0.05 (min: None max : 0.02) of 1950 items

@timestamp: 2021-08-03T03:28:15.758069Z
agent.name: XXXXXXX
denominator: 1950
num_hits: 40000
num_matches: 5
percentage: 0.05128205128205128

As you can see above, 'percentage_format_string' doesn't apply to 'percentage' value of 'match_body'.

If this is not intended behavior, simply adding those three lines of code to at the end of check_matches method of PercentageMatchrule class could fix this issue:

percentage_format_string = self.rules.get('percentage_format_string', None)
if percentage_format_string is not None:
match['percentage'] = percentage_format_string % (match_percentage)

Deleting the rule does not stop the alert

When i delete a rule it does not stop the alerting from that rule . The only way is to restart a elastalert

I think that was the old problem but was eventually solved on elastalert tree

[Helm] Secrets in Elastalert Helm Chart

Hello people!
Thank you for maintaining this project!

I wanted to point out the issue on the Helm Chart, that all Elasticsearch credentials are stored in a ConfigMap (and not in a Secret).

I have fixed this issue myself some half a year ago here: https://github.com/operatorequals/elastalert-helm/tree/master/templates
for my needs on a Cloud SIEM I am making (Blue-Baron).

I have not much time in my hands to make a proper PR, but as you are into maintaining Elastalert, it would be great to take ideas out of this Helm chart, that was made with security in mind.

I am really rooting for you in this one! I am using Elastalert and Praeco a lot in my SIEM builds (here is one example).

Support for match_enhancements in the helm chart

Hello,

Is there support for match_enhancements in the helm chart?

I'm trying to convert/format the @timestamp value in slack's alert body to a nicer date and time format.
Based on the documentation, I need to use Enhancements.
But that only gives me the following error in the pods logs:

Elastic Version: 7.2.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
Index elastalert already exists. Skipping index creation.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastalert/util.py", line 27, in get_module
    base_module = __import__(module_path, globals(), locals(), [module_class])
ModuleNotFoundError: No module named 'elastalert_modules'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastalert/loaders.py", line 126, in load
    rule = self.load_configuration(rule_file, conf, args)
  File "/usr/local/lib/python3.9/site-packages/elastalert/loaders.py", line 189, in load_configuration
    self.load_modules(rule, args)
  File "/usr/local/lib/python3.9/site-packages/elastalert/loaders.py", line 426, in load_modules
    enhancement = get_module(enhancement_name)
  File "/usr/local/lib/python3.9/site-packages/elastalert/util.py", line 30, in get_module
    raise EAException("Could not import module %s: %s" % (module_name, e)).with_traceback(sys.exc_info()[2])
  File "/usr/local/lib/python3.9/site-packages/elastalert/util.py", line 27, in get_module
    base_module = __import__(module_path, globals(), locals(), [module_class])
elastalert.util.EAException: Could not import module elastalert_modules.timestamp_conversion.TimestampConversion: No module named 'elastalert_modules'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 2108, in main
    client = ElastAlerter(args)
  File "/usr/local/lib/python3.9/site-packages/elastalert/elastalert.py", line 150, in __init__
    self.rules = self.rules_loader.load(self.conf, self.args)
  File "/usr/local/lib/python3.9/site-packages/elastalert/loaders.py", line 134, in load
    raise EAException('Error loading file %s: %s' % (rule_file, e))
elastalert.util.EAException: Error loading file /opt/rules/test_alert.yaml: Could not import module elastalert_modules.timestamp_conversion.TimestampConversion: No module named 'elastalert_modules'

My helm directory looks like this:

elastalert2
├── Chart.yaml
├── elastalert_modules
│   ├── __init__.py
│   └── timestamp_conversion.py
├── README.md
├── templates
│   ├── config.yaml
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── NOTES.txt
│   ├── podsecuritypolicy.yaml
│   ├── rolebinding.yaml
│   ├── role.yaml
│   ├── rules.yaml
│   └── serviceaccount.yaml
├── values-dev.yaml
├── values-prod.yaml
└── values.yaml

Thanks! :-)

Add jinja2 support to alert_subject

Right now, only alert_text supports jinja2 and this should apply to alert_subject as well and any other Alerter field that would like to use jinja2.

I'm creating a quick PR to add jinja2 support to alert_subject, but, at some point, I think we should refactor this part to use the BasicMatchString class in alerts.py for alert_subject, as the main difference, from what I see, is that, in Alerter:create_custom_title, it limits the len of the returned string.

limit_execution acts like a buffer

When an alert is triggered during the inactivity period of a rule having the limit_execution option set, elastalert sends an alert at wake up time.

However, the purpose of the limit_execution option is to disable the alert, not doing the same thing as aggregation.

example:

limit_execution: * 7-22 * * *

A valid event occurs at 01:05am. Therefore, at 7:00am, an alert is sent by elastalert saying:

an alert occured at 01:05am....

Some extraConfigOptions not working in helm chart

I have this helm config:

elasticsearch:
  extraConfigOptions:
    # Common slack alert settings
    generate_kibana_discover_url: True
    kibana_discover_app_url: "https://kibana.digitalnz.org/app/discover#/"
    kibana_discover_version: "7.14"
    slack_attach_kibana_discover_url: True

However the kibana URL is not showing under my alerts.

Putting them in every rules makes the URL to show again.

According to README, those extraConfigOptions should be propagated to every rules.

Also it looks like it should work?

elastalert2/chart/elastalert2/templates/config.yaml

Lines 46 to 47 in 02b8393

    
           {{- if .Values.extraConfigOptions }} 
        
           {{ toYaml .Values.extraConfigOptions | indent 4 }}

Error on jinja render for matched documents created by python-logstash-async

We use python-logstash-async in our python services, which adds a 'type' field to the document
https://github.com/eht16/python-logstash-async/blob/master/logstash_async/formatter.py#L104

Since #282 both the match and the rule are expanded into the same function call.
They clash since they both have a field called type

TypeError: render() got multiple values for keyword argument 'type'

Support multiple directories in FileRulesLoader

Add support a list of rule_folder in FileRulesLoader.

Pass a string, like before, for a single rule_folder, pass a list to support multiple rule_folder.

This can be useful when using many rules of many types/customer and/or you uses multiple ConfigMaps in Kubernetes to load your rules.

I'll create a PR.

--patience argument: writeback_alias / self.alias not used

ElastAlert 2.1.0

This morning, I was testing the --patience argument from the elastalert command (self.args.timeout):

/opt/elastalert $ elastalert --config config.yaml --verbose --rule toto.yaml --patience seconds=60
1 rules loaded
ERROR:elastalert:Writeback alias "elastalert_alerts" does not exist, did you run `elastalert-create-index`?

I found out that the class parameter self.writeback_alias = self.conf['writeback_alias'] is not documented and is not in use anywhere except in the elastalert.ElastAlerter.wait_until_responsive:

        while (clock() - ref) < timeout:
            try:
                if self.writeback_es.indices.exists(self.writeback_alias):
                    return
            except ConnectionError:
                pass
            time.sleep(1.0)

        if self.writeback_es.ping():
            elastalert_logger.error(
                'Writeback alias "%s" does not exist, did you run `elastalert-create-index`?',
                self.writeback_alias,
            )

There is a default:
conf.setdefault('writeback_alias', 'elastalert_alerts')

Which is not created by create_index_mappings, the alias variable is not in used:

        index = args.index if args.index is not None else data.get('writeback_index')
        alias = args.alias if args.alias is not None else data.get('writeback_alias')

So, I wonder, should this alias be used, somewhere?
Or should we clean this and use writeback_index, instead?

[BUG] - apscheduler skipping alerts

Firstly, thanks for maintaining the project.

Elastalert version - latest
Python version - Python 3.8.5
OS - Ubuntu 20.04.1 LTS

Problem description. - This problem comes from the original elastalert. We noticed that amount of rules actually being run by Elastalert was different every time it ran - this was viewed in the Elastalert Elasticsearch index.
We never had this issue with a "small" amount of rules and only noticed it when a large set of rules was loaded.

In the Elastalert logs you would see this intermittently:

May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:03 UTC)" was missed by 0:00:02.944895
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:05 UTC)" was missed by 0:00:02.912215
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:06 UTC)" was missed by 0:00:02.827846
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:05 UTC)" was missed by 0:00:02.758194
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:07 UTC)" was missed by 0:00:02.758226
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:08 UTC)" was missed by 0:00:02.617983
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:06 UTC)" was missed by 0:00:02.407513
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:07 UTC)" was missed by 0:00:02.351592
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:05 UTC)" was missed by 0:00:02.262315
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:05 UTC)" was missed by 0:00:02.244299
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:04 UTC)" was missed by 0:00:02.242278
May 24 09:27:06 minimdr python[2498368]: WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:08:00], next run at: 2021-05-24 09:35:04 UTC)" was missed by 0:00:02.237550

We modified elastalert.py and added misfire_grace_time to job as a hack to ensure all the rules runs.
The parameter was found here : https://apscheduler.readthedocs.io/en/stable/modules/job.html

This is the result of change:

STOP! Do not create issues for support requests. Use Discussions when requesting support

If you are looking for support, please open a new discussion topic rather than creating a new issue. Others may find they have a similar question and will be able to review all discussion topics, whereas issues get closed and hidden from view. Open a new discussion topic here: https://github.com/jertel/elastalert2/discussions

docker run start error

start parameter：docker run -d -v /opt/elastalert2/config.yaml:/opt/elastalert/config.yaml jertel/elastalert2
error log like this:

How to add url link in alert_text?

I added a url link to the alarm, but an error was reported when the alarm was generated.

alert_text_type: alert_text_only
alert_text: |
  =======================
  告警规则: APM_Alert_http500
  触发时间: {}
  告警主题: {} 服务异常
  --------详细信息--------
  trace.id: {}
  error.grouping_key: {}
  error.id: {}
  <a href=\"http://http://192.168.10.148:5601/s/apm/app/apm/services/{1}/errors/{4}?rangeFrom={0}&rangeTo={0}\">跳转到kibana</a>
  =======================
alert_text_args:
  - "@timestamp"
  - service.name
  - trace.id
  - error.grouping_key
  - error.id

INFO:elastalert:Skipping writing to ES: {'exponent': 0, 'rule_name': 'tuisong.http://xxxx.com/api/passingorder, {"id":"6911a67f-1a4b-4b0e-874d-f2b08180b7cd"}', '@timestamp': '2021-08-17T08:33:32.224858Z', 'until': '2021-08-17T08:48:32.224840Z'}
INFO:elastalert:Alert for tuisong, http://xxxx.com/api/passingorder, {"id":"6911a67f-1a4b-4b0e-874d-f2b08180b7cd"} at 2021-08-17T08:32:59.730Z:
ERROR:root:Traceback (most recent call last):
  File "/usr/local/python36/lib/python3.6/site-packages/elastalert/elastalert.py", line 1450, in alert
    return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
  File "/usr/local/python36/lib/python3.6/site-packages/elastalert/elastalert.py", line 1532, in send_alert
    alerter.alert(matches)
  File "/usr/local/python36/lib/python3.6/site-packages/elastalert/alerts.py", line 409, in alert
    elastalert_logger.info(str(BasicMatchString(self.rule, match)))
  File "/usr/local/python36/lib/python3.6/site-packages/elastalert/alerts.py", line 156, in __str__
    self._add_custom_alert_text()
  File "/usr/local/python36/lib/python3.6/site-packages/elastalert/alerts.py", line 91, in _add_custom_alert_text
    alert_text = alert_text.format(*alert_text_values)
IndexError: tuple index out of range

Support env var substitution in all config.

I would like to use env vars in any yaml:

name: API Error
type: any
index: filebeat-*
filter:
    - query:
          query_string:
              query: 'event.dataset: "dotnet" AND log.level: "Error"'
alert:
    - "discord"
discord_webhook_url: $ELAST_ALERT_DISCORD_URL

This seems to not work currently. Am I missing something?

Send interger tag to thehive alerter

Hi all,

We tried to send an alert to thehive by puting in TAGS a number (integer), This didn't work.
Afrer some analysis, we saw that thehive api is waiting for only strings in TAGS. We modified elatalert code and WE FORCED to convert Tags sent to thehive to string by :

class HiveAlerter(Alerter): 
  ...
  ...
  def load_tags(self, tag_names: list, match: dict):
    tag_values = set()
    for tag in tag_names:
        tag_value = self.lookup_field(match, tag, tag)
        if isinstance(tag_value, list):
            for sub_tag in tag_value:
                tag_values.add(str(sub_tag))
        else:
            tag_values.add(str(tag_value))

    return tag_values

For sub_tag and tag_value thehive4py is waiting string values, so we forced it by using str function.

https://github.com/TheHive-Project/TheHive4py/blob/1.x/thehive4py/models.py
[class Alert(JSONSerializable)]

This works.

Could you take into account this fix for the futur elastalert release please ?

Thank you very much

Charaf B
SSS

Support derivative aggregations on alerting for incremental data

Given metricbeat and it's network metricset we receive the total number of bytes in and out for each document. There is not delta provided. Unfortunately metricbeat is not going to implement non incremental value for this metricset: elastic/beats#2783

Now from what I have read the derivative aggregation would be required to be able to set a rule that says alert if this many bytes per second we have in the last timeperiod. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html#search-aggregations-pipeline-derivative-aggregation

See example document system.network.in.bytes and system.network.out.bytes. These fields are only increasing from doc to doc, they don't represent deltas, they represent totals. Now from the possible rules for elastalert2 as I see none of them fit to analyse for this type of data.

{
  "_index": "metricbeat-7.11.2-2021.05.06-000006",
  "_type": "_doc",
  "_id": "N8lhS3kBTvEe6jk05x8z",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-05-08T09:49:36.968Z",
    "system": {
      "network": {
        "name": "eth2",
        "in": {
          "bytes": 799223433,
          "packets": 618846,
          "errors": 0,
          "dropped": 0
        },
        "out": {
          "dropped": 0,
          "packets": 618255,
          "bytes": 65183723,
          "errors": 0
        }
      }
    },
    "ecs": {
      "version": "1.7.0"
    },
    "host": {
      "ip": [
        "172.18.0.2",
        "172.19.0.3",
        "172.20.0.10"
      ],
      "mac": [
        "02:42:ac:12:00:02",
        "02:42:ac:13:00:03",
        "02:42:ac:14:00:0a"
      ],
      "hostname": "10ea8b886435",
      "architecture": "x86_64",
      "os": {
        "version": "10 (buster)",
        "family": "debian",
        "name": "Debian GNU/Linux",
        "kernel": "4.15.0-140-generic",
        "codename": "buster",
        "platform": "debian"
      },
      "id": "888b2975747fbae801ada88be3481c09",
      "containerized": true,
      "name": "master-proxy"
    },
    "agent": {
      "ephemeral_id": "4bdc750f-16df-4714-b4b9-122f31326573",
      "id": "878038ed-5646-4915-9447-540c8a8529e6",
      "name": "master-proxy",
      "type": "metricbeat",
      "version": "7.11.2",
      "hostname": "10ea8b886435"
    },
    "event": {
      "duration": 793471,
      "dataset": "system.network",
      "module": "system"
    },
    "metricset": {
      "name": "network",
      "period": 10000
    },
    "service": {
      "type": "system"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-05-08T09:49:36.968Z"
    ]
  },
  "highlight": {
    "metricset.name": [
      "@kibana-highlighted-field@network@/kibana-highlighted-field@"
    ],
    "system.network.name": [
      "@kibana-highlighted-field@eth2@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1620467376968
  ]
}

Readme out of date

In newest image /opt/config/elastalert_config.yaml needs to be mounted. The readme says otherwise.

Support for Arm architectures

I have a kubernetes cluster running ElasticSearch 7.14 and would really like to use ElastAlert2 to monitor it. It should be as simple as adjusting your build pipeline to build for multi architectures:

docker buildx build --platform linux/amd64,linux/arm64 -t jertel/elastalert:2.3.1 . --push

This command will take quite a bit longer, so not desirable during development iterations, just release.
I cloned the repo and ran it myself, and it seems to work fine.

Elastalert error out when used with a bearer token

I'm trying to deploy elasrtalert2 version 2.1.0 on OCP 4.6 to integrate with OpenShift's out of the box EFK stack. I used the 'jertel/elastalert2:latest' image.

I created a dedicated service account for Elastalert. Then added it to ./sgconfig/roles_mapping.yml. However, the elastalert gave the following error.

elastalert-create-index --config <config> --index <index_name>

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/elastalert/create_index.py", line 253, in main
    es = Elasticsearch(
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/client/__init__.py", line 206, in __init__
    self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 108, in __init__
    self.set_connections(hosts)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 157, in set_connections
    connections = list(zip(connections, hosts))
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/transport.py", line 154, in _create_connection
    return self.connection_class(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/connection/http_requests.py", line 39, in __init__
    self.session.headers.setdefault('content-type', 'application/json')
AttributeError: 'str' object has no attribute 'setdefault'

The elastalert config looks like below:

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /opt/elastalert/rules

#
scan_subdirectories: false

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch service for metadata writeback
# Note that every rule can have it's own elasticsearch host
es_host: elasticsearch.openshift-logging.svc.cluster.local

# The elasticsearch port
es_port: 9200

es_bearer: <token>


# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

use_ssl: True
verify_certs: True

# The secrets from the fluent must be mounted!
ca_certs: <ca_cert>
#client_cert:
#client_key:

'search_phase_execution_exception', 'No mapping found for [@timestamp] in order to sort on'

got an error since today, all was working fine before, the timestamp in ES7 is shift of 10h but its working

Add alert_text support for HTTP POST

Original issue:
Yelp/elastalert#2083

Currently, there is no option to send alert_text with http post:
Neither via http_post_payload, nor via http_post_static_payload.

As a result, it is unable to generate arbitrary message string (with variables).
Please add alert_text support for HTTP POST.

Add self.rule parameters to jinja2 Template

Currently, only the match (self.match) dictionary is accessible in alert_text when using alert_text_type: alert_text_jinja.

As we can access self.rule dictionary using alert_text_args, adding the rule content to alert_text_jinja would make a lot of sense.

	{{- if .Values.extraConfigOptions }}
	{{ toYaml .Values.extraConfigOptions \| indent 4 }}