ELK 로그 경보 플러그인 ElastAlert 및 스파이크 경보 구성

ELK 로그 경고 플러그인 ElastAlert

    Elasticsearch        （       ）    。    Elasticsearch，           ，              。     ，              ，             。
          ，          ，           。

ElastAlert                 ：
	  Y      X      ”（frequency  ）
	              ”（spike  
	 Y     X        ”（flatline  
	         /        ”（blacklist whitelist  ）
	               ”（any  ）
	                        ”（change  ）
	                  ”（new_term  ）
	                （cardinality  ）   
	
      、  、  、          ；    es        

python3.6 설치

tar xf Python-3.6.8.tar.xz
yum -y install wget sqlite-devel xz gcc automake zlib-devel openssl-devel epel-release
cd Python-3.6.8/
./configure && make && make install

mkdir -p /app/elastalert/rule

elastalert 설치

cd /app/elastalert && git clone https://github.com/Yelp/elastalert.git

cd /app/elastalert/elastalert && pip3 install -r requirements.txt 

pip3 uninstall elasticsearch
pip3 install "elasticsearch>=5.0.0"
#    elasticsearch   ,elasticsearch6        pip      elasticsearch ，       pip3 install "elasticsearch>=5.0.0"  ，         requirements.txt  elasticsearch  

python3 setup.py install

elastalert 구성
cp/app/elastalert/elastalert/config.yaml.example/app/elastalert/elastalert/config.yaml
필요에 따라 수정하면 된다

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /app/elastalert/rule

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host:    es   IP

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 1

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#    handlers:
#      console:
#        class: logging.StreamHandler
#        formatter: logline
#        level: DEBUG
#        stream: ext://sys.stderr
#
#      file:
#        class : logging.FileHandler
#        formatter: logline
#        level: DEBUG
#        filename: elastalert.log
#
#    loggers:
#      elastalert:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch.trace:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      '':  # root logger
#        level: WARN
#          handlers:
#            - console
#            - file
#        propagate: false

#    
rules_folder:	 ElastAlert             。             .yaml  。        ，ElastAlert     。            ，ElastAlert       ，                    
run_every:	 ElastAlert      Elasticsearch   
buffer_time:	              ，   45  
Es_host:	elasticsearch host  
Es_port:	elasticsearch      
writeback_index:	 ElastAlert              
writeback_alias:	  
alert_time_limit:	          

구성이 완료되면 elastalert-create-index --config config를 실행합니다.yaml
스파이크 경보 플러그인 설치

wget https://github.com/xuyaoqiang/elastalert-dingtalk-plugin/archive/master.zip
unzip elastalert-dingtalk-plugin-master.zip
cd elastalert-dingtalk-plugin-master
pip3 install pyOpenSSL==16.2.0
pip3 install setuptools==46.1.3

cp -r elastalert_modules /app/elastalert/

규칙 범례

#   example_rules /             。
example_spike.yaml
 “  ”       ，                              。    2                 2        3  ，            。

example_frequency.yaml
 “  ”       ，                       。     4     50                   。

example_change.yaml
 “  ”       ，                ，      。     ，          “   ”   “ country_name”       ，  24            。

example_new_term.yaml
 “   ”       ，                    ，      。     ，            （“   ”，“   ”） ，         。

경고 규칙 구성
nginx 5XX 상태를 검사하여 1분 이내에 5회 이상 못을 박아 경고합니다

cat /app/elastalert/rule/nginx.yaml

name: the count of servnginx log that reponse status code is 5xx is greater than 5 in the period 1 minute

index: nginx-*
type: frequency
num_events: 5
timeframe: {minutes: 1}

filter:
- range:
    status:
       from: 500
       to: 599


alert_text: "
      : {}

    : {}

    : {}

     : {}

     : {}

        : {}
"

alert_text_type: alert_text_only

alert_text_args:
- host
- method
- request
- status
- upstream
- num_hits

alert: 
- "elastalert_modules.dingtalk_alert.DingTalkAlerter"
dingtalk_webhook: "XXXXXX"
dingtalk_msgtype: "text"

#    
name:	         。             ， ElastAlert     
type:	           ，         。 frequency      “ num_events           timeframe
index:	         
num_events:	      frequency  ，           。
timeframe:	 num_events        。
filter:	        Elasticsearch     
alert_text:             
alert_text_args:   alert_text   
#    
https://elastalert.readthedocs.io/en/latest/recipes/writing_filters.html#writingfilters

alert:	               
#    
https://elastalert.readthedocs.io/en/latest/ruletypes.html#alerts

테스트 규칙

elastalert-test-rule example_rules/my_rule.yaml

디버그 실행

/app/elastalert/bin/python3 -m elastalert.elastalert --verbose --rule /app/elastalert/rule/nginx.yaml

생산 운행
공식적으로 슈퍼비저로 시작하자고 제안했는데 테스트를 할 때 설정을 읽지 못하고 포기했어요.

nohup /app/elastalert/bin/python3 -m elastalert.elastalert --config /app/elastalert/elastalert/config.yaml --verbose >>/app/elastalert/nohup.out 2>&1 &

참조 블로그
https://www.cnblogs.com/opesn/p/12994199.html