개요

사쿠라의 지식의 설명이 가장 알기 쉬웠습니다.
htp://k의 w㎇d게. 사쿠라. 아 d. jp/t ch/1336/

환경

CentOS 6.7

fluentd 0.12.20

Install+ 설정

이런 식으로 ansible로 install했습니다.
webserver에 세운 apach의 log를 logserver로 받고 있습니다.
log를 보내는 쪽도 log를 받는 쪽 모두 td-agent라고 하는 같은 service가 움직이고 있습니다.
webserver가 여러 대가 되었을 때 logserver에 log가 집약되므로, 나머지는 Elastic search를 무엇인가 넣으면 좋다고 생각합니다.

td-agent라는 user로 실행하려고 하는 permission denied가 발생하고 있었으므로, root로 실행하도록 설정 file를 재기록하고 있다.

ansible-galaxy에서 다음 역할을 사용했습니다.

geerlingguy.apache

williamyeh.fluentd

webserver.yml

# webserver playbook
- name: configure the web server
  hosts: webserver
  vars:
    - apache_listen_port: 10443
  roles:
    - geerlingguy.apache
  tasks:
    - name: copy init files
      copy: src=roles/tomcat8/files/httpd-proxy.conf dest=/etc/httpd/conf/httpd-proxy.conf owner=root group=wheel mode=0644

    - name: deploy setting file
      lineinfile: dest=/etc/httpd/conf/httpd.conf line='Include /etc/httpd/conf/httpd-proxy.conf'

    - name: restart service and auto startup setting
      service: name=httpd state=restarted

- name: configure fluentd
  hosts: webserver
  vars:
    - tdagent_conf_template: "roles/td-agent/templates/web-td-agent.conf.j2"
    - aggregator_ip: "{{ logserver.ip }}"
    - tdagent_port: "{{ logserver.tdagent_port }}"
  roles:
    - williamyeh.fluentd
  tasks:
    - name: change start user to root
      lineinfile: >
        dest=/etc/init.d/td-agent
        regexp='TD_AGENT_USER'
        line='TD_AGENT_USER=root'
      notify: td-agent restart
    - name: change start group to root
      lineinfile: >
        dest=/etc/init.d/td-agent
        regexp='TD_AGENT_GROUP'
        line='TD_AGENT_GROUP=root'
      notify: td-agent restart
  handlers:
    - name: td-agent restart                                                         
      service: name=td-agent state=restarted

logserver.yml

# logserver playbook
- name: configure fluentd
  hosts: logserver
  vars:
    - tdagent_conf_template: "roles/td-agent/templates/log-td-agent.conf.j2"
    - tdagent_port: 24224
  roles:
    - williamyeh.fluentd

roles/td-agent/templates/web-td-agent.conf.j2

####
## Source descriptions:
##

#
# Apache Log
#
## access
<source>
  type tail
  path /var/log/httpd/access_log
  tag apache.access
  pos_file /var/log/td-agent/httpd-access_log.pos
  format apache2
</source>
## error
<source>
  type tail
  path /var/log/httpd/error_log
  tag apache.error
  pos_file /var/log/td-agent/httpd-error_log.pos
  format apache_error
</source>

####
## Output descriptions:
##

#
# Apache Log
#
<match apache.**>
  type forward
  <server>
    host {{ aggregator_ip }}
    port {{ tdagent_port }}
  </server>
</match>

roles/td-agent/templates/log-td-agent.conf.j2

####
## Source descriptions:
##

#
# Recieve Logs
#
<source>
  type forward
  port {{ tdagent_port }}
</source>

####
## Output descriptions:
##
<match apache.access>
  type file
  path /var/log/td-agent/httpd/access.log
  time_slice_format %Y%m%d
  time_slice-wait 10m
  compress gzip
</match>
<match apache.error>
  type file
  path /var/log/td-agent/httpd/error.log
  time_slice_format %Y%m%d
  time_slice-wait 10m
  compress gzip
</match>

구성은 이런 느낌

결과

webserver apache log

192.168.1.1 - - [07/Jul/2016:19:26:31 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.1.1 - - [07/Jul/2016:19:26:40 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET / HTTP/1.1" 403 4961 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
192.168.2.3 - - [07/Jul/2016:19:55:28 +0900] "GET /icons/apache_pb.gif HTTP/1.1" 304 - "http://164.70.6.213:10443/" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"

logserver로 보낸 log

/var/log/td-agent/httpd/access.log

2016-07-07T19:26:31+09:00       apache.access   {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:26:40+09:00       apache.access   {"host":"192.168.1.1","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00       apache.access   {"host":"192.168.2.3","user":null,"method":"GET","path":"/","code":403,"size":4961,"referer":null,"agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}
2016-07-07T19:55:28+09:00       apache.access   {"host":"192.168.2.3","user":null,"method":"GET","path":"/icons/apache_pb.gif","code":304,"size":null,"referer":"http://164.70.6.213:10443/","agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"}

편리한 도구

log의 정규 표현을 쓸 때 도움이 될 것 같은 사이트

ぅ tp : / / f ㅅ 엔츠 ぁ r. 어리석은 p. 코m/