Fluentd의 로그 운영

Fluentd란?

설정

conf

<match *.**>
  type forest
  subtype s3

  <template>
   aws_key_id xxx #IAMでS3への書き込み権限をもったものを指定(後述)
   aws_sec_key xxx #IAMでS3への書き込み権限をもったものを指定(後述)
   s3_bucket fluentd-logging
   path others/
   buffer_path /var/log/td-agent/buffer/${tag}

   time_slice_format ${tag}/%Y%m%d%H
   time_slice_wait 10m
   flush_interval 1m

   format_json true
   include_time_key true
   include_tag_key true

   buffer_chunk_limit 256m
  </template>
</match>

<source>
 type forward
</source>

<source>
 type config_expander
 <config>
  type tail
  format apache2
  path /var/log/httpd/access_log
  pos_file /var/log/td-agent/apache2.access_log.pos
  tag ${hostname}/apache.access
 </config>
</source>

<source>
 type config_expander
 <config>
  type tail
  format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] \[pid (?<pid>[^\]]*)\] \[client (?<client>[^\]]*)\] (?<message>.*)$/
  path /var/log/httpd/error_log
  pos_file /var/log/td-agent/apache2.error_log.pos
  tag ${hostname}/apache.error
 </config>
</source>

<source>
 type config_expander
 <config>
  type tail
  format syslog
  path /var/log/messages
  pos_file /var/log/td-agent/syslog.messages.pos
  tag ${hostname}/syslog.messages
 </config>
</source>

<source>
 type config_expander
 <config>
  type tail
  format syslog
  path /var/log/cron
  pos_file /var/log/td-agent/syslog.cron.pos
  tag ${hostname}/syslog.cron
 </config>
</source>

<source>
 type config_expander
 <config>
  type tail
  format syslog
  path /var/log/secure
  pos_file /var/log/td-agent/syslog.secure.pos
  tag ${hostname}/syslog.secure
 </config>
</source>

gem 'act-fluent-logger-rails', '0.0.4' #まだ自社環境が Rails3系だったので、0.0.4を指定

development:
  fluent_host: '0.0.0.0'
  fluent_port: 24224
  tag: 'rails.log'
  messages_type: 'string'

staging:
  fluent_host: '0.0.0.0'
  fluent_port: 24224
  tag: 'rails.log'
  messages_type: 'string'

production:
  fluent_host: '0.0.0.0'
  fluent_port: 24224
  tag: 'rails.log'
  messages_type: 'string'

  config.log_level = :info
  config.logger = ActFluentLoggerRails::Logger.new

권한 변경

하마 리소

WARNING: By default, files are created on an hourly basis (around xx:10). This means that when you first import records using the plugin, no file is created immediately. The file will be created when the time_slice_format condition has been met. To change the output frequency, please modify the time_slice_format value. To write files every minute, please use %Y%m%d%H%M for the time_slice_format.

아파치

로그 집계

활용

나중에