프로비저닝

일반 모드

디버깅 후 AP 서버에서kines-firehose로 전송, S3에 저장

컨디션

aws-kinesis-agent-1.1.2-1.amzn1.noarch

이벤트

구축할 때 문제가 없지만 대상 로그가 낮은 위치로 처리될 때 오류가 발생

객체 로그

/var/app/current/log/production.log

로그

/var/log/aws-kinesis-agent/aws-kinesis-agent.log

Initialization logs can be found in /tmp/aws-kinesis-agent.20170301165118.initlog
Unhandled exception during startup.
java.lang.IllegalStateException: Expected the service to be RUNNING, but the service has FAILED
    at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:285)
    at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:229)
    at com.google.common.util.concurrent.AbstractExecutionThreadService.awaitRunning(AbstractExecutionThreadService.java:197)
    at com.amazon.kinesis.streaming.agent.Agent.startUp(Unknown Source)
    at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
    at com.google.common.util.concurrent.MoreExecutors$2.execute(MoreExecutors.java:849)
    at com.google.common.util.concurrent.AbstractIdleService$2.doStart(AbstractIdleService.java:50)
    at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:174)
    at com.google.common.util.concurrent.AbstractIdleService.startAsync(AbstractIdleService.java:135)
    at com.amazon.kinesis.streaming.agent.Agent.main(Unknown Source)
Caused by: java.lang.IllegalStateException
    at com.google.common.base.Preconditions.checkState(Preconditions.java:158)
    at com.amazon.kinesis.streaming.agent.tailing.TrackedFileRotationAnalyzer.findCurrentOpenFileAfterTruncate(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.updateCurrentFile(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.SourceFileTracker.initialize(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.initialize(Unknown Source)
    at com.amazon.kinesis.streaming.agent.tailing.FileTailer.startUp(Unknown Source)
    at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:55)
    at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
    at java.lang.Thread.run(Thread.java:745)
FATAL: Thread FileTailer[fh:************:/var/app/current/log/production.log] threw an unrecoverable error. Aborting application

낮은 위치에서 처리할 때 무슨 일이 일어났는지 알 수 있다.

까닭

Kinessis 에이전트는 모든 fileParttern의 검사점/var/run/aws-kines-agent/checkpoint에 기록되어 있으며, 추적 대상이logrotte의coptyruncate일 경우 Kinessis 에이전트가 기록한 검사점과 일치하지 않아 오류가 발생할 수 있습니다.

/etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.webapp.conf

/var/app/containerfiles/logs/* {
 size 10M
 rotate 5
 missingok 
 compress
 notifempty
 copytruncate
 dateext
 dateformat %s
 olddir /var/app/containerfiles/logs/rotated
 }

대책

변경 로그의 출력 목적지

coptyruncate를copy 또는create

로 변경

filePattern에서 대상 파일 끝에 *

보태다

삭제/var/run/aws-kines-agent/checkpoint를 통해 aws-kines-agent

를 다시 시작할 수 있습니다

issue로 취급되기 때문에 업데이트 기대

Exception on LogRotate #50