flume 데이터를elasticsearch 5로 보내기X 솔루션
10943 단어 elasticsearch
agent.sources = r1
agent.sinks = es_sink
agent.channels = c1
# Describe/configure the source
agent.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.r1.zookeeperConnect = localhost:2181,localhost1:2181,
agent.sources.r1.topic =test
agent.sources.r1.groupId = flume
agent.sources.r1.kafka.consumer.timeout.ms = 100
# Describe the sink
agent.sinks.es_sink.type=com.cognitree.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.es_sink.es.bulkActions=5
agent.sinks.es_sink.es.bulkProcessor.name=bulkprocessor
agent.sinks.es_sink.es.bulkSize=5
agent.sinks.es_sink.es.bulkSize.unit=MB
agent.sinks.es_sink.es.concurrent.request=1
agent.sinks.es_sink.es.flush.interval.time=5m
agent.sinks.es_sink.es.backoff.policy.time.interval=50M
agent.sinks.es_sink.es.backoff.policy.retries=8
agent.sinks.es_sink.es.cluster.name=mycluster
agent.sinks.es_sink.es.client.transport.sniff=false
agent.sinks.es_sink.es.client.transport.ignore_cluster_name=false
agent.sinks.es_sink.es.client.transport.ping_timeout=5s
agent.sinks.es_sink.es.client.transport.nodes_sampler_interval=5s
agent.sinks.es_sink.es.client.hosts=localhost
agent.sinks.es_sink.es.client.port=9300
agent.sinks.es_sink.es.index=flume
agent.sinks.es_sink.es.type=flume-%y-%m-%d
agent.sinks.es_sink.es.index.builder=com.cognitree.flume.sink.elasticsearch.HeaderBasedIndexBuilder
agent.sinks.es_sink.es.serializer=com.cognitree.flume.sink.elasticsearch.SimpleSerializer
agent.sinks.es_sink.es.serializer.csv.fields=id:int,name:string,isemployee:boolean,leaves:float
agent.sinks.es_sink.es.serializer.csv.delimiter=,
agent.sinks.es_sink.es.serializer.avro.schema.file=/usr/local/schema.avsc
# Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 100000
agent.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.es_sink.channel = c1
이 내용에 흥미가 있습니까?
현재 기사가 여러분의 문제를 해결하지 못하는 경우 AI 엔진은 머신러닝 분석(스마트 모델이 방금 만들어져 부정확한 경우가 있을 수 있음)을 통해 가장 유사한 기사를 추천합니다:
kafka connect e elasticsearch를 관찰할 수 있습니다.No menu lateral do dashboard tem a opção de connectors onde ele mostra todos os clusters do kafka connect conectados atu...
텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
CC BY-SA 2.5, CC BY-SA 3.0 및 CC BY-SA 4.0에 따라 라이센스가 부여됩니다.