ELASTICSEARCH Health red 솔루션

1. 구성 항목 확인 recover_after_data_nodes: 3 # data nodes의 노드 수, 실제 노드 수를 초과하면active_shards_percent: %NaN
액세스/_cat/shards 회보 blocked by: [SERVICE_UNAVAILABLE/1/state not recovered/오류, 그리고head는 es 집단을 연결할 수 없습니다. 집단 상태는 red입니다.

curl http://metron01:9200/_cat/health?v

epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 
1534901569 09:32:49  metron  green           3         2      2   1    0    0        0             0                  -                NaN% 

2. 색인 디렉터리에서 파일 삭제

 elasticsearch.yml data  
data: "/home/opt/lmm/es_data"

#  
rm -rf /home/opt/lmm/es_data/*

 

3. shards는 UNASSIGNED가 있음

#   UNASSIGNED  
curl -s "http://localhost:9200/_cat/shards" | grep UNASSIGNED
snort_index_2018.08.17.15 2 p UNASSIGNED                                     
snort_index_2018.08.17.15 2 r UNASSIGNED                                     
snort_index_2018.08.17.15 1 p UNASSIGNED                                     
snort_index_2018.08.17.15 1 r UNASSIGNED                                     
snort_index_2018.08.17.15 3 p UNASSIGNED                                     
snort_index_2018.08.17.15 3 r UNASSIGNED                                     
snort_index_2018.08.17.15 0 p UNASSIGNED                                     
snort_index_2018.08.17.15 0 r UNASSIGNED

#  snort_index_2018.08.17.15   0,1,2,3

데이터 node ID 조회

curl 'localhost:9200/_nodes/process?pretty'
{
  "cluster_name" : "metron",
  "nodes" : {
    "I3_T2xI1RGCocy7sZcdg9w" : {
      "name" : "metron03",
      "transport_address" : "172.16.16.59:9300",
      "host" : "172.16.16.59",
      "ip" : "172.16.16.59",
      "version" : "2.3.3",
      "build" : "218bdf1",
      "http_address" : "172.16.16.59:9200",
      "attributes" : {
        "master" : "false"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 16002,
        "mlockall" : false
      }
    },
    "xOk36eUVQhSgAHYv6VT77Q" : {
      "name" : "metron01",
      "transport_address" : "172.16.16.57:9300",
      "host" : "172.16.16.57",
      "ip" : "172.16.16.57",
      "version" : "2.3.3",
      "build" : "218bdf1",
      "http_address" : "172.16.16.57:9200",
      "attributes" : {
        "data" : "false",
        "master" : "true"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 25044,
        "mlockall" : false
      }
    },
    "ncgAeS6nQp-L0d3U0roVMA" : {
      "name" : "metron02",
      "transport_address" : "172.16.16.58:9300",
      "host" : "172.16.16.58",
      "ip" : "172.16.16.58",
      "version" : "2.3.3",
      "build" : "218bdf1",
      "http_address" : "172.16.16.58:9200",
      "attributes" : {
        "master" : "false"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 12653,
        "mlockall" : false
      }
    }
  }
}

모든shard와 번호에 대한reroute 실행

curl -XPOST 'metron01:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "snort_index_2018.08.17.15",  #  
                  "shard" : 0,   # shard ID
                  "node" : "I3_T2xI1RGCocy7sZcdg9w", # data node ID
                  "allow_primary" : true
              }
            }
        ]
    }'
curl -XPOST 'metron01:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "snort_index_2018.08.17.15",  
                  "shard" : 1,   
                  "node" : "I3_T2xI1RGCocy7sZcdg9w",
                  "allow_primary" : true
              }
            }
        ]
    }'
curl -XPOST 'metron01:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "snort_index_2018.08.17.15",  
                  "shard" : 2,  
                  "node" : "I3_T2xI1RGCocy7sZcdg9w", 
                  "allow_primary" : true
              }
            }
        ]
    }'
curl -XPOST 'metron01:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "snort_index_2018.08.17.15",  
                  "shard" : 3,   
                  "node" : "I3_T2xI1RGCocy7sZcdg9w", 
                  "allow_primary" : true
              }
            }
        ]
    }'

다음 오류가 발생하면

{
	"error": {
		"root_cause": [
			{
				"type": "illegal_argument_exception",
				"reason": "[allocate] allocation of [.kibana][0] on node {metron03}{WPJIoZ6CRTetEAH_j8nPBw}{172.16.16.59}{172.16.16.59:9300}{master=false} is not allowed, reason: [YES(primary is already active)][YES(allocation disabling is ignored)][YES(target node version [2.3.3] is same or newer than source node version [2.3.3])][YES(node passes include/exclude/require filters)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(no allocation awareness enabled)][YES(enough disk for shard on node, free: [187.9gb])][YES(allocation disabling is ignored)][YES(shard not primary or relocation disabled)][YES(below shard recovery limit of [4])][NO(shard cannot be allocated on same node [WPJIoZ6CRTetEAH_j8nPBw] it already exists on)]"
			}
		],
		"type": "illegal_argument_exception",
		"reason": "[allocate] allocation of [.kibana][0] on node {metron03}{WPJIoZ6CRTetEAH_j8nPBw}{172.16.16.59}{172.16.16.59:9300}{master=false} is not allowed, reason: [YES(primary is already active)][YES(allocation disabling is ignored)][YES(target node version [2.3.3] is same or newer than source node version [2.3.3])][YES(node passes include/exclude/require filters)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(no allocation awareness enabled)][YES(enough disk for shard on node, free: [187.9gb])][YES(allocation disabling is ignored)][YES(shard not primary or relocation disabled)][YES(below shard recovery limit of [4])][NO(shard cannot be allocated on same node [WPJIoZ6CRTetEAH_j8nPBw] it already exists on)]"
	},
	"status": 400
}

복사본 수를 0으로 설정하면 es는 자동으로 남은 복사본을 삭제하고 다시 2로 바꾸어 처리할 수 있습니다.

curl -XPUT  http://metron01:9200/.kibana/_settings -d'{
    "index" : {
        "number_of_replicas" : 0
    }
}'
 ：{"acknowledged":true}
curl -XPUT  http://metron01:9200/.kibana/_settings -d'{
    "index" : {
        "number_of_replicas" : 2
    }
}'
 ：{"acknowledged":true}