Hive:insert 데이터 시 Error during job,obtaining debugging 정보 및 beyond physical memory limits...
insert overwrite table canal_amt1......
2014-10-09 10:40:27,368 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2772.48 sec
2014-10-09 10:40:28,426 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2772.48 sec
2014-10-09 10:40:29,481 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2774.12 sec
2014-10-09 10:40:30,885 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 2774.36 sec
2014-10-09 10:40:31,963 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:33,071 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:34,126 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
2014-10-09 10:40:35,182 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2693.96 sec
MapReduce Total cumulative CPU time: 44 minutes 53 seconds 960 msec
Ended Job = job_1409124602974_0745 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1409124602974_0745_m_000003 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_m_000002 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_r_000000 (and more) from job job_1409124602974_0745
Examining task ID: task_1409124602974_0745_r_000006 (and more) from job job_1409124602974_0745
Task with the most failures(4):
-----
Task ID:
task_1409124602974_0745_r_000003
URL:
http://HADOOP2:8088/taskdetails.jsp?jobid=job_1409124602974_0745&tipid=task_1409124602974_0745_r_000003
-----
Diagnostic Messages for this Task:
Container [pid=22068,containerID=container_1409124602974_0745_01_000047] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1409124602974_0745_01_000047 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 22087 22068 22068 22068 (java) 2536 833 2730713088 265378 /usr/jdk64/jdk1.6.0_31/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2048m -Djava.io.tmpdir=/hadoop/yarn/local/usercache/root/appcache/application_1409124602974_0745/container_1409124602974_0745_01_000047/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 54.0.88.58 41150 attempt_1409124602974_0745_r_000003_3 47
|- 22068 2381 22068 22068 (bash) 1 1 110755840 302 /bin/bash -c /usr/jdk64/jdk1.6.0_31/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx2048m -Djava.io.tmpdir=/hadoop/yarn/local/usercache/root/appcache/application_1409124602974_0745/container_1409124602974_0745_01_000047/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 54.0.88.58 41150 attempt_1409124602974_0745_r_000003_3 47 1>/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047/stdout 2>/hadoop/yarn/log/application_1409124602974_0745/container_1409124602974_0745_01_000047/stderr
Container killed on request. Exit code is 143
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 23 Reduce: 7 Cumulative CPU: 2693.96 sec HDFS Read: 6278784712 HDFS Write: 590228229 FAIL
Total MapReduce CPU Time Spent: 44 minutes 53 seconds 960 msec
이유: 공간 부족
해결 방법:
하이브 문장을 실행하기 전에 덧붙이기
set mapreduce.map.memory.mb=1025;// 1024,hive , 2048M
set mapreduce.reduce.memory.mb=1025;
실행 결과:
MapReduce Total cumulative CPU time: 0 days 1 hours 10 minutes 14 seconds 590 msec
Ended Job = job_1409124602974_0746
Loading data to table default.canal_amt1
Table default.canal_amt1 stats: [num_partitions: 0, num_files: 7, num_rows: 0, total_size: 4131948868, raw_data_size: 0]
MapReduce Jobs Launched:
Job 0: Map: 23 Reduce: 7 Cumulative CPU: 4214.59 sec HDFS Read: 6278784712 HDFS Write: 4131948868 SUCCESS
Total MapReduce CPU Time Spent: 0 days 1 hours 10 minutes 14 seconds 590 msec
OK
Time taken: 673.851 seconds
인터넷 조회의 다른 원인:
1.map 단계 비우기 포인터
이유: 데이터 필드에 빈 값이 삽입되었습니다.
2.Exception in thread "Thread-19"java.lang.IllegalArgumentException: Does not contain a valid host:port authority: local
참고http://grokbase.com/p/cloudera/cdh-user/126wqvfwyt/hive-refuses-to-work-with-yarn
해결 방법:
바로 하이브 사이트입니다.xml에 설정 추가
In the meantime I recommend doing the following if you need to run Hive on
MR2:
* Keep Hive happy by setting mapred.job.tracker to a bogus value.
* Disable task log retrieval by setting
hive.exec.show.job.failure.debug.info=false
3.protuf 버전이 일치하지 않습니다.
다음으로 전송:https://www.cnblogs.com/kxdblog/p/4034244.html
이 내용에 흥미가 있습니까?
현재 기사가 여러분의 문제를 해결하지 못하는 경우 AI 엔진은 머신러닝 분석(스마트 모델이 방금 만들어져 부정확한 경우가 있을 수 있음)을 통해 가장 유사한 기사를 추천합니다:
다양한 언어의 JSONJSON은 Javascript 표기법을 사용하여 데이터 구조를 레이아웃하는 데이터 형식입니다. 그러나 Javascript가 코드에서 이러한 구조를 나타낼 수 있는 유일한 언어는 아닙니다. 저는 일반적으로 '객체'{}...
텍스트를 자유롭게 공유하거나 복사할 수 있습니다.하지만 이 문서의 URL은 참조 URL로 남겨 두십시오.
CC BY-SA 2.5, CC BY-SA 3.0 및 CC BY-SA 4.0에 따라 라이센스가 부여됩니다.