hive0.13 mapjoin hashtable에서 찾을 수 없는 버그

온라인 작업 오류:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /home/vipshop/hard_disk/1/yarn/local/usercache/hdfs/appcache/application_1420458339569_0548/container_1420458339569_0548_01_000005/Stage-5.tar.gz/MapJoin-mapfile12--.hashtable (No such file or directory)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: /home/vipshop/hard_disk/1/yarn/local/usercache/hdfs/appcache/application_1420458339569_0548/container_1420458339569_0548_01_000005/Stage-5.tar.gz/MapJoin-mapfile12--.hashtable (No such file or directory)
        at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:104)
        at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:152)
        at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:178)
        at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1029)
        at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033)
        at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033)
        at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1033)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:505)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
        ... 8 more
Caused by: java.io.FileNotFoundException: /home/vipshop/hard_disk/1/yarn/local/usercache/hdfs/appcache/application_1420458339569_0548/container_1420458339569_0548_01_000005/Stage-5.tar.gz/MapJoin-mapfile12--.hashtable (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at java.io.FileInputStream.<init>(FileInputStream.java:101)
        at org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:95)
        ... 16 more

이것은 사실mapjoin의 버그입니다.mapjoin은 작은 테이블을 통해hashtable를 생성한 다음distributecache에 넣고, 뒤의task는distributecache를 통해 로컬로 다운로드하여 사용합니다.여기에는job에 두 개의 mapjoin이 포함되어 있지만 Hash Table Sink Operator에서 첫 번째hashtable만 생성되어 Hash Table Loader에서load hashtable를 진행하는 데 오류가 발생했습니다.버그 트리거 조건: 1.두 개 이상의 맵조인2.테이블 중 하나는 비어 있는 Bugid:https://issues.apache.org/jira/browse/HIVE-6913이 버그 Hive0.14는 해결 방법을 수정했습니다.

./ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
Operator<? extends OperatorDesc> forwardOp = work.getAliasToWork().get(alias);
if (fetchOp.isEmptyTable()) {
  //generate empty hashtable for empty table
  this.generateDummyHashTable(alias, bigTableBucket);
  forwardOp.close(false);
  continue;
}

mapjoin에 대한 전체 절차와 트리거 조건을 뒤에 놓고 쓰십시오.