利用hive0.12批量导入数据到hbase0.96中

Question

在官方文档上看见一种方法用hive直接生成hfile文件，然后实现向hbase批量导入数据，但老是报找不到分区文件的异常，但分区文件明明存在HDFS上，但就是找不到，求大神指点

我们用的测试集群是cdh的hadoop2.3+hive0.12+hbase0.96

hive生成hfile官方方法链接：hortworks网站的:http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.2/bk_user-guide/content/user-guide-hbase-import-1.html;apache的:https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad

所报异常如下：

Diagnostic Messages for this Task:

Error: java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:569)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.FileNotFoundException: File file:/inm/app/cdh5/cdhworkspace/yarn/local/usercache/hadoop/appcache/application_1398705638986_0038/container_1398705638986_0038_01_000005/_partition.lst does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:402)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:301)
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:88)
        ... 10 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1  Reduce: 5   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

请大神指点，谢谢！

Answer 1

hive和hbase整合之前就弄过，那个速度达不到要求，数据只要上G性能就很差了，一般不采用，因hive底层采用的是hbase的put进行同步的，这个性能很低，主采用的还是生成hfile文件再导入，有时间你再测试测试效率

Answer 2

看来文档，测试如何？

Answer 3

这个问题最后问同事，经翻看源码解决了,在以上所说的环境中，路径名称已经更换，得需进行如下设置，示例：

ADD JAR /inm/app/cdh5/hive-0.12.0-cdh5.0.0/lib/hbase-client-0.96.1.1-cdh5.0.0.jar;
ADD JAR /inm/app/cdh5/hive-0.12.0-cdh5.0.0/lib/hbase-common-0.96.1.1-cdh5.0.0.jar
ADD JAR /inm/app/cdh5/hive-0.12.0-cdh5.0.0/lib/hive-hbase-handler-0.12.0-cdh5.0.0.jar;
SET mapred.reduce.tasks=5;
SET hive.mapred.partitioner=org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
set mapreduce.totalorderpartitioner.path=/tmp/hbase_splits;
INSERT OVERWRITE TABLE hbase_hfiles SELECT * FROM pgc CLUSTER BY rowkey;