使用Java连接到EMR HBase

Question

我正在尝试从EMR 5.35（Hadoop 2.10，Spark 2.4.8，HBase 1.4.13）上运行的SPARK程序连接到HBase。 当不尝试连接到HBASE时，我的Spark程序运行完美。

但是，当我添加HBase代码时，Spark程序在创建配置时就会死亡：

 conf = HBaseConfiguration.create();

 for (Iterator<Map.Entry<String, String>> it = conf.iterator(); it.hasNext(); ) {
     Map.Entry<String, String> e = it.next();
     System.out.println(e);
 }

 connection = ConnectionFactory.createConnection(conf);
 admin = connection.getAdmin();

我尝试添加资源：

    conf = HBaseConfiguration.create();
    conf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
    conf.addResource(new Path("/etc/hbase/conf/hbase-site.xml"));

没有成功。

我已经在hbaseconfiguration.create（）之后评论了所有线条，但是该程序还是死了。我相信问题就在那里。我没有有用的堆栈跟踪。驾驶员撞上线时立即死亡。

POM：

<properties>
    <spark.version>2.4.8</spark.version>
    <hbase.version>1.4.13</hbase.version>
    <hadoop.version>2.10.1</hadoop.version>
    <jackson.version>2.13.2</jackson.version>
    <!-- Maven stuff -->
    <java.build.version>1.8</java.build.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.17.103</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.77.Final</version>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.9.9.Final</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-aws</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>${hbase.version}</version>
        <scope>provided</scope>
    </dependency>

    <!--  AWS -->
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>athena</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>auth</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>opensearch</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>apache-client</artifactId>
    </dependency>

    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpcore</artifactId>
        <version>4.4.15</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>elasticsearch-rest-client</artifactId>
        <version>5.6.16</version>
    </dependency>

Answer 1

通过删除“提供”的范围解决了问题，因此包括对Uber -Jar的依赖性，加上添加HBase POM依赖性（不是100％确定这是否严格必要）

    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>${hbase.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase</artifactId>
        <version>${hbase.version}</version>
        <type>pom</type>
    </dependency>