Hive-metastore 找不到 Hadoop Datanode

发布于 2025-01-12 19:48:58 字数 2457 浏览 0 评论 0原文

我有一个 Hadoop 集群,其中有一个名称节点和一个使用 docker compose 实例化的数据节点。另外,我正在尝试启动 Hive,但 Hive-metastore 似乎找不到我的数据节点,即使这个节点已启动并正在运行,事实上检查它显示的日志:

namenode:9870 is available.
check for datanode:9871...
datanode:9871 is not available yet
try in 5s once again ...

这是我的 docker-compose.yml

#HADOOP
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    expose:
      - "9870"
      - "54310"
      - "9000"
    ports:
      - 9870:9870
      - 9000:9000
    volumes:
      - ./data/hadoop_data/:/hadoop_data
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec

      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
      - HDFS_CONF_dfs_safemode_threshold_pct=0

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    expose:
      - "9871"
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    ports:
      - "9871:9871"
    env_file:
      - hive.env

hive-server:
    image: bde2020/hive:2.3.2-postgresql-metastore
    container_name: hive-server
    volumes:
      - ./employee:/employee
    env_file:
      - hive.env
    environment:
      HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore"
      SERVICE_PRECONDITION: "hive-metastore:9083"
    depends_on:
      - hive-metastore
    ports:
      - "10000:10000"

  hive-metastore:
    image: bde2020/hive:2.3.2-postgresql-metastore
    container_name: hive-metastore
    env_file:
      - hive.env
    command: /opt/hive/bin/hive --service metastore
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode:9871 hive-metastore-postgresql:5432"
    depends_on:
      - hive-metastore-postgresql
    ports:
      - "9083:9083"

  hive-metastore-postgresql:
    image: bde2020/hive-metastore-postgresql:2.3.0
    container_name: hive-metastore-postgresql
    volumes:
      - ./metastore-postgresql/postgresql/data:/var/lib/postgresql/data
    depends_on:
      - datanode

I have an Hadoop cluster with one namenode and one datanode instantiated with docker compose. In addition I am trying to launch Hive but the Hive-metastore seems not to find my datanode even if this one is up and running, infact checking the log it shows:

namenode:9870 is available.
check for datanode:9871...
datanode:9871 is not available yet
try in 5s once again ...

Here is my docker-compose.yml

#HADOOP
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    restart: always
    expose:
      - "9870"
      - "54310"
      - "9000"
    ports:
      - 9870:9870
      - 9000:9000
    volumes:
      - ./data/hadoop_data/:/hadoop_data
    environment:
      - CLUSTER_NAME=test
      - CORE_CONF_fs_defaultFS=hdfs://namenode:9000
      - CORE_CONF_hadoop_http_staticuser_user=root
      - CORE_CONF_hadoop_proxyuser_hue_hosts=*
      - CORE_CONF_hadoop_proxyuser_hue_groups=*
      - CORE_CONF_io_compression_codecs=org.apache.hadoop.io.compress.SnappyCodec

      - HDFS_CONF_dfs_webhdfs_enabled=true
      - HDFS_CONF_dfs_permissions_enabled=false
      - HDFS_CONF_dfs_namenode_datanode_registration_ip___hostname___check=false
      - HDFS_CONF_dfs_safemode_threshold_pct=0

  datanode:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode
    restart: always
    expose:
      - "9871"
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    ports:
      - "9871:9871"
    env_file:
      - hive.env

hive-server:
    image: bde2020/hive:2.3.2-postgresql-metastore
    container_name: hive-server
    volumes:
      - ./employee:/employee
    env_file:
      - hive.env
    environment:
      HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://hive-metastore/metastore"
      SERVICE_PRECONDITION: "hive-metastore:9083"
    depends_on:
      - hive-metastore
    ports:
      - "10000:10000"

  hive-metastore:
    image: bde2020/hive:2.3.2-postgresql-metastore
    container_name: hive-metastore
    env_file:
      - hive.env
    command: /opt/hive/bin/hive --service metastore
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode:9871 hive-metastore-postgresql:5432"
    depends_on:
      - hive-metastore-postgresql
    ports:
      - "9083:9083"

  hive-metastore-postgresql:
    image: bde2020/hive-metastore-postgresql:2.3.0
    container_name: hive-metastore-postgresql
    volumes:
      - ./metastore-postgresql/postgresql/data:/var/lib/postgresql/data
    depends_on:
      - datanode

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文