将水槽与Kafka连接时错误 - 找不到A' kafkaclient'进入JAAS配置

发布于 2025-02-09 06:00:35 字数 3522 浏览 1 评论 0原文

Currently we are using CDP 7.1.7 and client wants to use Flume. Since CDP has removed Flume, we need to install it as a separate application. I have installed Flume at one of the data nodes. 
here are the config files:

Flume-env.sh


    export JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/etc/krb5.conf
     -Djava.security.auth.login.config=/opt/cloudera/security/flafka_jaas.conf "

flafka_jaas.conf


    Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      keyTab="flume.keytab" 
      principal="flume/hostname@realm";
    };
    
    KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      serviceName="kafka"
      keyTab="flume.keytab"
      principal="flume/hostname@realm";
    };


flume.conf


    KafkaAgent.sources = source_kafka
       KafkaAgent.channels = MemChannel
       KafkaAgent.sinks = LoggerSink
    
       #Configuring Source
       KafkaAgent.sources.source_kafka.type = org.apache.flume.source.kafka.KafkaSource
       KafkaAgent.sources.source_kafka.kafka.bootstrap.servers = hostn1:9092,host2:9092,host3:9092
       KafkaAgent.sources.source_kafka.kafka.topics = cim
       KafkaAgent.sources.source_kafka.kafka.consumer.group.id = flume
        KafkaAgent.sources.source_kafka.channels = MemChannel
       #KafkaAgent.sources.source_kafka.kafka.consumer.timeout.ms = 100
       KafkaAgent.sources.source_kafka.agent-principal=flume/hostname@realm
       KafkaAgent.sources.source_kafka.agent-keytab=flume.keytab
       KafkaAgent.sources.source_kafka.kafka.consumer.security.protocol = SASL_PLAINTEXT
       KafkaAgent.sources.source_kafka.kafka.consumer.sasl.kerberos.service.name = kafka
        KafkaAgent.sources.source_kafka.kafka.consumer.sasl.mechanism = GSSAPI
       KafkaAgent.sources.source_kafka.kafka.consumer.security.protocol = SASL_PLAINTEXT
       #Configuring Sink
       KafkaAgent.sinks.LoggerSink.type = logger
    
       #Configuring Channel
       KafkaAgent.channels.MemChannel.type = memory
       KafkaAgent.channels.MemChannel.capacity = 10000
        KafkaAgent.channels.MemChannel.transactionCapacity = 1000
    
       #bind source and sink to channel
       KafkaAgent.sinks.LoggerSink.channel = MemChannel

运行此命令:

`flume-ng agent -n KafkaAgent -c -conf /opt/cdpdeployment/apache-flume-1.9.0-bin/conf/ -f /opt/cdpdeployment/apache-flume-1.9.0-bin/conf/kafka-flume.conf  -Dflume.root.logger=DEBUG,console`

我收到以下错误:

    

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
    at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:133)
    at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:98)
    at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:119)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:713)

就配置而言,有人可以告诉我我所缺少的东西吗?

Currently we are using CDP 7.1.7 and client wants to use Flume. Since CDP has removed Flume, we need to install it as a separate application. I have installed Flume at one of the data nodes. 
here are the config files:

flume-env.sh


    export JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/etc/krb5.conf
     -Djava.security.auth.login.config=/opt/cloudera/security/flafka_jaas.conf "

flafka_jaas.conf


    Client {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      keyTab="flume.keytab" 
      principal="flume/hostname@realm";
    };
    
    KafkaClient {
      com.sun.security.auth.module.Krb5LoginModule required
      useKeyTab=true
      storeKey=true
      serviceName="kafka"
      keyTab="flume.keytab"
      principal="flume/hostname@realm";
    };


Flume.conf


    KafkaAgent.sources = source_kafka
       KafkaAgent.channels = MemChannel
       KafkaAgent.sinks = LoggerSink
    
       #Configuring Source
       KafkaAgent.sources.source_kafka.type = org.apache.flume.source.kafka.KafkaSource
       KafkaAgent.sources.source_kafka.kafka.bootstrap.servers = hostn1:9092,host2:9092,host3:9092
       KafkaAgent.sources.source_kafka.kafka.topics = cim
       KafkaAgent.sources.source_kafka.kafka.consumer.group.id = flume
        KafkaAgent.sources.source_kafka.channels = MemChannel
       #KafkaAgent.sources.source_kafka.kafka.consumer.timeout.ms = 100
       KafkaAgent.sources.source_kafka.agent-principal=flume/hostname@realm
       KafkaAgent.sources.source_kafka.agent-keytab=flume.keytab
       KafkaAgent.sources.source_kafka.kafka.consumer.security.protocol = SASL_PLAINTEXT
       KafkaAgent.sources.source_kafka.kafka.consumer.sasl.kerberos.service.name = kafka
        KafkaAgent.sources.source_kafka.kafka.consumer.sasl.mechanism = GSSAPI
       KafkaAgent.sources.source_kafka.kafka.consumer.security.protocol = SASL_PLAINTEXT
       #Configuring Sink
       KafkaAgent.sinks.LoggerSink.type = logger
    
       #Configuring Channel
       KafkaAgent.channels.MemChannel.type = memory
       KafkaAgent.channels.MemChannel.capacity = 10000
        KafkaAgent.channels.MemChannel.transactionCapacity = 1000
    
       #bind source and sink to channel
       KafkaAgent.sinks.LoggerSink.channel = MemChannel

After running this command:

`flume-ng agent -n KafkaAgent -c -conf /opt/cdpdeployment/apache-flume-1.9.0-bin/conf/ -f /opt/cdpdeployment/apache-flume-1.9.0-bin/conf/kafka-flume.conf  -Dflume.root.logger=DEBUG,console`

I am getting this below error:

    

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
    at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:133)
    at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:98)
    at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:119)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:65)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:88)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:713)

Can someone tell me what I am missing as far as configs are concerned.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文