Spark-Cassandra（java.lang.noclassdeffounderror：org/apache/spark/sql/sql/cassandra/package）

Question

我正在使用Spark 3.2.1使用Scala 2.12.5和SBT 1.6.2读取Cassandra 4.0.3的数据框，但我有问题。

这是我的sbt文件：

name := "StreamHandler"

version := "1.6.2"

scalaVersion := "2.12.15"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "3.2.1" % "provided",
    "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided",
    "org.apache.cassandra" % "cassandra-all" % "4.0.3" % "test",
    "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided",
    "com.datastax.spark" %% "spark-cassandra-connector" % "3.2.0",
    "com.datastax.cassandra" % "cassandra-driver-core" % "4.0.0" 
)
libraryDependencies += "com.datastax.dse" % "dse-java-driver-core" % "2.1.1" % "provided" 
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "3.2.1" % "provided"
libraryDependencies += "org.apache.commons" % "commons-math3" % "3.6.1" % "provided"

这是我的scala文件：

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming._
import org.apache.spark.sql.types._
import org.apache.spark.sql.cassandra._

import com.datastax.oss.driver.api.core.uuid.Uuids
import com.datastax.spark.connector._


object StreamHandler {
    def main(args: Array[String]){

        val spark = SparkSession
            .builder
            .appName("Stream Handler")
            .config("spark.cassandra.connection.host","localhost")
            .getOrCreate()

        import spark.implicits._


        val Temp_DF = spark
            .read
            .cassandraFormat("train_temperature", "project")
            .load()
        
        Temp_DF.show(10)
    }
}

这是结果：

Answer 1

通常问题是，当您执行SBT软件包时，它仅使用您的代码构建JAR，而无需依赖。为了减轻此问题，您有两种方法：

1. 使用spark-submit与- packages com.datastax.spark.spark：spark-cassandra-connector_2.12：3.2时指定Cassandra连接器。 0如

   
   

2. “ 依赖项）使用 sbt组装插件脂肪罐。