错误的 FS s3://ss-pprd-v2-dart //tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -预期 s3a://ss-pprd-v2-dart

Question

我正在使用 Spark 3.2.1、Java8 ->1.8.0_292 (AdoptOpenJDK)、Scala 2.12.10 并尝试使用下面提到的 jar 和包从/向 redshift 读取和写入数据。但我无法将数据写回。 同时将数据写回redshift。它正在使用临时目录中的一个manifest.json 文件创建avro 文件，但在我当前的版本中，它无法创建manifest.json 文件，但它正在创建所有avro 文件。

罐子和包：-

RedshiftJDBC42-no-awssdk-1.2.54.1082.jar,
hadoop-aws-3.3.1.jar,aws-java-sdk-1.12.173.jar ,
org.apache.spark:spark-avro_2.12:3.2.1,
io.github.spark-redshift-community:spark-redshift_2.12:5.0.3,
com.eclipsesource.minimal-json:minimal-json:0.9.5

我试图运行的代码：

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

conf=SparkConf().setAppName("Testing")
sc=SparkContext.getOrCreate(conf)
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", AWS_SECRET_KEY)

df.write \
        .format("io.github.spark_redshift_community.spark.redshift")\
        .option("url", REDSHIFT_JDBC_URL) \
        .option("dbtable",MASTER_TABLE) \
        .option("forward_spark_s3_credentials", "true") \
        .option("extracopyoptions", EXTRACOPYOPTIONS) \
        .option("tempdir", "s3a://" + str(S3_BUCKET) + "/tempdir") \
        .mode("append") \
        .save()

print("Sucesss")

堆栈跟踪：

Traceback (most recent call last):
  File "/Users/brajeshmishra/Documents/TEMP/Temp_Py.py", line 65, in <module>
    .mode("append") \
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 738, in save
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/opt/homebrew/Cellar/apache-

List item

spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
pyspark.sql.utils.IllegalArgumentException: Wrong FS s3://ss-pprd-v2-dart//tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -expected s3a://ss-pprd-v2-dart

Answer 1

我遇到了同样的错误，尝试这些版本，应该可以工作。

  hadoopVersion = 3.2.1
  sparkVersion = 3.1.2
  "org.apache.hadoop" % "hadoop-common" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-aws" % hadoopVersion,
  "org.apache.hadoop" % "hadoop-hdfs" % hadoopVersion,
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-avro" % sparkVersion,
  "com.fasterxml.jackson.core"    % "jackson-databind"     % "2.12.2",
  "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.12.2",
  "io.github.spark-redshift-community" %% "spark-redshift" % "5.0.3",
  "com.amazonaws" % "aws-java-sdk-core" % "1.12.286",
  "com.amazonaws" % "aws-java-sdk-s3" % "1.12.286",
  "com.eclipsesource.minimal-json" % "minimal-json" % "0.9.4"