用SPARK 3：远程FileChangedException在LocalStack上写入S3 - S3在打开位置时报告的更改。 ETAG不可用

Question

我试图将Parquet写入我的TestContainers LocalStack中，并获得此错误：

org.apache.hadoop.fs.s3a.RemoteFileChangedException: open `s3a://***.snappy.parquet': Change reported by S3 during open at position ***. ETag *** was unavailable

它正在与Real S3一起使用，并且可以与Spark 2.4和Hadoop 2.7一起使用。

我正在使用：Scala 2.12.15，Spark 3.2.1，Hadoop-aws 3.3.1，testContainers-scala-localstack 0.40.8

代码非常简单，只需将dataframe写入S3位置：

val path = "s3a://***"
import spark.implicits._

val df = Seq(UserRow("1", List("10", "20"))).toDF()
df.write.parquet(path)

Answer 1

当您创建它时，可以禁用存储桶版本。
这是一个示例：

      //create an S3 client using localstack container
       S3Client s3Client = S3Client.builder ()
        .endpointOverride (localStackContainer.getEndpointOverride (LocalStackContainer.Service.S3))
        .credentialsProvider (StaticCredentialsProvider.create (AwsBasicCredentials
            .create (localStackContainer.getAccessKey (), localStackContainer.getSecretKey ())))
        .region (Region.of (localStackContainer.getRegion ()))
        .build ();
        

    // create desired bucket
    s3Client.createBucket (builder -> builder.bucket (<your-bucket-name>));


    //disable versioning on your bucket
    s3Client.putBucketVersioning (builder -> builder
        .bucket (<your-bucket-name>)
        .versioningConfiguration (builder1 -> builder1
            .status (BucketVersioningStatus.SUSPENDED)));