如何使用Apache Beam Java将镶木quet文件写入AWS S3？

Question

我正在尝试转换JSON - ＆GT;通用记录 - ＆gt; PARQUET-到 - ＆GT; S3。我能够将其转换为Parquet，但我不知道如何将Parquet文件直接放置在S3的情况下，而无需将其存储到文件系统中。

我为此写的代码：

public static void main(String[] args) throws IOException {

    PipelineOptionsFactory.register(MainConfig.class);
    MainConfig options = PipelineOptionsFactory.fromArgs(args).withValidation().as(MainConfig.class);
    Pipeline pipeLine = Pipeline.create(options);
    BasicAWSCredentials awsCredentials = new BasicAWSCredentials(options.getAWSAccessKey(), options.getAWSSecretKey());
    options.setAwsCredentialsProvider(new AWSStaticCredentialsProvider(awsCredentials));

    Schema jsonSchema = new Schema.Parser().parse(schemaString);
    logger.info(jsonSchema.getFields());
    pipeLine.apply("ReadMyFile", TextIO.read().from(options.getInput()))
            .apply("Convert Json To General Record", ParDo.of(new JsonToGeneralRecord(jsonSchema)))
            .setCoder(AvroCoder.of(GenericRecord.class, jsonSchema))
            .apply("Generation the parquet files", FileIO.<GenericRecord>write().via(ParquetIO.sink(jsonSchema)).to(options.getOutput()).withNumShards(1).withSuffix(".parquet"));
    pipeLine.run();


}

最后，我只想将该镶木添加到S3