为Avroio作家添加标题和页脚

Question

我们有一个要求将标题和页脚添加到输出AVRO文件中，但SDK似乎不支持它。对于Textio Writer，似乎具有该功能withHeader和。

也就是说，在不创建单独的管道的情况下，最好的方法是什么？我尝试在作者之后添加另一个一步，但似乎假定该管道是在作者之后结束的。

Answer 1

最终通过扩展filebasedsink找到了解决方案。

public class CustomAvroSink extens FileBasedSink{
...
 @Override
  public WriteOperation<DestinationT, GenericRecord> createWriteOperation() {
    return new CustomAvroWriteOperation(this, this.genericRecords, this.header, this.footer);
....
  }

private static class CustomAvroWriteOperation<DestinationT, OutputT> extends
      WriteOperation<DestinationT, OutputT> {

    private final DynamicAvroDestinations<?, DestinationT, OutputT> dynamicDestinations;
    private final boolean genericRecords;
    private final OutputT header;
    private final OutputT footer;

    private CustomAvroWriteOperation(HeaderFooterAvroSink<?, DestinationT, OutputT> sink,
        boolean genericRecords, OutputT header, OutputT footer) {
      super(sink);
      this.dynamicDestinations = sink.getDynamicDestinations();
      this.genericRecords = genericRecords;
      this.header = header;
      this.footer = footer;
    }

    public Writer<DestinationT, OutputT> createWriter() throws Exception {
      return new CustomAvroWriter<>(this, this.dynamicDestinations, this.genericRecords,
          this.header, this.footer);
    }
  }

...
  private static class CustomAvroWriter<DestinationT, OutputT> extends
      Writer<DestinationT, OutputT> {
   @Override
    protected void writeHeader() throws Exception {
      if (this.header != null) {
        this.dataFileWriter.append(this.header);
      }
    }

  @Override
    protected void writeFooter() throws Exception {
      if (this.footer != null) {
        this.dataFileWriter.append(this.footer);
      }
    }
 }
}

然后，我只能做mypCollection.apply（“ header footer”，writefiles.to（new customavrosink（... header，footer ...）））>