无法在 Flink 的影子 jar 上加载自定义文件系统

发布于 2025-01-17 08:01:50 字数 1319 浏览 3 评论 0原文

我需要在S3对象上进行一些元数据，因此我不得不覆盖Flink提供的S3文件系统。

我跟随本指南到字母中，现在我有一个自定义文件系统，当我在IDE中运行我的应用程序时，我的本地计算机上有效。

现在，我试图在本地的Kafka群集或Docker部署上使用它，并且我继续获得此错误无法找到方案“ S3C”的文件系统实现。 Flink不直接支持该方案，并且可以加载支持此方案的No Hadoop文件系统。

使用以下配置使用ShadowJar包装我的应用程序：

shadowJar {
  configurations = [project.configurations.flinkShadowJar]
  mainClassName = "dev.vox.collect.delivery.Application"
  mergeServiceFiles()
}

我在src/main//中都有我的服务文件Resources/META-INF/SERVICE/org.apache.flink.core.fs.filesystemfactory，其中包含一个带有我工厂名称和名称的单行：dev.vox.collect.delect.delect.delevery.filesystem。 s3cfilesystemfactory

如果我解压缩了我的影子jar，我可以在其org.apache.flink.core.fs.filesystemfactory文件中看到我的工厂和其他由Flink声明的文件，应该是正确的，这应该是正确的：

dev.vox.collect.delivery.filesystem.S3CFileSystemFactory
org.apache.flink.fs.s3hadoop.S3FileSystemFactory
org.apache.flink.fs.s3hadoop.S3AFileSystemFactory

当我使用Flink提供的S3文件系统时，一切都没有。

我假设服务加载程序没有加载我的工厂，因为它找不到它，或者因为它没有正确声明。我该如何使其工作？我想念什么吗？

原文

I needed some metadata on my S3 objects, so I had to override the S3 file system provided by flink.

I followed this guide to the letter and now I have a custom file system which works on my local machine, when I run my application in the IDE.

Now I am trying to use it on a local kafka cluster OR on my docker deployment, and I keep getting this error Could not find a file system implementation for scheme 's3c'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

I package my application using shadowJar, using the following configuration:

shadowJar {
  configurations = [project.configurations.flinkShadowJar]
  mainClassName = "dev.vox.collect.delivery.Application"
  mergeServiceFiles()
}

I have my service file in src/main/resources/META-INF/services/org.apache.flink.core.fs.FileSystemFactory that contains a single line with the namespace and name of my factory :dev.vox.collect.delivery.filesystem.S3CFileSystemFactory

If I unzip my shadowJar I can see in its org.apache.flink.core.fs.FileSystemFactory file it has both my factory and the others declared by Flink, which should be correct:

dev.vox.collect.delivery.filesystem.S3CFileSystemFactory
org.apache.flink.fs.s3hadoop.S3FileSystemFactory
org.apache.flink.fs.s3hadoop.S3AFileSystemFactory

When I use the S3 file system provided by flink everything works, it is just mine that does not.

I am assuming the service loader is not loading my factory, either because it does not find it or because it is not declared correctly.
How can I make it work? Am I missing something?

分享到QQ

分享到微博