使用 Cloud Data Fusion 将数据从 SFTP 提取到 GCS 或 BigQuery 时出错
我正在尝试使用数据融合将 SFTP 文件夹中的 CSV 文件移动到 GCS。但我无法做到这一点并抛出以下错误：
以下是 FTP 和 GCS 插件的属性。令人惊讶的是，我可以在所有阶段的预览模式下看到数据，但当我尝试部署管道时，它失败了。我尝试在源（FTP）和接收器（GCS）之间使用 CSVParser 以及 TRANSFORM。但它仍然显示相同的错误。我在 Hub 中使用 FTP 插件，版本为 3.0.0。请帮我解决它。
当我尝试部署管道时，错误如下，即使预览数据我也能够看到数据。
I am trying to move CSV files in SFTP folder to GCS using Data Fusion. But I am unable to do it and throwing below error:
Here are the properties of both FTP and GCS plugins. Surprisingly, I could see the data in PREVIEW mode in all the stages but when I try to deploy the pipeline it fails. I tried using CSVParser as well as a TRANSFORM in between source(FTP) and sink (GCS). Still it shows the same error. I am using FTP plugin in Hub with version 3.0.0. Please help me to solve it.
And the error is as below, when I try to deploy the pipeline, eventhough Preview Data I was able to see the data.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
我通过在数据融合中将管道执行引擎从 SPARK 更改为 MAPREDUCE 解决了这个问题。现在它正在发挥作用。
I solved this issue by changing the Pipeline execution engine from SPARK to MAPREDUCE in Data Fusion. Now it is working.
嗯，我对此进行了很多挖掘，我发现这个插件在运行 ftp-plugins< 时存在问题/a>，所以目前你不能做太多事情。幸运的是，有解决方法。仅举几例：
您可以使用旧版本（ Dataproc 映像为 1.5/1.3 ），如公开案例中所示，该案例也参考了此问题。有关此案例的更多详细信息，您可以查看该问题的链接：SFTP 源在部署时失败 (SftpExecption)但不在预览中。不要忘记投票并发表评论。
另一种方法是使用
SFTPCopy插件（从中心获取后，它应该出现在条件和操作下）。因此，您将能够从 SFTP 获取文件到本地路径，并使用 SourceFILE继续处理文件。有一个关于 从 SFTP 读取并写入 BigQuery这有点极端，但您也可以使用不同的工作流管理平台，例如 airflow 用于文件处理。
Well I have dig a lot on this, I found that this plugins have issues when running ftp-plugins, so at the moment you can't do much on it. Fortunately, there are workarounds for this. To name a few here are some:
You can use an old version ( Dataproc image to 1.5/1.3 ) as indicated on the public case that also makes reference to this issue. For more details about this case, you can check the link for the issue, SFTP Source fails when deployed (SftpExecption) but not in preview. Don't forget to upvote and leave a comment too.
Another way is to use
SFTPCopyplugin (once you pick up from the hub it should appear underConditions and Actions). So you will be able to pick up the file from your SFTP into a local path and the use SourceFILEto continue with the processing of your file. There is a small guide on Reading from SFTP and writing to BigQueryThis one is a bit extreme but you can also use a different workflow management platform like airflow for file processing.