EMR Serverless无法连接到另一个区域中的S3

发布于 2025-02-13 07:26:13 字数 290 浏览 0 评论 0原文

我有一个无法连接到其他区域中S3存储桶的EMR无服务器应用程序。有解决方法吗？也许是在提交新作业时在作业参数或火花参数中设置的参数。错误是：

ExitCode: 1. Last few exceptions: Caused by: java.net.SocketTimeoutException: connect timed out Caused by: com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectTimeoutException

原文

I have an EMR serverless app that cannot connect to S3 bucket in another region. Is there a workaround for that? Maybe a parameter to set in Job parameters or Spark parameters when submitting a new job.
The error is this:

ExitCode: 1. Last few exceptions: Caused by: java.net.SocketTimeoutException: connect timed out Caused by: com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.ConnectTimeoutException

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

初与友歌 2025-02-20 07:26:13

为了连接其他区域或访问外部服务中的S3存储桶，需要使用VPC创建EMR无服务器应用程序。

这在考虑页面 page ：

没有VPC连接，作业可以访问同一AWS区域中的某些AWS服务端点。这些服务包括Amazon S3，AWS胶，Amazon DynamoDB，Amazon CloudWatch，AWS KMS和AWS Secrets Manager。

这是一个示例AWS CLI命令，可以在VPC中创建应用程序 - 您需要提供子网ID和安全组ID的列表。更多详细信息可以在配置VPC访问< /a>。

aws emr-serverless create-application \
  --type SPARK \                                                                                               
  --name etl-jobs \
  --release-label "emr-6.6.0" \
  --network-configuration '{
        "subnetIds": ["subnet-01234567890abcdef","subnet-01234567890abcded"],
        "securityGroupIds": ["sg-01234566889aabbcc"]
    }'

In order to connect to an S3 bucket in another region or access external services, the EMR Serverless application needs to be created with a VPC.

This is mentioned on the considerations page:

Without VPC connectivity, a job can access some AWS service endpoints in the same AWS Region. These services include Amazon S3, AWS Glue, Amazon DynamoDB, Amazon CloudWatch, AWS KMS, and AWS Secrets Manager.

Here's an example AWS CLI command to create an application in a VPC - you need to provide a list of Subnet IDs and Security Group IDs. More details can be found in configuring VPC access.

aws emr-serverless create-application \
  --type SPARK \                                                                                               
  --name etl-jobs \
  --release-label "emr-6.6.0" \
  --network-configuration '{
        "subnetIds": ["subnet-01234567890abcdef","subnet-01234567890abcded"],
        "securityGroupIds": ["sg-01234566889aabbcc"]
    }'

回复收藏 0 原文

~没有更多了~