当前位置：文江博客话题详情

AWS CDK如何创建胶水工作室连接器

发布于 2025-02-02 18:44:21 字数 67 浏览 2 评论 0原文

我需要为胶合作业创建一个自定义的火花连接器，但是我只能找到有关创建数据库连接而不是通过CDK连接器的信息。这甚至可能吗？

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

丑疤怪 2025-02-09 18:44:21

这就是我所做的。我手动创建了市场连接以使用命令来提取属性。

aws glue get-connection --catalog-id <aws_account_id> --name ElasticsearchGlueConnector

然后，我获得了以下Elasticsearch的输出

{
"Connection": {
    "Name": "ElasticsearchGlueConnector",
    "Description": "",
    "ConnectionType": "MARKETPLACE",
    "MatchCriteria": [
        "Connection",
        "Elasticsearch Connector 7.13.4 for AWS Glue 3.0"
    ],
    "ConnectionProperties": {
        "CONNECTOR_CLASS_NAME": "org.elasticsearch.spark.sql",
        "CONNECTOR_TYPE": "Spark",
        "CONNECTOR_URL": "https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/elasticsearch:7.13.4-glue3.0-2"
    },
    "PhysicalConnectionRequirements": {
        "SubnetId": "xxxxx",
        "SecurityGroupIdList": [
            "xxxxxx"
        ],
        "AvailabilityZone": "xxxx"
    },
    "CreationTime": 1664056085.788,
    "LastUpdatedTime": 1664056085.788
}

}

然后，我使用此信息通过CDK创建连接，并且成功。

      new Connection(this, 'elasticSearchConnection', {
        type: new ConnectionType('MARKETPLACE'),
        connectionName: 'ElasticsearchGlueConnector',
        matchCriteria: [
            "Connection",
            "Elasticsearch Connector 7.13.4 for AWS Glue 3.0"
        ],
        description: 'ElasticSearch Connector for AWS Glue Jobs',
        securityGroups: [glueSecurityGroup],
        subnet: subnet,
        properties: {
            'CONNECTOR_CLASS_NAME': 'org.elasticsearch.spark.sql',
            'CONNECTOR_TYPE': 'Spark',
            'CONNECTOR_URL': 'https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/elasticsearch:7.13.4-glue3.0-2'
        },
    });

This is what I did. I created the marketplace connection manually to pull the properties using the command.

aws glue get-connection --catalog-id <aws_account_id> --name ElasticsearchGlueConnector

I got the following output for Elasticsearch

{
"Connection": {
    "Name": "ElasticsearchGlueConnector",
    "Description": "",
    "ConnectionType": "MARKETPLACE",
    "MatchCriteria": [
        "Connection",
        "Elasticsearch Connector 7.13.4 for AWS Glue 3.0"
    ],
    "ConnectionProperties": {
        "CONNECTOR_CLASS_NAME": "org.elasticsearch.spark.sql",
        "CONNECTOR_TYPE": "Spark",
        "CONNECTOR_URL": "https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/elasticsearch:7.13.4-glue3.0-2"
    },
    "PhysicalConnectionRequirements": {
        "SubnetId": "xxxxx",
        "SecurityGroupIdList": [
            "xxxxxx"
        ],
        "AvailabilityZone": "xxxx"
    },
    "CreationTime": 1664056085.788,
    "LastUpdatedTime": 1664056085.788
}

}

I then used this information to create connection through CDK and it was successful.

      new Connection(this, 'elasticSearchConnection', {
        type: new ConnectionType('MARKETPLACE'),
        connectionName: 'ElasticsearchGlueConnector',
        matchCriteria: [
            "Connection",
            "Elasticsearch Connector 7.13.4 for AWS Glue 3.0"
        ],
        description: 'ElasticSearch Connector for AWS Glue Jobs',
        securityGroups: [glueSecurityGroup],
        subnet: subnet,
        properties: {
            'CONNECTOR_CLASS_NAME': 'org.elasticsearch.spark.sql',
            'CONNECTOR_TYPE': 'Spark',
            'CONNECTOR_URL': 'https://709825985650.dkr.ecr.us-east-1.amazonaws.com/amazon-web-services/glue/elasticsearch:7.13.4-glue3.0-2'
        },
    });

回复收藏 0 原文

~没有更多了~