AWS胶 - 版本控制和设置连续集成

发布于 2025-02-06 20:36:55 字数 779 浏览 3 评论 0原文

我们正在为AWS胶水ETL过程设置CI / CD过程。现有的ETL流程包含以下AWS胶组件 - 爬行者,目录中的注册表,作业,触发器和工作流程。

显然,第一步是设置一个代码存储库,并将现有工件从上面提到的不同组件链接到存储库,理想情况下将需要促进开发人员执行登机手续并从该工具中提取请求(类似于ADF类似于ADF和Databricks)。但是,据我们探索,AWS胶水与任何源代码存储库都没有集成,除非我们缺少某些内容,否则可以直接提供此功能。

因此,设置CI环境的方法是什么(我仍然不是在谈论CD),以下链接给出了CI/CD的参考:

https:///aws.amazon。 com/blogs/big-data/enlument-instriun-contegration and-del-of-serverless-aws-aws-aws-glue-etl-applications-using-aws-aws-aws-developer-tooverer-tools/

但它在开始时就会提及,用于部署ETL作业的 aws云形式模板文件都致力于版本控制 - 因此,不清楚如何为开发人员进行的常规投入做到这一点。

We are in the process of setting up the CI / CD process for AWS Glue ETL Process. The existing ETL process contains the following AWS Glue Components - Crawlers, Registered tables in catalog, Jobs, Triggers and workflows.

Obviously the first step is to set up a code repository and link the existing artifacts from different components mentioned above to the repository, which will ideally need to facilitate the developers in performing the check-ins and pull request from the tool (Something similar to ADF and Databricks). However as far as we have explored, AWS glue does not have integration to any of the source code repository which can directly provide this feature unless we are missing something.

Hence what is the method to setup the environment for CI (I'm still not talking about CD), the below link gives a reference for CI/CD:

https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/

However it mentions at the beginning that, AWS CloudFormation template file for deploying the ETL jobs are both committed to version control - so not clear on how this is done for the on-going regular commits from the developers.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

你げ笑在眉眼 2025-02-13 20:36:56

但是,就我们探索而言,AWS胶没有集成
到任何可以直接提供此的源代码存储库
除非我们缺少某些东西,否则功能。

正确,胶水没有VC集成。

我在VSCODE上本地开发(Python和Cloudformation),并使用其GIT集成插件。我使用容器如果我想在本地测试某些东西,但是胶水也具有类似任务的DEV端点。

However as far as we have explored, AWS glue does not have integration
to any of the source code repository which can directly provide this
feature unless we are missing something.

Correct, Glue does not have VC integration.

I develop (python and cloudformation) locally on vscode and use it's git integration plugin. And I use a container if I want to test something locally, but Glue also has a Dev Endpoint for similar tasks.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文