AWS胶 - 版本控制和设置连续集成
我们正在为AWS胶水ETL过程设置CI / CD过程。现有的ETL流程包含以下AWS胶组件 - 爬行者,目录中的注册表,作业,触发器和工作流程。
显然,第一步是设置一个代码存储库,并将现有工件从上面提到的不同组件链接到存储库,理想情况下将需要促进开发人员执行登机手续并从该工具中提取请求(类似于ADF类似于ADF和Databricks)。但是,据我们探索,AWS胶水与任何源代码存储库都没有集成,除非我们缺少某些内容,否则可以直接提供此功能。
因此,设置CI环境的方法是什么(我仍然不是在谈论CD),以下链接给出了CI/CD的参考:
但它在开始时就会提及,用于部署ETL作业的 aws云形式模板文件都致力于版本控制 - 因此,不清楚如何为开发人员进行的常规投入做到这一点。
We are in the process of setting up the CI / CD process for AWS Glue ETL Process. The existing ETL process contains the following AWS Glue Components - Crawlers, Registered tables in catalog, Jobs, Triggers and workflows.
Obviously the first step is to set up a code repository and link the existing artifacts from different components mentioned above to the repository, which will ideally need to facilitate the developers in performing the check-ins and pull request from the tool (Something similar to ADF and Databricks). However as far as we have explored, AWS glue does not have integration to any of the source code repository which can directly provide this feature unless we are missing something.
Hence what is the method to setup the environment for CI (I'm still not talking about CD), the below link gives a reference for CI/CD:
However it mentions at the beginning that, AWS CloudFormation template file for deploying the ETL jobs are both committed to version control - so not clear on how this is done for the on-going regular commits from the developers.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
正确,胶水没有VC集成。
我在VSCODE上本地开发(Python和Cloudformation),并使用其GIT集成插件。我使用容器如果我想在本地测试某些东西,但是胶水也具有类似任务的DEV端点。
Correct, Glue does not have VC integration.
I develop (python and cloudformation) locally on vscode and use it's git integration plugin. And I use a container if I want to test something locally, but Glue also has a Dev Endpoint for similar tasks.