在MWAA中列出包裹,并带有一个损坏的调度程序

发布于 2025-01-27 01:57:18 字数 727 浏览 4 评论 0原文

因此,我目前正在使用MWAA进行气流安装。我的依赖性破裂的问题是:

错误:PIP的依赖项解析器当前未考虑所有已安装的软件包。此行为是以下依赖性冲突的根源。

apache-airFlow-Providers-amazon 1.4.0需要watchtower〜 = 0.7.3,但是您的watchtower 2.0.1不兼容。

我想获取当前安装的软件包的列表以进行一些调试,但是损坏的安装似乎也打破了调度程序。另外,工人日志表明,它只是不断尝试和无法安装requients.txt中列出的内容。

除了仅以破坏的依赖性删除软件包外,是否有任何方法可以在创建DAG之外列出当前安装的软件包?我知道DAG不难编写,这只是因为调度程序在

此处全神贯注是当前的要求。TXT:

--trusted-host <A private repository>
--index-url <A private repository>
gnupg
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake

edit:我尝试从我们的内部存储库并获得了一些奇怪的结果。除了未连接到标准PYPI储备金之外,日志中没有报告错误(此错误也会在sup者者的工作版本中发生。但是,调度程序似乎仍然被打破。

So, I'm currently working with an Airflow installation via MWAA. I'm having this issue with a broken dependency, specifically:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

apache-airflow-providers-amazon 1.4.0 requires watchtower~=0.7.3, but you have watchtower 2.0.1 which is incompatible.

I want to get a list of the currently installed packages to do some debugging, but the broken install also seems to break the scheduler. Also, the worker logs suggest that it just keeps trying and failing to install what is listed in requirements.txt.

Besides just removing the package with a broken dependency, is there any way of listing the currently installed packages outside of just creating a DAG to do it? I know the DAG shouldn't be difficult to write, It's just that it will never be executed anyways since the scheduler is preoccupied

here is the current requirements.txt:

--trusted-host <A private repository>
--index-url <A private repository>
gnupg
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake

EDIT: I tried adding "apache-airflow-providers-amazon" from our internal repo and got some strange results. No errors reported in the log, except for a failure to connect to the standard pypi repo (this error also occurs with working versions of requirements.txt however). However, the scheduler still appears to be broken.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

涙—继续流 2025-02-03 01:57:18

我弄清楚了这个问题。 Apache-airflow-Providers-snowflake的版本与气流2.0.2不兼容。我升级到气流2.2.2,并在大多数情况下解决了问题。

I figured out the issue. The version of apache-airflow-providers-snowflake was incompatible with Airflow 2.0.2. I upgraded to airflow 2.2.2 and it fixed the issue for the most part.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文