在MWAA中列出包裹,并带有一个损坏的调度程序
因此,我目前正在使用MWAA进行气流安装。我的依赖性破裂的问题是:
错误:PIP的依赖项解析器当前未考虑所有已安装的软件包。此行为是以下依赖性冲突的根源。
apache-airFlow-Providers-amazon 1.4.0需要watchtower〜 = 0.7.3,但是您的watchtower 2.0.1不兼容。
我想获取当前安装的软件包的列表以进行一些调试,但是损坏的安装似乎也打破了调度程序。另外,工人日志表明,它只是不断尝试和无法安装requients.txt中列出的内容。
除了仅以破坏的依赖性删除软件包外,是否有任何方法可以在创建DAG之外列出当前安装的软件包?我知道DAG不难编写,这只是因为调度程序在
此处全神贯注是当前的要求。TXT:
--trusted-host <A private repository>
--index-url <A private repository>
gnupg
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake
edit:我尝试从我们的内部存储库并获得了一些奇怪的结果。除了未连接到标准PYPI储备金之外,日志中没有报告错误(此错误也会在sup者者的工作版本中发生。但是,调度程序似乎仍然被打破。
So, I'm currently working with an Airflow installation via MWAA. I'm having this issue with a broken dependency, specifically:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow-providers-amazon 1.4.0 requires watchtower~=0.7.3, but you have watchtower 2.0.1 which is incompatible.
I want to get a list of the currently installed packages to do some debugging, but the broken install also seems to break the scheduler. Also, the worker logs suggest that it just keeps trying and failing to install what is listed in requirements.txt.
Besides just removing the package with a broken dependency, is there any way of listing the currently installed packages outside of just creating a DAG to do it? I know the DAG shouldn't be difficult to write, It's just that it will never be executed anyways since the scheduler is preoccupied
here is the current requirements.txt:
--trusted-host <A private repository>
--index-url <A private repository>
gnupg
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake
EDIT: I tried adding "apache-airflow-providers-amazon" from our internal repo and got some strange results. No errors reported in the log, except for a failure to connect to the standard pypi repo (this error also occurs with working versions of requirements.txt however). However, the scheduler still appears to be broken.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
![扫码二维码加入Web技术交流群](/public/img/jiaqun_03.jpg)
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
我弄清楚了这个问题。 Apache-airflow-Providers-snowflake的版本与气流2.0.2不兼容。我升级到气流2.2.2,并在大多数情况下解决了问题。
I figured out the issue. The version of apache-airflow-providers-snowflake was incompatible with Airflow 2.0.2. I upgraded to airflow 2.2.2 and it fixed the issue for the most part.