GCP作曲家气流 - 无法使用PYPI安装包装

Question

我创建了一个具有图像版本的作曲家环境 - ＆gt; COMPOSER-2.0.13-AIRFLOW-2.2.5

尝试使用PYPI安装软件时，它会失败。 下面的详细信息：

Command :
gcloud composer environments update $AIRFLOW     --location us-east1      --update-pypi-packages-from-file requirements.txt


requirement.txt
---------------
google-api-core
google-auth
google-auth-oauthlib
google-cloud-bigquery
google-cloud-core
google-cloud-storage
google-crc32c
google-resumable-media
googleapis-common-protos
google-endpoints
joblib
json5
jsonschema
pandas
requests
requests-oauthlib


Error :
Karans-MacBook-Pro:composer_dags karanalang$ gcloud composer environments update $AIRFLOW     --location us-east1      --update-pypi-packages-from-file requirements.txt
Waiting for [projects/versa-sml-googl/locations/us-east1/environments/versa-composer3] to be updated with [projects/versa-sml-googl/locations/us-east1/operations/c23b77a9-f46b-4222-bafd-62527bf27239]..
.failed.                                                                                                                                                                                                 
ERROR: (gcloud.composer.environments.update) Error updating [projects/versa-sml-googl/locations/us-east1/environments/versa-composer3]: Operation [projects/versa-sml-googl/locations/us-east1/operations/c23b77a9-f46b-4222-bafd-62527bf27239] failed: Failed to install PyPI packages. looker-sdk 22.4.0 has requirement attrs>=20.1.0; python_version >= "3.7", but you have attrs 17.4.0.
 Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/60ac972a-8f5e-4b4f-a4a7-d81049fb19a3?project=939354532596 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation

请注意： 我有一个较旧的作曲家集群（作曲家版本-1.16.8，气流版本-1.10.15），上面的命令正常工作。 但是，它与新群集不起作用，

需要做些什么才能调试/修复此问题？

蒂亚！

Answer 1

我能够使用以下代码来完成此工作：

path = "gs://dataproc-spark-configs/pip_install.sh"

CLUSTER_GENERATOR_CONFIG = ClusterGenerator(
    project_id=PROJECT_ID,
    zone="us-east1-b",
    master_machine_type="n1-standard-4",
    worker_machine_type="n1-standard-4",
    num_workers=4,
    storage_bucket="dataproc-spark-logs",
    init_actions_uris=[path],
    metadata={'PIP_PACKAGES': 'pyyaml requests pandas openpyxl kafka-python'},
).make()


with models.DAG(
    'Versa-Alarm-Insights-UsingComposer2',
        # Continue to run DAG twice per day
        default_args=default_dag_args,
        schedule_interval='0 0/12 * * *',
        catchup=False,
        ) as dag: 

        create_dataproc_cluster = DataprocCreateClusterOperator(
          task_id="create_dataproc_cluster",  
          cluster_name="versa-composer2",
          region=REGION,
          cluster_config=CLUSTER_GENERATOR_CONFIG
     )
     

较早的命令通过从文件中读取来安装软件包，在Composer1（AirFlow 1.X）中工作，但是Composer 2.x（AirFlow 2.X）失败了

Answer 2

从错误中，很明显，您正在运行Attrs软件包的旧版本。

运行以下命令，然后尝试

PIP install attrs == 20.3.0

或

pip install attrs == 20.1.0