在MWAA环境中使用S3FiletransFormerator

Question

我正在尝试在MWAA环境中使用S3FiletransFormerator，但是我在脚本文件中缺乏许可：

PermissionError: [Errno 13] Permission denied

我试图在任务前将带有CHMOD命令的bash运算符添加，但没有成功。

有没有人在MWAA中使用过S3FiletransFormerator？

Answer 1

Hi!

Unfortunately with MWAA the worker containers are both ephemeral and limited to user level access. The S3 operator should work with .sh files added to the /dags folder and referred to in the operator as /usr/local/airflow/dags/my_script.sh. The alternative would be to use the contents of your .py file from a Python operator and use the S3Hook to retrieve and store the file.

Thanks!

参考： https> https'> https'> https：// repost。 AWS/QUASION/QUVSBZDS_NQTG7JSXCQ1DJQ/s-3-file-transform-operator-permission-permission-permission-indied-on-script

Answer 2

我尝试了这一点，但第一步又有了另一个错误，

from airflow.operators.bash import BashOperator
from airflow.operators.s3_file_transform_operator import S3FileTransformOperator

with DAG(...) as dag:
    chmod = BashOperator(
        task_id="chmod",
        bash_command="chmod +x /usr/local/airflow/dags/transform.py"
    )
    transform = S3FileTransformOperator(
        task_id="transform",
        source_s3_key="s3://bucket/path/to/sample.csv",
        dest_s3_key="s3://bucket/path/to/result.csv",
        transform_script="/usr/local/airflow/dags/transform.py",
    )

    chmod >> expose

chmod: changing permissions of ‘/usr/local/airflow/dags/transform.py’: Read-only file system

AWS的另一项引用表明可以运行Bash脚本转换，但是我对Python没有任何成功。