如何将日志从远程调用，停泊的芹菜工人返回到气流容器
我正在开发一个dockerized Python/Django项目，其中包括一个用于芹菜工人的容器，我一直在其中集成了现成的气流码头容器。
我的气流通过使用REDIS代理和指定的后端实例化芹菜应用程序，并通过send_task实例芹菜应用程序在预先存在的容器中成功运行芹菜任务；但是，芹菜任务进行的伐木都没有使其回到气流日志。
最初，作为我完全是气流的概念证明，我已经将其设置为运行相同的代码，通过将其暴露于气流容器并创建气流任务以在气流芹菜工人的容器上运行它。此 diD 会导致所有捕获的记录，但这绝对不是我们希望它进行架构的方式，因为这会使气流容器由于Django Project的依赖性重复而变得非常胖。
文档显示“大多数任务处理人员在完成任务后发送日志”，但我找不到更多的细节，这可能会使我知道如何在我的情况下启用同样的信息。
远程运行芹菜任务时，有什么方法可以将这些日志恢复到气流？
I am working on a Dockerized Python/Django project including a container for Celery workers, into which I have been integrating the off-the-shelf airflow docker containers.
I have Airflow successfully running celery tasks in the pre-existing container, by instantiating a Celery app with the redis broker and back end specified, and making a remote call via send_task; however none of the logging carried out by the celery task makes it back to the Airflow logs.
Initially, as a proof of concept as I am completely new to Airflow, I had set it up to run the same code by exposing it to the Airflow containers and creating airflow tasks to run it on the airflow celery worker container. This did result in all the logging being captured, but it's definitely not the way we want it architectured, as this makes the airflow containers very fat due to the repetition of dependencies from the django project.
The documentation says "Most task handlers send logs upon completion of a task" but I wasn't able to find more detail that might give me a clue how to enable the same in my situation.
Is there any way to get these logs back to airflow when running the celery tasks remotely?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
与其“将日志返回到气流”，而是一种易于实现的替代方案（因为气流本来支持它）是激活远程日志记录。这样，所有工人的所有日志都将在S3上最终出现，并且网络服务器将自动获取它们。
以下说明了如何使用S3后端配置远程记录。其他选项（例如Google Cloud Storage，Elastic）可以类似地实现。
remote_loggingtotrueinairflow.cfg＆nbsp;＆nbsp;＆nbsp;＆nbsp;＆nbsp; nbsp; nbsp; nbsp; nbsp;如果需要，也可以通过WebServer GUI创建Connectino。
airflow_conn_ {your_connection_name}可用。连接名称的示例remote_logs_s3：remote_log_conn_id在airflow.cfgairflow.cfgto所需的存储桶/前缀。示例：this so so so 在远程记录上更深入地触摸。
如果需要调试，请查看本地的任何工人日志（即工人内部）应该有所帮助。
Instead of "returning the logs to Airflow", an easy-to-implement alternative (because Airflow natively supports it) is to activate remote logging. This way, all logs from all workers would end up e.g. on S3, and the webserver would automatically fetch them.
The following illustrates how to configure remote logging using an S3 backend. Other options (e.g. Google Cloud Storage, Elastic) can be implemented similarly.
remote_loggingtoTrueinairflow.cfgIt is also possible to create the connectino through the webserver GUI, if needed.
AIRFLOW_CONN_{YOUR_CONNECTION_NAME}is available. Example for connection nameREMOTE_LOGS_S3:remote_log_conn_idto the connection name (e.g.REMOTE_LOGS_S3) inairflow.cfgremote_base_log_folderinairflow.cfgto the desired bucket/prefix. Example:This related SO touches deeper on remote logging.
If debugging is needed, looking into any worker logs locally (i.e., inside the worker) should help.