在存储帐户中,从Azure函数到容器的慢速上传速度

发布于 2025-02-07 23:14:30 字数 1774 浏览 1 评论 0原文

我经历的上传速度较慢,而不是达到项目目标。我们需要将最高0.5GB的文件从AZ功能服务上传到我们的一个存储帐户之一。这个过程花费的时间超过了我们预期的时间(> 10分钟)。引起我们注意的另一个问题是文件大小和上传时间之间关系的非线性。一个160MB的文件需要1.5分钟才能上传,但290MB超过10分钟。有人知道为什么会发生这种情况吗?我们没有通过社区/文档来弄清楚这一点。

我的 function.json 文件看起来像这样:

{
  "scriptFile": "az_func.py",
  "bindings": [
    {
      "name": "mytimer",
      "type": "timerTrigger",
      "direction": "in",
      "schedule": "0 0 0 * * 2"
    },
    {
      "name": "$return",
      "type": "blob",
      "path": "cvmdailyreports/test_func_json.csv",
      "connection": "AzureWebJobsCVMDataStorageConn",
      "direction": "out"
    }
  ]
}

az_func.py 脚本只是从网站上删除一些数据,操纵一些数据并返回CSV(此过程不进行太长了,所以这不是不受欢迎的来源)。

任何人都知道,AZ功能需要这么长时间将文件上传到AZ存储帐户还是更快的方法是正常的?

az_func.py 代码如下:

import datetime
import logging

import azure.functions as func

from src.data.import_data import import_daily_reports
from src.data.transform_data import transform_daily_reports


def main(mytimer: func.TimerRequest):
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()

    if mytimer.past_due:
        logging.info('The timer is past due!')

    logging.info('Python timer trigger function ran at %s', utc_timestamp)

    csv = transform_daily_reports(import_daily_reports(n=8)).to_csv(index=False)

    logging.info('The csv was created, sending to upload now.')

    return csv

转换和导入功能是自定义功能,可以从网站导入某些数据并将其连接到数据范围中。如前所述,它们毫无问题地运行,正如应用程序见解的日志中可以看到的: invocation详细信息 - 超时错误logs

函数应用程序在linux上运行,带有runtime版本

4.0当前使用的消费计划(超时有10分钟的限制),我已经配置了 host.json 文件以使最大超时。

谢谢!

I'm experiencing a slower upload speed than I need to meet a project goals. We need to upload files up to 0.5GB from an AZ Function service to one of our storage accounts. This process is taking more time than we expect (>10min). Another issue that got our attention is the non-linearity of the relationship between file size and upload time. A 160MB file is taking 1.5min to upload but a 290MB exceeds 10min. Does anyone understand why this is happening? We didn't manage to figure it out through the community/docs.

My function.json file looks like this:

{
  "scriptFile": "az_func.py",
  "bindings": [
    {
      "name": "mytimer",
      "type": "timerTrigger",
      "direction": "in",
      "schedule": "0 0 0 * * 2"
    },
    {
      "name": "$return",
      "type": "blob",
      "path": "cvmdailyreports/test_func_json.csv",
      "connection": "AzureWebJobsCVMDataStorageConn",
      "direction": "out"
    }
  ]
}

The az_func.py script just scrapes some data from a website, manipulates some data and returns a csv (this process doesn't take too long, so it is not the source of inneficiency).

Anyone knows if it is normal for AZ functions to take that long to upload files to AZ Storage Accounts or if there is a faster way to do it?

The az_func.py code is the following:

import datetime
import logging

import azure.functions as func

from src.data.import_data import import_daily_reports
from src.data.transform_data import transform_daily_reports


def main(mytimer: func.TimerRequest):
    utc_timestamp = datetime.datetime.utcnow().replace(
        tzinfo=datetime.timezone.utc).isoformat()

    if mytimer.past_due:
        logging.info('The timer is past due!')

    logging.info('Python timer trigger function ran at %s', utc_timestamp)

    csv = transform_daily_reports(import_daily_reports(n=8)).to_csv(index=False)

    logging.info('The csv was created, sending to upload now.')

    return csv

The transform and import functions are custom functions to import some data from a website and to concatenate that into a dataframe. As stated before, they run without a problem, as can be seen in the logs from the App Insights:
Invocation Details - Timeout Error Logs

The Function App is running on linux, with runtime version 4.0

We are currently using the Consumption plan, which has the 10 mins limitation for timeout, and I already configured the host.json file to have the max timeout possible.

Thanks!

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文