cloudblockblob.putblockasync给出错误:system.invalidoperationException:流的长度超过允许的长度

发布于 2025-01-29 18:42:03 字数 1971 浏览 4 评论 0 原文

我正在尝试使用 CloudBlockBlob.putblockasync 方法上传大量对象。 在调用该方法时,我将获得 system.invalidoperationException:流的长度超过允许的长度。

我单独验证了块的内存流本身没有给出OOM或任何其他错误。 (IE块对象适合在内存流中)。 cloudBlockBlob.putblockasync 方法是否有任何流尺寸限制? 我在任何文档中都找不到。我正在使用 microsoft.windowsazure.storage 9.3.2,用于.net 4.5

以下是代码片段: (它在数据的测试环境中起作用,但在生产中破裂)

foreach (IList<MyDTO> chunk in chunkedDTOs)
{
   
    string blockId = Convert.ToBase64String(Guid.NewGuid().ToByteArray());

    using (var stream = new MemoryStream())
    using (var sw = new StreamWriter(stream))
    using (JsonWriter writer = new JsonTextWriter(sw))
    {
        var serializer = JsonSerializer.Create();
        serializer.Serialize(writer, chunk);
        await sw.FlushAsync();
        stream.Position = 0;

        
        await cloudBlockBlob.PutBlockAsync(blockId, stream, contentMD5: null, accessCondition: null,
            options: null, operationContext: null, cancellationToken);
    }

    blockIds.Add(blockId); // ids collected to commit later
}

错误堆栈跟踪:

System.InvalidOperationException: The length of the stream exceeds the permitted length.
   at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndPutBlock(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 2079
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 121
--- End of stack trace from previous location where exception was thrown ---

I am trying to upload a large collection of objects in chunks using CloudBlockBlob.PutBlockAsync method.
While calling the method, I am getting System.InvalidOperationException: The length of the stream exceeds the permitted length.

I verified separately that the memory stream itself for a chunk is not giving OOM or any other errors. (i.e. chunk object is fitting in a memory stream). Is there any stream size limit for the CloudBlockBlob.PutBlockAsync method ?
I could not find it in any documentation. I am using Microsoft.WindowsAzure.Storage 9.3.2 for .Net 4.5

Below is the code snippet:
(It works in test environments where data is very less, but it breaks in production)

foreach (IList<MyDTO> chunk in chunkedDTOs)
{
   
    string blockId = Convert.ToBase64String(Guid.NewGuid().ToByteArray());

    using (var stream = new MemoryStream())
    using (var sw = new StreamWriter(stream))
    using (JsonWriter writer = new JsonTextWriter(sw))
    {
        var serializer = JsonSerializer.Create();
        serializer.Serialize(writer, chunk);
        await sw.FlushAsync();
        stream.Position = 0;

        
        await cloudBlockBlob.PutBlockAsync(blockId, stream, contentMD5: null, accessCondition: null,
            options: null, operationContext: null, cancellationToken);
    }

    blockIds.Add(blockId); // ids collected to commit later
}

Error Stack trace :

System.InvalidOperationException: The length of the stream exceeds the permitted length.
   at Microsoft.WindowsAzure.Storage.Core.Util.StorageAsyncResult`1.End() in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\StorageAsyncResult.cs:line 77
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.EndPutBlock(IAsyncResult asyncResult) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlockBlob.cs:line 2079
   at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 121
--- End of stack trace from previous location where exception was thrown ---

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(1

橘香 2025-02-05 18:42:03

源泉说,作为 Microsoft.windowsazure.score.9.3.2 是一个传统库,它可能支持每个块100 MIB。由于内存流不仅如此,因此请求失败了。


我减少了块大小,因此内存流的大小比100 MIB少,并且有效。
该库最后一次更新为2018年11月27日的9.3.3。因此,更新引用是没有意义的。

”稍后将在使用microsoft.azure.storage.blob库中调查。

Fount that as the Microsoft.WindowsAzure.Storage 9.3.2 is a legacy library, it might be supporting 100 MiB per block. As the memory stream was more than this, the request was failing.

https://learn.microsoft.com/en-us/azure/storage/blobs/scalability-targets#scale-targets-for-blob-storage
enter image description here

I reduced the chunk size so that the memory stream size is lesser that 100 MiB, and that worked.
This library was last updated to 9.3.3 on 11/27/2018. So there is no point in updating the references.
https://www.nuget.org/packages/WindowsAzure.Storage/

I would later investigate in using the Microsoft.Azure.Storage.Blob library instead.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文