为什么 Flurl.Http DownloadFileAsync / HttpClient GetAsync 中任务取消需要这么长时间
在编写一个类以并行从服务器下载图像时（采用由 DataFlow TPL 库），使用 ActionBlock 和 Flurl.Http 工具方法DownloadFileAsync，我意识到取消需要花费很多时间。由于所有下载操作共享一个 CancellationToken 我预计所有任务都会立即（或几乎）取消。实际上，如果我生成大量并行下载，取消所有任务/线程可能需要几分钟的时间。我通过将 ExecutionDataflowBlockOptions.MaxDegreeOfParallelism 属性设置为 10 来解决这个问题。这样，在任何给定时间最多都会有 10 个并发下载需要取消（这仍然不是立即执行）正如我所期待的那样）。
我创建了一个控制台 .NET 5 程序，可以单独重现该问题（没有 DataFlow TPL、ActionBlock 等）。它首先询问并发下载数（按 Enter 键获取默认值：即 100 个下载）。然后，它使用 Flurl.Http（使用 HttpClient）并行生成所有这些下载，并将 CancellationToken 传递给每个操作。然后，它等待按下某个键，然后通过调用 CancellationTokenSource.Cancel 方法取消挂起的下载。最后，它会打印一些统计信息，包括成功和失败/取消下载的数量，以及完成取消所需的时间。这是完整的代码：
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using Flurl.Http;
const string imageSource = "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png";
const int defaultCount = 100;
var watch = Stopwatch.StartNew();
int completed = 0;
int failed = 0;
Console.WriteLine("Flurl.DownloadFileAsync Cancellation test!");
Console.Write($"Number of downloads ({defaultCount}): ");
var input = Console.ReadLine();
if (!int.TryParse(input, out var count))
count = defaultCount;
Console.WriteLine($"Will spawn {count} parallel downloads of '{imageSource}'");
CancellationTokenSource cts = new CancellationTokenSource();
List<Task> tasks = new();
for (int i = 0; i < count; i++)
tasks.Add(Download(i));
Console.WriteLine("Hit anything to cancel...");
Console.ReadKey(true);
log("Cancelling pending downloads");
var cancelMoment = watch.Elapsed;
cts.Cancel();
Task.WaitAll(tasks.ToArray());
log("Downloads cancelled. Program ended!");
Console.WriteLine($"### Total cancellation time: {watch.Elapsed - cancelMoment} -> Completed: {completed}, Failed/Cancelled: {failed}");
async Task Download(int i) {
var fn = $"test_{i}.png";
try {
await imageSource.DownloadFileAsync(fn, cancellationToken: cts.Token);
Interlocked.Increment(ref completed);
log($"DONE: {fn}");
} catch(Exception e) {
Interlocked.Increment(ref failed);
log($"# ERROR: {fn}/r/n >> {e.Message}");
}
}
void log(string s) => Console.WriteLine($"{watch.Elapsed}- {s}");
最让我印象深刻的是，允许所有下载完成（即使我输入 1000 个下载）比取消操作更快。我不知道是否存在任何类型的死锁（这会导致操作在锁定超时后结束），或者这些下载的取消是否只是被破坏。我找不到这个问题的良好解释或解决方案。
要重现该问题，您必须在所有下载完成之前按某个键取消挂起的下载。如果时机正确，您可以让几次下载成功。如果您点击得太快，您将取消所有下载。如果您等待太久，所有下载都将已完成。
此运行产生以下结果：
花了超过 55 秒的时间才取消了 99 个待处理的操作。如果我只是允许所有下载完成，则比取消相同操作所需的时间要少很多。
更新
我已经完全删除了 Flurl 并直接使用 HttpClient 下载文件，问题仍然存在。我已将 Downlod 方法更改为以下内容：
async Task Download(int i) {
var fn = $"test_{i}.png";
try {
var r = await client.GetAsync(imageSource, cancellationToken: cts.Token);
using var httpStm = await r.Content.ReadAsStreamAsync(cts.Token);
var file = new FileInfo(fn);
using var f = file.OpenWrite();
await httpStm.CopyToAsync(f, cts.Token);
Interlocked.Increment(ref completed);
log($"DONE: {fn}");
} catch(Exception e) {
Interlocked.Increment(ref failed);
log($"# ERROR: {fn}/r/n >> {e.Message}");
}
}
结果与基于 FLURL 的实现相同（毕竟，Furl.Http 只是 HttpClient 的包装器）。
更新 2
我已将 Download 方法更改为简单地等待可取消的 Task.Delay ，并且 100 个操作的取消时间现在约为 2 秒。虽然它更快，但它不是瞬时的，并且根据屏幕上日志的时间，在我看来，取消是按顺序触发的，而不是并行/立即触发的。此下载的代码为：
async Task Download(int i) {
var fn = $"test_{i}.png";
try {
await Task.Delay(TimeSpan.FromMinutes(1), cts.Token);
Interlocked.Increment(ref completed);
log($"DONE: {fn}");
} catch (Exception e) {
Interlocked.Increment(ref failed);
log($"# ERROR: {fn}/r/n >> {e.Message}");
}
}
以下屏幕截图显示了上述代码的结果：
有人对此有好的解释或解决方案吗？
While writing a class to download images from a server in parallel (employing a consumer/producer pattern backed by the DataFlow TPL library), using an ActionBlock and Flurl.Http facility method DownloadFileAsync, I've realized that cancellation was taking a lot of time. Since all download operations share a CancellationToken I was expecting that all tasks would cancel instantly (or almost). In reality, if I spawn lots of parallel downloads, cancellation of all tasks/threads can take minutes. I've worked around that by setting the ExecutionDataflowBlockOptions.MaxDegreeOfParallelism property to 10. In this way there'll be at most 10 concurrent downloads to cancel at any given time (which, still, is not an immediate action as I was expecting).
I've created a console .NET 5 program that reproduces the issue in isolation (without the DataFlow TPL, ActionBlock, etc.). It starts by asking the number of concurrent downloads (press enter for the default: which is 100 downloads). Then it spawn all these downloads in parallel using Flurl.Http (which uses HttpClient), passing a CancellationToken to each operation. Then it waits for a key to be pressed and then it cancels the pending downloads by calling the CancellationTokenSource.Cancel method. At the end, it prints some statistics including the number of successful and failed/cancelled downloads, plus the time it took for the cancellation to complete. Here is the full code:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
using Flurl.Http;
const string imageSource = "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png";
const int defaultCount = 100;
var watch = Stopwatch.StartNew();
int completed = 0;
int failed = 0;
Console.WriteLine("Flurl.DownloadFileAsync Cancellation test!");
Console.Write(quot;Number of downloads ({defaultCount}): ");
var input = Console.ReadLine();
if (!int.TryParse(input, out var count))
count = defaultCount;
Console.WriteLine(quot;Will spawn {count} parallel downloads of '{imageSource}'");
CancellationTokenSource cts = new CancellationTokenSource();
List<Task> tasks = new();
for (int i = 0; i < count; i++)
tasks.Add(Download(i));
Console.WriteLine("Hit anything to cancel...");
Console.ReadKey(true);
log("Cancelling pending downloads");
var cancelMoment = watch.Elapsed;
cts.Cancel();
Task.WaitAll(tasks.ToArray());
log("Downloads cancelled. Program ended!");
Console.WriteLine(quot;### Total cancellation time: {watch.Elapsed - cancelMoment} -> Completed: {completed}, Failed/Cancelled: {failed}");
async Task Download(int i) {
var fn = quot;test_{i}.png";
try {
await imageSource.DownloadFileAsync(fn, cancellationToken: cts.Token);
Interlocked.Increment(ref completed);
log(quot;DONE: {fn}");
} catch(Exception e) {
Interlocked.Increment(ref failed);
log(quot;# ERROR: {fn}/r/n >> {e.Message}");
}
}
void log(string s) => Console.WriteLine(quot;{watch.Elapsed}- {s}");
What strikes me the most is that allowing all downloads to complete (even if I input 1000 downloads) is faster than cancelling the operation. I don't know if there is any kind of deadlock going on (which causes the operations to end after a lock timeout), or if the cancellation of these downloads are simply broken. I couldn't find a good explanation or solution for this issue.
To reproduce the issue you must hit a key to cancel the pending downloads before all downloads complete. If you time it right you can allow a few downloads to succeed. If you hit it too fast, you'll cancel all downloads. If you wait too long all downloads will have completed already.
This run resulted in the following results:
It took whooping 55+ seconds to cancel the 99 pending operations. If I simply allow all downloads to complete, it takes a LOT less than the time it takes to cancel the same operations.
UPDATE
I've removed Flurl altogether and used HttpClient directly to download the file and the problem persists. I've changed the Downlod method to the following:
async Task Download(int i) {
var fn = quot;test_{i}.png";
try {
var r = await client.GetAsync(imageSource, cancellationToken: cts.Token);
using var httpStm = await r.Content.ReadAsStreamAsync(cts.Token);
var file = new FileInfo(fn);
using var f = file.OpenWrite();
await httpStm.CopyToAsync(f, cts.Token);
Interlocked.Increment(ref completed);
log(quot;DONE: {fn}");
} catch(Exception e) {
Interlocked.Increment(ref failed);
log(quot;# ERROR: {fn}/r/n >> {e.Message}");
}
}
Results are the same as with the FLURL based implementation (after all, Flurl.Http is just a wrapper around HttpClient).
UPDATE 2
I've changed the Download method to simply wait on a cancellable Task.Delay and the cancellation time of 100 operations is now around 2 seconds. While it's faster, it's not instantaneous, and by the timing of the logs on the screen it seems to me that the cancellations are being triggered sequentially and not in parallel/at once. The code for this download is:
async Task Download(int i) {
var fn = quot;test_{i}.png";
try {
await Task.Delay(TimeSpan.FromMinutes(1), cts.Token);
Interlocked.Increment(ref completed);
log(quot;DONE: {fn}");
} catch (Exception e) {
Interlocked.Increment(ref failed);
log(quot;# ERROR: {fn}/r/n >> {e.Message}");
}
}
The following screenshot shows the result with the above code:
Does anyone have a good explanation or solution for that?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
我没有研究 Flurl.Http 但我认为取消没有得到充分支持。如果您使用 .Net6，则可以使用 WaitAsync。如果没有，您可以创建一个任务，您可以取消并使用 Task.WhenAny 中的下载任务等待该任务，如下所示：
缺点是我不知道如何真正取消与下载相关的工作。
I have not looked into Flurl.Http but I assume cancellation is not sufficiently supported. If you are using .Net6 you could use WaitAsync. If not you might create a Task that you can cancel and await this task with the download task within Task.WhenAny like so:
The downside is that I have no idea how to really cancel the work associated with the downloads.
如果您将取消令牌单独传递给任务而不是父
Task.WaitAll()，则在调用cts.Cancel()后，其他任务将不会有任何取消令牌知道取消请求，直到为每个请求调用Download()为止。将令牌传递给父级，因此它就像所有任务的协调器：
然后，一旦取消触发，剩余的任务将不需要执行，从而节省了取消时间。
请注意，取消时间要短得多，因为我们只需要取消当时正在执行的任务，而不是所有任务都自行收到取消通知。我们完成了 70 个任务，其中 4 个被取消，剩下的被丢弃，导致取消时间更快。
If you are passing the cancellation token individually to the tasks but not to the parent
Task.WaitAll(), after you invokects.Cancel(), other tasks will have no knowledge of the cancellation request untilDownload()is invoked for each of them.Pass the token to the parent, so it acts like an orchestrator for all tasks:
Then, once the cancellation fires, remaining tasks will not be required to executed, saving you cancellation time.
Mind you cancellation time is much smaller since we only need to cancel the tasks being executed at that moment, instead of all tasks getting notified of the cancellation by themselves. We completed 70 tasks, 4 of them are cancelled, and the remaining was just dropped, resulting much faster cancellation time.