如何使我的代码与dask并行化?

发布于 2025-01-20 02:39:21 字数 1147 浏览 4 评论 0 原文

首先导入一些软件包:

import numpy as np
from dask import delayed

假设我有两个numpy数组:

a1 = np.ones(5000000)
a2 = np.ones(8000000)

我想显示两个阵列的总和和长度,并且该功能显示为:

def sum(x):
  result = 0
  for data in x:
      result = result + data
  return result, len(x)

def get_result(x, y):
  return x, y

我在Colab中有两个示例,顺序示例就是这样:

%%time
result1 = sum(a1)
result2 = sum(a2)
result = get_result(result1, result2)
print(result)

输出是:

((5000000.0, 5000000), (8000000.0, 8000000))
CPU times: user 1.41 s, sys: 3.7 ms, total: 1.42 s
Wall time: 1.42 s

但是,我想计算这些值 parallelly

result1 = delayed(sum)(a1)
result2 = delayed(sum)(a2)
result = delayed(get_result)(result1, result2)
result = result.compute()
print(result)

输出是:

Delayed('get_result-ffbb6330-1014-42c5-b625-06e3e66a56ed')
CPU times: user 1.42 s, sys: 7.97 ms, total: 1.42 s
Wall time: 1.43 s

为什么第二个程序不起作用?因为壁时间两个示例几乎是相同的

First import some packages:

import numpy as np
from dask import delayed

Suppose I have two NumPy arrays:

a1 = np.ones(5000000)
a2 = np.ones(8000000)

I would like to show the sum and length of the two arrays, and the functions are shown as:

def sum(x):
  result = 0
  for data in x:
      result = result + data
  return result, len(x)

def get_result(x, y):
  return x, y

I have two examples in colab, the sequential example is like this:

%%time
result1 = sum(a1)
result2 = sum(a2)
result = get_result(result1, result2)
print(result)

And the output is:

((5000000.0, 5000000), (8000000.0, 8000000))
CPU times: user 1.41 s, sys: 3.7 ms, total: 1.42 s
Wall time: 1.42 s

However, I would like to compute these values parallelly.

result1 = delayed(sum)(a1)
result2 = delayed(sum)(a2)
result = delayed(get_result)(result1, result2)
result = result.compute()
print(result)

And the output is:

Delayed('get_result-ffbb6330-1014-42c5-b625-06e3e66a56ed')
CPU times: user 1.42 s, sys: 7.97 ms, total: 1.42 s
Wall time: 1.43 s

Why the second program didn't work parallelly? Because the wall time two examples are almost the same.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文