多重处理具有多个输入的函数

发布于 2024-10-07 23:36:08 字数 489 浏览 2 评论 0原文

在 Python 中，multiprocessing 模块可用于在一系列值上并行运行函数。例如，这会生成 f 的前 100000 个评估的列表。

def f(i):
    return i * i

def main():
    import multiprocessing
    pool = multiprocessing.Pool(2)
    ans = pool.map(f, range(100000))

    return ans

当 f 接受多个输入但只有一个变量发生变化时，是否可以完成类似的操作？例如，您将如何并行化：

def f(i, n):
    return i * i + 2*n

def main():
    ans = []
    for i in range(100000):
        ans.append(f(i, 20))

    return ans

原文

In Python the multiprocessing module can be used to run a function over a range of values in parallel. For example, this produces a list of the first 100000 evaluations of f.

def f(i):
    return i * i

def main():
    import multiprocessing
    pool = multiprocessing.Pool(2)
    ans = pool.map(f, range(100000))

    return ans

Can a similar thing be done when f takes multiple inputs but only one variable is varied? For example, how would you parallelize this:

def f(i, n):
    return i * i + 2*n

def main():
    ans = []
    for i in range(100000):
        ans.append(f(i, 20))

    return ans

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

〆一缕阳光ご 2024-10-14 23:36:08

您可以使用 functools.partial()

def f(i, n):
    return i * i + 2*n

def main():
    import multiprocessing
    pool = multiprocessing.Pool(2)
    ans = pool.map(functools.partial(f, n=20), range(100000))

    return ans

You can use functools.partial()

def f(i, n):
    return i * i + 2*n

def main():
    import multiprocessing
    pool = multiprocessing.Pool(2)
    ans = pool.map(functools.partial(f, n=20), range(100000))

    return ans

回复收藏 0 原文

〆凄凉。 2024-10-14 23:36:08

有几种方法可以做到这一点。在问题中给出的示例中，您可以定义一个包装函数

def g(i):
    return f(i, 20)

并将该包装函数传递给 map()。更通用的方法是使用一个包装器，它接受单个元组参数并将元组解包为多个参数

def g(tup):
    return f(*tup)

，或者使用等效的 lambda 表达式：lambda tup: f(*tup)。

There are several ways to do this. In the example given in the question, you could just define a wrapper function

def g(i):
    return f(i, 20)

and pass this wrapper to map(). A more general approach is to have a wrapper that takes a single tuple argument and unpacks the tuple to multiple arguments

def g(tup):
    return f(*tup)

or use a equivalent lambda expression: lambda tup: f(*tup).

回复收藏 0 原文

入画浅相思 2024-10-14 23:36:08

如果您使用我的 multiprocessing 分支（称为 pathos），您可以获得采用多个参数的池……并且还采用 lambda 函数。它的好处是您不必改变编程结构来适应并行工作。

>>> def f(i, n):
...   return i * i + 2*n
... 
>>> from itertools import repeat
>>> N = 10000
>>>
>>> from pathos.pools import ProcessPool as Pool
>>> pool = Pool()
>>>
>>> ans = pool.map(f, xrange(1000), repeat(20))
>>> ans[:10]
[40, 41, 44, 49, 56, 65, 76, 89, 104, 121]
>>>
>>> # this also works
>>> ans = pool.map(lambda x: f(x, 20), xrange(1000))
>>> ans[:10]
[40, 41, 44, 49, 56, 65, 76, 89, 104, 121]

If you use my fork of multiprocessing, called pathos, you can get pools that take multiple arguments… and also take lambda functions. The nice thing about it is that you don't have to alter your programming constructs to fit working in parallel.

>>> def f(i, n):
...   return i * i + 2*n
... 
>>> from itertools import repeat
>>> N = 10000
>>>
>>> from pathos.pools import ProcessPool as Pool
>>> pool = Pool()
>>>
>>> ans = pool.map(f, xrange(1000), repeat(20))
>>> ans[:10]
[40, 41, 44, 49, 56, 65, 76, 89, 104, 121]
>>>
>>> # this also works
>>> ans = pool.map(lambda x: f(x, 20), xrange(1000))
>>> ans[:10]
[40, 41, 44, 49, 56, 65, 76, 89, 104, 121]

回复收藏 0 原文

农村范ル 2024-10-14 23:36:08

这种技术被称为柯里化：https://en.wikipedia.org/wiki/Currying

另一种不使用 functools.partial 的方法是使用 pool.map 中的经典 map 命令：

def f(args):
   x, fixed = args
   # FUNCTIONALITY HERE

pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
pool.map(f, map(lambda x: (x, fixed), arguments))

This technique is know as Currying: https://en.wikipedia.org/wiki/Currying

Another way to do it without using functools.partial using the classical map command inside pool.map:

def f(args):
   x, fixed = args
   # FUNCTIONALITY HERE

pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1)
pool.map(f, map(lambda x: (x, fixed), arguments))

回复收藏 0 原文