如何使用numpy转换以下处理

发布于 2025-01-18 12:17:25 字数 1828 浏览 0 评论 0原文

我正在尝试改进部分代码，该代码会显着减慢整个脚本的速度，甚至使其不可行。具体来说，这段代码是：

for vectors1 in EC1:
    for vectors2 in EC2:
        r = np.add(vectors1, vectors2)
        for vectors3 in CDC:
            result = np.add(r, vectors3).tolist()
            if result not in states:  # This is what makes it very slow
                states.append(result)

EC1、EC2 和 CDC 是包含列表列表元素的列表，作为一次迭代的示例，我们得到：

vectors1: [[2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0]]

vectors2: [[0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]

vectors3: [[0, 0, 0], [0, 0, 0], [2, 1, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [2, 1, 0], [2, 1, 0]]

result:   [[2, 0, 0], [2, 0, 0], [2, 1, 0], [2, 0, 0], [2, 0, 0], [2, 0, 0], [2, 0, 0], [4, 1, 0], [2, 1, 0]]

注意向量 1、向量 2 和向量 3 如何对应于 EC1、EC2 和 CDC 中的一个元素CDC 还分别说明了“结果”如何是向量 1、向量 2 和向量 3 的总和，因此之前的向量不能以任何方式更改或排序，否则会更改“结果”变量的预期结果。

在前两个循环中，EC1 和 EC2 中的每个项目都被求和，稍后将先前的结果与 CDC 中的项目求和。为了将来自 EC1 和 EC2 的列表列表以及随后的先前结果 ('r') 与来自 CDC 的列表列表相加，我使用 numpy.add()。最后，我将“结果”重新转换回列表。所以基本上我将列表列表作为来自 EC1、EC2 和 CDC 的元素进行管理。

问题是我必须处理数十万（接近 1M）的结果，并且必须检查状态列表中是否存在结果会大大减慢速度，特别是因为状态列表随着处理更多结果而增长。

我试图通过将所有内容作为 numpy 数组进行管理来保持在 numpy 世界中。首先将状态声明为：

states = np.empty([9, 3], int)

然后，将结果 numpy 数组连接到状态 numpy 数组，事先检查状态中是否已存在：

for vectors1 in EC1:
    for vectors2 in EC2:
        r = np.add(vectors1, vectors2)
        for vectors3 in CDC:
            result = np.add(r, vectors3)
            if not np.isin(states, result).any():
                np.concatenate(states, result, axis=0)

但肯定我做错了什么，因为结果没有连接到状态，我也尝试过但没有成功：

np.append(states, result, axis=0)

可以这可以以某种方式并行化吗？

原文

I am trying to improve a part of code that is slowing down the whole script significantly, right to the point of making it unfeasible. In particular the piece of code is:

for vectors1 in EC1:
    for vectors2 in EC2:
        r = np.add(vectors1, vectors2)
        for vectors3 in CDC:
            result = np.add(r, vectors3).tolist()
            if result not in states:  # This is what makes it very slow
                states.append(result)

EC1, EC2 and CDC are lists that contains as elements, lists of lists, as an example of one iteration, we get:

vectors1: [[2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0]]

vectors2: [[0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]

vectors3: [[0, 0, 0], [0, 0, 0], [2, 1, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [2, 1, 0], [2, 1, 0]]

result:   [[2, 0, 0], [2, 0, 0], [2, 1, 0], [2, 0, 0], [2, 0, 0], [2, 0, 0], [2, 0, 0], [4, 1, 0], [2, 1, 0]]

Notice how vectors1, vectors2 and vectors3 correspond to one element from EC1, EC2 and CDC respectively, also how 'result' is the summation from vectors1, vectors2 and vectors3, hence the previous vectors cannot be altered in any manner or sorted, otherwise it would change the expected result from the 'result' variable.

In the first two loops each item in EC1 and EC2 are summed, for later on sum up the previous result with items in CDC. To sum the list of lists from EC1 and EC2 and later on the previous result ('r') with the list of lists from CDC I use numpy.add(). Finally, I reconvert 'result' back to list. So Basically I am managing lists of lists as elements from EC1, EC2 and CDC.

The problem is that I must deal with hundreds of thousands (close to 1M) of results and having to check if a result exists in states list is slowing things drastically, specially since states list grows as more results are processed.

I've tried to keep inside the numpy world by managing everything as numpy arrays. First declaring states as:

states = np.empty([9, 3], int)

Then, concatenating the result numpy array to states numpy array, prior checking if already exists in states:

for vectors1 in EC1:
    for vectors2 in EC2:
        r = np.add(vectors1, vectors2)
        for vectors3 in CDC:
            result = np.add(r, vectors3)
            if not np.isin(states, result).any():
                np.concatenate(states, result, axis=0)

But definitely I am doing something wrong because result is not being concatenated to states, I've also tried without success:

np.append(states, result, axis=0)

Could this be parallelized in some way?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

哆兒滾 2025-01-25 12:17:25

您可以使用广播仅在 numpy 中进行求和

res = ((EC1[:,None,:] + EC2).reshape(-1, 1, 3) + CDC).reshape(-1, 3)

鉴于 EC1、EC2 和 CDC 是数组，

。之后，您可以使用过滤掉重复项，

np.unique(res, axis=0)

但是像卢卡斯一样，我强烈建议您事先过滤数组。对于示例数组，会将 res 中的行数从 729 减少到 8。

You can do the sums solely in numpy by using broadcasting

res = ((EC1[:,None,:] + EC2).reshape(-1, 1, 3) + CDC).reshape(-1, 3)

given that EC1, EC2 and CDC are arrays.

Afterwards you can filter out the duplicates with

np.unique(res, axis=0)

But like Lucas, I would strongly advise you to filter the arrays beforehand. For your example arrays that would shrink the number of rows in res from 729 to 8.

回复收藏 0 原文

简单气质女生网名 2025-01-25 12:17:25

我不确定您正在使用的数据有多大，但这可能会有所加快：

EC1 = [[2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0]]
EC2 = [[0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]
CDC = [[0, 0, 0], [0, 0, 0], [2, 1, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [2, 1, 0], [2, 1, 0]]
EC1.sort()
EC2.sort()
CDC.sort()
unique_triples = dict()

for v1 in EC1:
    for v2 in EC2:
        for v3 in CDC:
            if str(v1)+str(v2)+str(v3) not in unique_triples: # list not hashable but strings are
                unique_triples[str(v1)+str(v2)+str(v3)] = list(np.add(np.add(v1, v2), v3))

基本想法是删除（ec1，ec2）的重复三元组。，cdc）条目，仅在唯一的三元组上进行添加，对列表进行排序，以便在词典上订购词典，

词典具有O（1）查找，以便这些查找更快。

这是否更快，可能取决于多大的数量以及三元组的独特值 - 正在处理的数据。

3矢量总和是字典的值，例如
list（unique_triples.values（））对我提供：

>>> list(unique_triples.values())
[[0, 0, 0], [2, 1, 0], [2, 0, 0], [4, 1, 0], [2, 0, 0], [4, 1, 0], [4, 0, 0], [6, 1, 0]]

我没有在此处的原始列表中删除重复项。如果您要查看的应用程序允许，则在ec1，ec2和cdc ec2 中删除这些重复项也可能有益。强> 在值上迭代。

I'm not sure how large the data are that you are working with but this may speed things up somewhat:

EC1 = [[2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0]]
EC2 = [[0, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [2, 0, 0], [2, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]
CDC = [[0, 0, 0], [0, 0, 0], [2, 1, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [2, 1, 0], [2, 1, 0]]
EC1.sort()
EC2.sort()
CDC.sort()
unique_triples = dict()

for v1 in EC1:
    for v2 in EC2:
        for v3 in CDC:
            if str(v1)+str(v2)+str(v3) not in unique_triples: # list not hashable but strings are
                unique_triples[str(v1)+str(v2)+str(v3)] = list(np.add(np.add(v1, v2), v3))

The basic idea is to remove duplicate triples of (EC1,EC2, CDC) entries and only do the additions on unique triples, sort the lists so that they are ordered lexicographically

A dictionary has O(1) lookups so these lookups are (maybe) faster.

Whether this is faster or not might depend on how large-and how many unique values of triples-the data are that are being processed.

The 3-vector sums are the values of the dictionary, e.g.
list(unique_triples.values()) for me gives:

>>> list(unique_triples.values())
[[0, 0, 0], [2, 1, 0], [2, 0, 0], [4, 1, 0], [2, 0, 0], [4, 1, 0], [4, 0, 0], [6, 1, 0]]

I did not remove the duplicates in the original lists of lists here. If the application you are looking at allows, it is also likely beneficial to remove these duplicates in EC1, EC2, and CDC before iterating over the values.

回复收藏 0 原文

~没有更多了~