什么是组合数据框架矢量创建的最快方法？

Question

如何通过某个矢量化过程创建Master dataframe？如果不可能，执行此操作的时间效率最高（不关心内存）方法是什么？

可以更换前面的循环吗？

如您所见，Compinations很快产生了很大的数量，因此我需要一种快速的方法来生成此数据框架。

请参阅下面的最低可再现示例：

%%time

import pandas as pd
import string
import numpy as np
from itertools import combinations

# create dummy data
cols = list(string.ascii_uppercase)
dummy = pd.DataFrame()
for col in cols:
    dummy = dummy.append([[col, 0] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
    dummy = dummy.append([[col, 1] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
    dummy = dummy.append([[col, 2] + np.random.randint(2, 100, size=(1, 10)).tolist()[0]])
dummy.columns=['name', 'id', 'v1', 'v2', 'v3', 'v4', 'v5', 'v1', 'v6', 'v7', 'v8', 'v9']

# create all possible unique combinations
combos = list(combinations(cols, 2))

# generate DataFrame with all combinations
master = pd.DataFrame()
for i, combo in enumerate(combos):
    A = dummy[dummy.name == combo[0]]
    B = dummy[dummy.name == combo[1]]
    joined = pd.merge(A, B, on=["id"], suffixes=('_A', '_B'))
    joined = joined.sort_values("id")
    joined['pair_id'] = i
    master = pd.concat([master, joined])

输出：

CPU times: total: 1.8 s
Wall time: 1.8 s

谢谢！

Answer 1

由于您的数据是结构性的，因此您可以掉落到Numpy来利用矢量化操作。

names = list(string.ascii_uppercase)
ids = [0, 1, 2]
columns = pd.Series(["v1", "v2", "v3", "v4", "v5", "v1", "v6", "v7", "v8", "v9"])

# Generate the random data
data = np.random.randint(2, 100, (len(names), len(ids), len(columns)))

# Pair data for every 2-combination of names
arr = [np.hstack([data[i], data[j]]) for i,j in combinations(range(len(names)), 2)]

# Assembling the data to final dataframe
idx = pd.MultiIndex.from_tuples([
    (p,a,b,i) for p, (a, b) in enumerate(combinations(names,2)) for i in ids
], names=["pair_id", "name_A", "name_B", "id"])
cols = pd.concat([columns + "_A", columns + "_B"])

master = pd.DataFrame(np.vstack(arr), index=idx, columns=cols)

原始代码：4s。新代码：7ms