并行化熊猫中的虚拟数据生成

Question

我想生成一个使用多个处理器N内核组成的虚拟数据集和4000万个记录的姓氏。

以下是一个单个任务循环，该循环生成名字和姓氏，并将其附加到列表：

import pandas as pd
from faker import Faker

def fake_data_generation(records):
    fake = Faker(['en_US','en_GB'])
    
    person = []
    
    for i in range(records):
        first_name = fake.first_name()
        last_name = fake.last_name()
        person.append({"First_Name": first_name,
                       "Last_Name": last_name}
                     )
    return person

输出：

for i in range(5):
    df = pd.DataFrame(fake_data_generation(i))

>>> df
  First_Name Last_Name
0      Colin   Stewart
1    Barbara      Rios
2     Victor     Green
3  Stephanie     Booth

Answer 1

也许您可以直接使用提供程序：

import pandas as pd
import numpy as np
from faker.providers.person.en_US import Provider as us
from faker.providers.person.en_GB import Provider as gb

first_names = list(set(us.first_names).union(gb.first_names))
last_names = list(set(us.last_names).union(gb.last_names))

N = 40_000_000
df = pd.DataFrame({'First_Name': np.random.choice(first_names, N),
                   'Last_Name': np.random.choice(last_names, N)})

输出：

>>> df
         First_Name Last_Name
0             Kayla      Tran
1              Gary     Bates
2             Daisy   Leblanc
3           Tiffany     Ahmed
4            Kellie       May
...             ...       ...
39999995   Kristine   Collier
39999996      Joyce     Mccoy
39999997       Paul   Padilla
39999998      Tonya     Bevan
39999999      Julie    Bright

[40000000 rows x 2 columns]

Answer 2

我尝试了下面与我合作的下面。我感谢任何评论或修改，以更好地表现或减少任何不必要的步骤。

from joblib import Parallel, delayed
import pandas as pd
from faker import Faker
from itertools import chain

fake = Faker(['en_US','en_GB'])

def generate_names_df():
    names = []
    first_name = fake.first_name()
    last_name = fake.last_name()
    names.append({"First_Name": first_name,
                  "Last_Name": last_name}
                )
    return names

results = Parallel(n_jobs=15)(delayed(generate_names_df)() for i in range(40000000))
results_unlisted = list(chain(*results))
df = pd.DataFrame(results_unlisted)

>>> df.shape
(40000000, 2)