如何分组并获得最频繁的ngram？

Question

我的数据帧看起来像这样：

ID topics   text
1     1        twitter is my favorite social media
2     1        favorite social media
3     2        rt twitter tomorrow
4     3        rt facebook today
5     3        rt twitter
6     4        vote for the best twitter
7     2        twitter tomorrow
8     4        best twitter

我想按主题进行分组并使用count vectorizer（我真的更喜欢使用CountVectorize，因为它允许以多种语言删除停止单词，并且可以设置3，4克的范围）来计算最频繁的范围Bigrams。在获得最频繁的Bigram之后，我想创建一个称为“ Biagram”的新列，并将每个主题最常见的BigRam分配给该列。

我希望我的输出看起来像这样。

ID topics      text                                 biagram
1     1        twitter is my favorite social       favorite social
2     1        favorite social media               favorite  social
3     2        rt twitter tomorrow                 twitter tomorrow
4     2        twitter tomorrow                    twitter tomorrow
5     3        rt twitter                          rt twitter
6     3        rt facebook today           rt twitter 
7     4        vote for the bes twitter               best twitter
8     4        best twitter                        best twitter

请注意，“主题”列不需要按主题订立。在创建此帖子时，我订购了为了可视化。

该代码将在6m行数据上运行，因此需要快速。

使用熊猫的最佳方法是什么？如果看起来太复杂，我深表歉意。

Answer 1

更新

您可以使用sklearn：

trom sklearn.feature_extraction.text import CountVectorizer

vect = CountVectorizer(analyzer='word', ngram_range=(2, 2), stop_words='english')
data = vect.fit_transform(df['text'])
bigram = (pd.DataFrame(data=data.toarray(),
                       index=df['topics'],
                       columns=vect.get_feature_names_out())
            .groupby('topics').sum().idxmax(axis=1))
df['bigram'] = df['topics'].map(bigram)
print(df)

# Output
   ID  topics                                 text            bigram
0   1       1  twitter is my favorite social media   favorite social
1   2       1                favorite social media   favorite social
2   3       2                  rt twitter tomorrow  twitter tomorrow
3   4       3                    rt facebook today    facebook today
4   5       3                           rt twitter    facebook today
5   6       4            vote for the best twitter      best twitter
6   7       2                     twitter tomorrow  twitter tomorrow
7   8       4                         best twitter      best twitter

更新2

如果我想要3个最频繁的ngrams怎么样。我可以使用什么代替idxmax（）？

most_common3 = lambda x: x.sum().nlargest(3).index.to_frame(index=False).squeeze()
bigram = (pd.DataFrame(data=data.toarray(),
                       index=df['topics'],
                       columns=vect.get_feature_names_out())
            .groupby('topics').apply(most_common3)
            .rename(columns=lambda x: f"bigram{x+1}").reset_index())
df = df.merge(bigram, on='topics')
print(df)

# Output
   topics                                 text           bigram1       bigram2           bigram3
0       1  twitter is my favorite social media   favorite social  social media  twitter favorite
1       1                favorite social media   favorite social  social media  twitter favorite
2       2                  rt twitter tomorrow  twitter tomorrow    rt twitter      best twitter
3       2                     twitter tomorrow  twitter tomorrow    rt twitter      best twitter
4       3                    rt facebook today    facebook today   rt facebook        rt twitter
5       3                           rt twitter    facebook today   rt facebook        rt twitter
6       4            vote for the best twitter      best twitter     vote best    facebook today
7       4                         best twitter      best twitter     vote best    facebook today

---

旧答案

您可以使用nltk：

import nltk

to_bigram = lambda x: list(nltk.bigrams(x.split()))
most_common = (df.set_index('topics')['text'].map(to_bigram)
                 .groupby(level=0).apply(lambda x: x.mode()[0][0]))

df['bigram'] = df['topics'].map(most_common)
print(df)

# Output
   ID  topics                                 text              bigram
0   1       1  twitter is my favorite social media  (favorite, social)
1   2       1                favorite social media  (favorite, social)
2   3       2                  rt twitter tomorrow       (rt, twitter)
3   4       3                    rt facebook today      (rt, facebook)
4   5       3                           rt twitter      (rt, facebook)
5   6       4            vote for the best twitter     (best, twitter)
6   7       2                     twitter tomorrow       (rt, twitter)
7   8       4                         best twitter     (best, twitter)