如何在项目列表中搜索并根据3个列表的元素提取一些关键词？

Question

我有一个列是一个数组

假设我

id               desxription                 
1         ['this is bad', 'summerfull']     
2         ['city tehran, country iran']       
3         ['uA is a country', 'winternice'] 
5         ['this, is, summer']              
6         ['this is winter','uAsal']       
7         ['this is canada' ,'great']       
8         ['this is toronto']            

有3个

L1 = ['summer', 'winter', 'fall']
L2 = ['iran', 'uA']
L3 = ['tehran', 'canada', 'toronto']

列表， L2，L3）。然后在“描述”列中的列表的每个元素中搜索。如果行具有列表的元素，请在列中提取它，否则NA：

注意：我希望提取确切的匹配。例如summerfull不应在夏季提取。

id               desxription                  L1          L2         L3
1         ['this is bad', 'summerfull']       NA         NA          NA
2         ['city tehran, country iran']       NA         iran        tehran
3         ['uA is a country', 'winternice']   NA         uA          NA
5         ['this, is, summer']               summer      NA          NA
6         ['this is winter','uAsal']         winter      NA          NA
7         ['this is canada' ,'great']        NA         NA          canada
8         ['this is toronto']                NA         NA          toronto

Answer 1

注释的代码

# Create dictionary of key-vals
L = {'L1': L1, 'L2': L2, 'L3': L3}

for key, vals in L.items():
    # regex pattern for extracting vals
    pat = r'\\b(%s)\\b' % '|'.join(vals)

    # extract matching occurrences
    col = F.expr("regexp_extract_all(array_join(desxription, ' '), '%s')" % pat)

    # Mask the rows with null when there are no matches
    df = df.withColumn(key, F.when(F.size(col) == 0, None).otherwise(col))

---

>>> df.show()

+---+--------------------+--------+------+---------+
| id|         desxription|      L1|    L2|       L3|
+---+--------------------+--------+------+---------+
|  1|[this is bad, sum...|    null|  null|     null|
|  2|[city tehran, cou...|    null|[iran]| [tehran]|
|  3|[uA is a country,...|    null|  [uA]|     null|
|  5|  [this, is, summer]|[summer]|  null|     null|
|  6|[this is winter, ...|[winter]|  null|     null|
|  7|[this is canada, ...|    null|  null| [canada]|
|  8|   [this is toronto]|    null|  null|[toronto]|
+---+--------------------+--------+------+---------+