如何将Groupby带入Pyspark中具有字符串数据类型的列？

Question

我想分组多个列，但是其中一个具有字符串型值。 在此处发布示例数据集，我正在使用的数据框架有多个列，上面有int和1 String类型列。

给定的数据框：

    # |Year|           Movie|
    # +----+----------------+
    # |2020|       Inception|
    # |2018|   The Godfather|
    # |2018| The Dark Knight|
    # |2015|    12 Angry Men|
    # |2020|Schindler's List|
    # |2015|    Pulp Fiction|
    # |2018|      Fight Club|

必需的数据框：

    # |Year|Movie                                       |
    # +----+--------------------------------------------+
    # |2020|[Inception, Schindler's List]               |
    # |2018|[The Godfather, The Dark Knight, Fight Club]|
    # |2015|[12 Angry Men, Pulp Fiction]                |

Answer 1

您可以按年进行分组，并使用collect_set将所有项目分组：

import pyspark.sql.functions as F

df = sqlContext.createDataFrame([
    ('2020','Inception'),
    ('2018','The Godfather'),
    ('2018','The Dark Knight'),
    ('2015','12 Angry Men'),
    ('2020','Schindlers List'),
    ('2015','Pulp Fiction'),
    ('2018','Fight Club')
     ], ["Year", "Movie"])\
    .withColumn('Year', F.col('Year').cast('integer'))\
    .withColumn('Movie', F.col('Movie').cast('string'))
    
# +----+---------------+
# |Year|          Movie|
# +----+---------------+
# |2020|      Inception|
# |2018|  The Godfather|
# |2018|The Dark Knight|
# |2015|   12 Angry Men|
# |2020|Schindlers List|
# |2015|   Pulp Fiction|
# |2018|     Fight Club|
# +----+---------------+

df\
    .groupby("Year")\
    .agg(F.collect_set("Movie"))\
    .show(truncate=False)

# +----+--------------------------------------------+
# |Year|collect_set(Movie)                          |
# +----+--------------------------------------------+
# |2018|[The Godfather, Fight Club, The Dark Knight]|
# |2015|[Pulp Fiction, 12 Angry Men]                |
# |2020|[Schindlers List, Inception]                |
# +----+--------------------------------------------+