在一个键的新列中解析变化键的pyspark数据框列

Question

我有一个输入Pyspark DataFrame DF。 DataFrame DF具有具有字典值的列“ Field1”。词典并非都有相同的键。我想将“ B”键解析到一个新领域“ Newcol”中。为了进一步使事物复杂化，field1是数据类型字符串。我已经尝试了下面的代码，但是下面遇到了错误。有人有建议如何做到这一点吗？

输入DF：

+--+---------------------+
|id|field1               |
+--+---------------------+
| 1|{"a":1,"b":"f"}      |
+--+---------------------+
| 2|{"a":1,"b":"e","c":3}|
+--+---------------------+

输出DF：

+--+---------------------+------+
|id|field1               |newcol|
+--+---------------------+------+
| 1|{"a":1,"b":"f"}      |'f'   |
+--+---------------------+------+
| 2|{"a":1,"b":"e","c":3}|'e'   |
+--+---------------------+------+

代码：

df.select(
    col('id'),
    col('field1'),
    from_json(col("field1"), ArrayType(StringType())).getItem("b")
).show(truncate=False)

错误：

An error was encountered:
An error occurred while calling o571.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 8.0 failed 4 times, most recent failure: Lost task 2.3 in stage 8.0 (TID 59, ip-10-100-190-16.us-west-2.compute.internal, executor 49): org.apache.spark.sql.execution.datasources.FileDownloadException: Failed to download file path: s3://treez-data-lake/product_api/products/part-00005-ab9a676d-e9fa-4594-a998-77e8ae0dd95b-c000.snappy.parquet, range: 0-41635218, partition values: [empty row], isDataPresent: false

...

Answer 1

让我们尝试外国库LILLAL_EVAL转换为maptype，然后使用pyspark method map_values将值获取列表并将值由索引

from ast import literal_eval
import pyspark.sql.functions as F
df2 = df.withColumn('newcol', map_values(F.udf(literal_eval, 'map<string,string>')('field1'))[1])#.select(map_values("newcol").alias("newcol")[1])
df2.show(truncate=False)


+---+---------------------+------+
|d  |field1               |newcol|
+---+---------------------+------+
|1  |{"a":1,"b":"f"}      |f     |
|2  |{"a":1,"b":"e","c":3}|e     |
+---+---------------------+------+