AWS胶 - iLlegalargumentException：路径的重复值

Question

我有一个混乱的数据源，其中某些字段值可以带有两个不同的名称，但应映射到输出上的一个符合的字段名称。

例如，数据源包含update_date或modified_date，并且两个都应映射到timestamp。

两个字段名称永远不会同时存在于同一行数据上。

胶脚本看起来像这样。有些线已被编辑为清晰：

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Script generated for node Data Catalog table
DataCatalogtable_node1 = glueContext.create_dynamic_frame.from_catalog(
    database="mydb",
    table_name="crawl_rawdata",
    transformation_ctx="DataCatalogtable_node1",
)

# Script generated for node ApplyMapping
ApplyMapping_node2 = ApplyMapping.apply(
    frame=DataCatalogtable_node1,
    mappings=[
        ...
        ("update_date", "string", "timestamp", "string"),
        ...
        ("modified_date", "string", "timestamp", "string"),
        ...
    ],
    transformation_ctx="ApplyMapping_node2",
)

# Script generated for node S3 bucket
S3bucket_node3 = glueContext.write_dynamic_frame.from_options(
    frame=ApplyMapping_node2,
    connection_type="s3",
    format="orc",
    connection_options={
        "path": "s3://mybucket/data-lake/glue/",
        "compression": "snappy",
        "partitionKeys": [ ... ],
    },
    transformation_ctx="S3bucket_node3",
)

job.commit()

如何使其运行？

Answer 1

您能否提供有关引起此错误的行的信息？您可以在AWS日志中找到它。我也遇到了同样的错误。就我而言，解决方案只是用此字符串列表替换一个字符串列表。让我向您展示一个代码：

#This raises the IllegalArgumentException: Duplicate value for path
list_of_columns = ["col_a","col_b"]
data.select_fields(list_of_columns)

#This raises no error
data.select_fields( ["col_a","col_b"])