AWS GLUE 3.0 -PYSPARK的铸造问题

发布于 2025-01-22 12:25:35 字数 563 浏览 4 评论 0原文

我正在使用胶3.0

data = [("Java", "6241499.16943521594684385382059800664452")]
rdd = spark.sparkContext.parallelize(data)
df = rdd.toDF()
df.show()
df.select(f.col("_2").cast("decimal(15,2)")).show()

我在本地获得以下结果

+----+--------------------+
|  _1|                  _2|
+----+--------------------+
|Java|6241499.169435215...|
+----+--------------------+

+----+
|  _2|
+----+
|null|
+----+

，并使用pyspark =“ == 3.2.1”没有问题将字符串施放为DECIMAL（）但是胶水作业无法做到

原文

I'm using Glue 3.0

data = [("Java", "6241499.16943521594684385382059800664452")]
rdd = spark.sparkContext.parallelize(data)
df = rdd.toDF()
df.show()
df.select(f.col("_2").cast("decimal(15,2)")).show()

I get the following result

+----+--------------------+
|  _1|                  _2|
+----+--------------------+
|Java|6241499.169435215...|
+----+--------------------+

+----+
|  _2|
+----+
|null|
+----+

locally with pyspark= "==3.2.1" there is no issue to cast the string to decimal() but the Glue job is not able to do so

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

对岸观火 2025-01-29 12:25:35

问题是 aws胶水！为了遇到这一点，我曾经在执行cast

def prepareStringDecimal(str_):
    """
    Pyspark UDF
    :param str_: "1234.123456789"
    :return: 1234.12345
    """
    arr = str(str_).split(".")
    if len(arr) > 1:
        return arr[0] + "." + arr[1][:5]
    else:
        return str_

# convert function to UDF
convertUDF = udf(lambda z: prepareStringDecimal(z), StringType())

data = [("Java", "6241499.16943521594684385382059800664452")]
df = spark.sparkContext.parallelize(data).toDF()
df.show()
df.select(convertUDF(f.col("_2")).cast("decimal(15,2)")).show()

输出

+----+--------------------+
|  _1|                  _2|
+----+--------------------+
|Java|6241499.169435215...|
+----+--------------------+

+-----------------------------------+
|CAST(<lambda>(_2) AS DECIMAL(15,2))|
+-----------------------------------+
|                         6241499.17|
+-----------------------------------+

注：显然！我们可以使用 spark sql functions

The problem is with AWS Glue ! in order to encounter this, I used to convert my string before doing the cast

def prepareStringDecimal(str_):
    """
    Pyspark UDF
    :param str_: "1234.123456789"
    :return: 1234.12345
    """
    arr = str(str_).split(".")
    if len(arr) > 1:
        return arr[0] + "." + arr[1][:5]
    else:
        return str_

# convert function to UDF
convertUDF = udf(lambda z: prepareStringDecimal(z), StringType())

data = [("Java", "6241499.16943521594684385382059800664452")]
df = spark.sparkContext.parallelize(data).toDF()
df.show()
df.select(convertUDF(f.col("_2")).cast("decimal(15,2)")).show()

Output

+----+--------------------+
|  _1|                  _2|
+----+--------------------+
|Java|6241499.169435215...|
+----+--------------------+

+-----------------------------------+
|CAST(<lambda>(_2) AS DECIMAL(15,2))|
+-----------------------------------+
|                         6241499.17|
+-----------------------------------+

Note: Obviously ! we can use Spark SQL Functions instead

回复收藏 0 原文

~没有更多了~