Spark Scala,缺少扩展功能的参数类型
我具有此功能:
def extractTemplateMeraki(IS_serviceCharacteristics: String,keyword:String): String = {
//some operations here and return a string in the end as a result
}
我想将此函数传递给UDF这样的功能:
val getTemplateMerakiUdf= functions.udf(extractTemplateMeraki(_,"HEBERGEMENT"))
在这种情况下,第二个参数始终是修复的,因此,当我读取我的data frame时,我使用此udf创建一个带有其返回值的新列每一行:
val df = mySourceDatframe
.withColumn("TemplateMeraki",getTemplateMerakiUdf(col("InstalledService_serviceCharacteristics")))
df
.select("TemplateMeraki")
.where("TemplateMeraki NOT IN ('')")
.show()
启动程序时,我会被异常的Bellow捕获:
missing parameter type for expanded function ((x$1) => extractTemplateMeraki(x$1, "HEBERGEMENT"))
val getTemplateMerakiUdf= functions.udf(extractTemplateMeraki(_,"HEBERGEMENT"))
我是Scala的新手,我想知道如何在处理数据框时不影响UDF的过程而在不影响UDF的过程中传递第二个参数。
I have this function:
def extractTemplateMeraki(IS_serviceCharacteristics: String,keyword:String): String = {
//some operations here and return a string in the end as a result
}
I want to pass this function to a udf like that:
val getTemplateMerakiUdf= functions.udf(extractTemplateMeraki(_,"HEBERGEMENT"))
the second parameter is always fixe in this case, so when I'm reading my dataframe, I use this udf to create a new column with its returned value for each row:
val df = mySourceDatframe
.withColumn("TemplateMeraki",getTemplateMerakiUdf(col("InstalledService_serviceCharacteristics")))
df
.select("TemplateMeraki")
.where("TemplateMeraki NOT IN ('')")
.show()
when I start the program, I'm caught by the exception bellow:
missing parameter type for expanded function ((x$1) => extractTemplateMeraki(x$1, "HEBERGEMENT"))
val getTemplateMerakiUdf= functions.udf(extractTemplateMeraki(_,"HEBERGEMENT"))
I'm new to scala, and I want to know how to pass the second argument without impacting the process of the UDF while handling the dataframe.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论