Spark Java-如何将Java对象存储在Spark DataFrame中?
我在Java中使用Spark,我想将Java对象存储在Spark DataFrame中。我搜索了许多博客和链接,但找不到这样做的方法。假设我的对象类是 -
class Temporary{
Integer id;
String name;
String another;
public Temporary(Integer id, String name, String another){
this.id = id;
this.name = name;
this.another = another;
}
}
我有一个DataFrame DF,我想将此临时类的对象存储在列中。假设我正在添加一个将具有临时类对象的数组类型的临时。我想做类似的事情 -
df = df.withColumn("Temporary", functions.array(
functions.expr("CASE WHEN something_is_true THEN Temporary(1,"john","tempstring") ELSE empty_object END"),
functions.expr("CASE WHEN something_else_is_true THEN Temporary(2,"johny","tempstring") ELSE empty_object END")
));
// expected result would be
+------------------------------------------+
| Temprorary |
+------------------------------------------+
| [Temproary_class_object, empty_object] |
+------------------------------------------+
我知道这是行不通的,因为Expr仅接受SQL字符串而不是对象。有什么办法吗?任何帮助将不胜感激。
I'm using spark in java and I want to store java objects in spark dataframe. I've searched many blogs and links but couldn't find a way to do this. Let's say my object class is -
class Temporary{
Integer id;
String name;
String another;
public Temporary(Integer id, String name, String another){
this.id = id;
this.name = name;
this.another = another;
}
}
I have a dataframe df and I want to store an object of this Temporary class in a column. Assume I am adding a column Temporary
of array type that will have objects of Temporary class. I want to do something like-
df = df.withColumn("Temporary", functions.array(
functions.expr("CASE WHEN something_is_true THEN Temporary(1,"john","tempstring") ELSE empty_object END"),
functions.expr("CASE WHEN something_else_is_true THEN Temporary(2,"johny","tempstring") ELSE empty_object END")
));
// expected result would be
+------------------------------------------+
| Temprorary |
+------------------------------------------+
| [Temproary_class_object, empty_object] |
+------------------------------------------+
I know this will not work because expr only accepts a SQL string and not objects. Any way to do this? Any help would be appreciated.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论