Spark Java-如何将Java对象存储在Spark DataFrame中?

发布于 2025-02-13 07:54:51 字数 1110 浏览 1 评论 0原文

我在Java中使用Spark,我想将Java对象存储在Spark DataFrame中。我搜索了许多博客和链接,但找不到这样做的方法。假设我的对象类是 -

class Temporary{
    Integer id;
    String name;
    String another;

    public Temporary(Integer id, String name, String another){
        this.id = id;
        this.name = name;
        this.another = another;
    }
}

我有一个DataFrame DF,我想将此临时类的对象存储在列中。假设我正在添加一个将具有临时类对象的数组类型的临时。我想做类似的事情 -

df = df.withColumn("Temporary", functions.array(
                               functions.expr("CASE WHEN something_is_true THEN Temporary(1,"john","tempstring") ELSE empty_object END"),
                               functions.expr("CASE WHEN something_else_is_true THEN Temporary(2,"johny","tempstring") ELSE empty_object END")
                              ));

// expected result would be 

+------------------------------------------+
|                Temprorary                |
+------------------------------------------+
|  [Temproary_class_object, empty_object]  |
+------------------------------------------+

我知道这是行不通的,因为Expr仅接受SQL字符串而不是对象。有什么办法吗?任何帮助将不胜感激。

I'm using spark in java and I want to store java objects in spark dataframe. I've searched many blogs and links but couldn't find a way to do this. Let's say my object class is -

class Temporary{
    Integer id;
    String name;
    String another;

    public Temporary(Integer id, String name, String another){
        this.id = id;
        this.name = name;
        this.another = another;
    }
}

I have a dataframe df and I want to store an object of this Temporary class in a column. Assume I am adding a column Temporary of array type that will have objects of Temporary class. I want to do something like-

df = df.withColumn("Temporary", functions.array(
                               functions.expr("CASE WHEN something_is_true THEN Temporary(1,"john","tempstring") ELSE empty_object END"),
                               functions.expr("CASE WHEN something_else_is_true THEN Temporary(2,"johny","tempstring") ELSE empty_object END")
                              ));

// expected result would be 

+------------------------------------------+
|                Temprorary                |
+------------------------------------------+
|  [Temproary_class_object, empty_object]  |
+------------------------------------------+

I know this will not work because expr only accepts a SQL string and not objects. Any way to do this? Any help would be appreciated.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文