为 Spark 中的每个组应用特定的过滤条件

发布于 2025-01-19 19:01:44 字数 758 浏览 0 评论 0原文

我有一个如下的数据框：

id    |original_date |date1      |date2      |name
1     |03-30-2022    |03-29-2022 |04-02-2022 | John  
1     |03-27-2022    |03-29-2022 |04-02-2022 | Mary
2     |04-01-2022    |03-29-2022 |04-02-2022 | Joe
2     |03-30-2022    |04-02-2022 |04-08-2022 | Susan
3     |04-03-2022    |04-02-2022 |04-08-2022 | Mallory

我希望获得以下结果数据框，以便对于每组 id ，我想应用一个过滤条件，使得 date1 date1 date1 date2原始日期 <= 日期2。

id    |original_date |date1      |date2      |name
1     |03-30-2022    |03-29-2022 |04-02-2022 | John  
2     |04-01-2022    |03-29-2022 |04-02-2022 | Joe
3     |04-03-2022    |04-02-2022 |04-08-2022 | Mallory

我该怎么做？

原文

I have a data frame as below:

id    |original_date |date1      |date2      |name
1     |03-30-2022    |03-29-2022 |04-02-2022 | John  
1     |03-27-2022    |03-29-2022 |04-02-2022 | Mary
2     |04-01-2022    |03-29-2022 |04-02-2022 | Joe
2     |03-30-2022    |04-02-2022 |04-08-2022 | Susan
3     |04-03-2022    |04-02-2022 |04-08-2022 | Mallory

I am looking to get the following resultant dataframe such that for each group of id, I want to apply a filter condition such that date1 < original_date <= date2.

id    |original_date |date1      |date2      |name
1     |03-30-2022    |03-29-2022 |04-02-2022 | John  
2     |04-01-2022    |03-29-2022 |04-02-2022 | Joe
3     |04-03-2022    |04-02-2022 |04-08-2022 | Mallory

How can I do this?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

旧城空念 2025-01-26 19:01:44

这不是只是直接的过滤条件，还是我缺少一些东西？

from pyspark.sql import functions as F

(df
    .where(
          (F.col('date1') < F.col('original_date'))
        & (F.col('original_date') <= F.col('date2'))
    )
    .show()
)

+---+-------------+----------+----------+-------+
| id|original_date|     date1|     date2|   name|
+---+-------------+----------+----------+-------+
|  1|   03-30-2022|03-29-2022|04-02-2022|   John|
|  2|   04-01-2022|03-29-2022|04-02-2022|    Joe|
|  3|   04-03-2022|04-02-2022|04-08-2022|Mallory|
+---+-------------+----------+----------+-------+

Isn't that just a straightforward filter condition, or am I missing something?

from pyspark.sql import functions as F

(df
    .where(
          (F.col('date1') < F.col('original_date'))
        & (F.col('original_date') <= F.col('date2'))
    )
    .show()
)

+---+-------------+----------+----------+-------+
| id|original_date|     date1|     date2|   name|
+---+-------------+----------+----------+-------+
|  1|   03-30-2022|03-29-2022|04-02-2022|   John|
|  2|   04-01-2022|03-29-2022|04-02-2022|    Joe|
|  3|   04-03-2022|04-02-2022|04-08-2022|Mallory|
+---+-------------+----------+----------+-------+

回复收藏 0 原文

~没有更多了~