如何计算Pyarrow数组中的唯一值数量？

发布于 2025-02-06 04:34:33 字数 1277 浏览 1 评论 0 原文

我有一个pyarrow int32 包含我从兽人文件中获得的18个块：

import pyarrow.dataset
import pyarrow.compute
t = pyarrow.dataset.dataset("my/orc/file", format="orc").to_table()
a = t.column("a_column")
print(type(a))
print(a.num_chunks)
print(a.type)
<class 'pyarrow.lib.ChunkedArray'>
18
int32

我想计算列中唯一值的数量。对于 count_distinct ：

>>> print(pyarrow.compute.count_distinct(a))
36

然而， unique（） 表示只有两个非零值：

>>> print(pyarrow.compute.unique(a))
[
  null,
  100,
  250
]

建议在块上求和，建议 count_distinct（）。这是预期的行为吗？获取数组中非null值数量的正确方法是什么？

原文

I have a pyarrow int32 ChunkedArray
containing 18 chunks that I got from an ORC file:

import pyarrow.dataset
import pyarrow.compute
t = pyarrow.dataset.dataset("my/orc/file", format="orc").to_table()
a = t.column("a_column")
print(type(a))
print(a.num_chunks)
print(a.type)
<class 'pyarrow.lib.ChunkedArray'>
18
int32

I want to compute the number of unique values in the column. Seems like a straightforward job for count_distinct:

>>> print(pyarrow.compute.count_distinct(a))
36

However, unique() indicates that there are only two non-null values:

>>> print(pyarrow.compute.unique(a))
[
  null,
  100,
  250
]

Suggesting that that count_distinct() is summed over the chunks. Is this the expected behavior? What is the correct way to get the number of non-null values in array (either chunked or non-chunked)?