将数据框架存储为镶木木，并在一列中使用混合数据类型（时间戳和字符串）

发布于 2025-02-11 17:49:10 字数 346 浏览 2 评论 0原文

我想将大熊猫的数据框架存储为镶木素文件。但是我得到了这个错误：

pyarrow.lib.arrowtypeerror ：（转换为int”，'转换失败的columt 对象'）

列具有混合数据类型。我认为这是问题。但是我该如何解决呢？

#!/usr/bin/env python3
import pandas
df = pandas.DataFrame(
    data={
        'foo': [pandas.Timestamp('2022-06-01'), 'foobar']
        }
    )
print(df)

原文

I want to store a pandas data frame as Parquet file.
But I got this error:

pyarrow.lib.ArrowTypeError: ("object of type <class 'str'> cannot be
converted to int", 'Conversion failed for column foo with type
object')

The column has mixed data types. I assume this is the problem. But how can I solve that?

#!/usr/bin/env python3
import pandas
df = pandas.DataFrame(
    data={
        'foo': [pandas.Timestamp('2022-06-01'), 'foobar']
        }
    )
print(df)

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

南渊 2025-02-18 17:49:10

箭头仅支持具有定义明确的类型的结构化列数据，因此不支持混合类型。

但是，您可以使用Pyarrow Union类型。但这不是用户友好：

import pyarrow as pa
import pandas as pd

def series_to_union_array(series: pd.Series) -> pa.UnionArray:
    series_type = series.apply(type)
    types = series_type.unique().tolist()

    return pa.UnionArray.from_sparse(
        types=pa.array(series_type.apply(types.index), pa.int8()),
        children=[
            pa.array(series.where(series_type == dtype, None), None)
            for dtype in types
        ],
        field_names=[str(t) for t in types]
    )


df = pd.DataFrame(
    data={
        'foo': [pd.Timestamp('2022-06-01'), 'foobar']
        }
    )

table = pa.Table.from_arrays(
    [series_to_union_array(df['foo'])],
    names=['foo']
)

table['foo'].to_pylist()
>>> [datetime.datetime(2022, 6, 1, 0, 0), 'foobar']

Arrow only supports structured column data with a well defined type, so mixed types are not supported.

However you could use the pyarrow union type. But it's not very user friendly:

import pyarrow as pa
import pandas as pd

def series_to_union_array(series: pd.Series) -> pa.UnionArray:
    series_type = series.apply(type)
    types = series_type.unique().tolist()

    return pa.UnionArray.from_sparse(
        types=pa.array(series_type.apply(types.index), pa.int8()),
        children=[
            pa.array(series.where(series_type == dtype, None), None)
            for dtype in types
        ],
        field_names=[str(t) for t in types]
    )


df = pd.DataFrame(
    data={
        'foo': [pd.Timestamp('2022-06-01'), 'foobar']
        }
    )

table = pa.Table.from_arrays(
    [series_to_union_array(df['foo'])],
    names=['foo']
)

table['foo'].to_pylist()
>>> [datetime.datetime(2022, 6, 1, 0, 0), 'foobar']

回复收藏 0 原文

~没有更多了~