阅读木薯片到熊猫filenotfounderror

Question

我的代码如下，运行良好。 数据

April_data = sc.read.parquet('somepath/data.parquet')
type(April_data)
pyspark.sql.dataframe.DataFrame

为火花

df_pp = pd.read_parquet('somepath/data.parquet')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_4244/1910461502.py in <module>
----> 1 df_pp = pd.read_parquet('somepath/data.parquet')

/usr/local/anaconda//parquet.py in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
    498         storage_options=storage_options,
    499         use_nullable_dtypes=use_nullable_dtypes,
--> 500         **kwargs,
    501     )

/usr/local/anaconda//io/parquet.py in read(self, path, columns, use_nullable_dtypes, storage_options, **kwargs)
    234             kwargs.pop("filesystem", None),
    235             storage_options=storage_options,
--> 236             mode="rb",
    237         )
    238         try:

/usr/local/anaconda/parquet.py in _get_path_or_handle(path, fs, storage_options, mode, is_dir)
    100         # this branch is used for example when reading from non-fsspec URLs
    101         handles = get_handle(
--> 102             path_or_handle, mode, is_text=False, storage_options=storage_options
    103         )
    104         fs = None

/usr/local/anaconda/common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
    709         else:
    710             # Binary mode
--> 711             handle = open(handle, ioargs.mode)
    712         handles.append(handle)
    713 

FileNotFoundError: [Errno 2] No such file or directory: 'somepath/data.parquet'

读

!pip install fastparquet
Successfully installed cramjam-2.5.0 fastparquet-0.8.1

它

框当我这样做时，我可以

hdfs_location = 'somepath/'
!hdfs dfs -ls $hdfs_location

在同一文件中运行所有这些代码

Answer 1

每个文档， 与其他兄弟姐妹IO模块类似，不支持HDFS位置的阅读。虽然有 ，它没有读取镶木木材或其他已知格式。

对于Read_Parquet中的字符串值，当前支持CPU文件路径或仅在线方案（HTTP，FTP）和两个特定的存储路径（Amazon S3存储桶，Google Cloud Storage或GS）。

但是，您可以通过类似文件的对象。因此，请考虑阅读所需的镶木quet文件并传递内容。以下是使用各种HDFS软件包的示例：

from hdfs import Client

with client.read('somepath/data.parquet') as f: 
    df_pp = pd.read_parquet(f.read())

from hdfs3 import HDFileSystem 
hdfs = HDFileSystem(host='localhost', port=8020)

with hdfs.open('somepath/data.parquet') as f: 
    df_pp = pd.read_parquet(f)

另外， > 支持转换为熊猫数据框架：

from fastparquet import ParquetFile

pf = ParquetFile('somepath/data.parquet') 
df = pf.to_pandas()