Python 实现 avro 慢？
我正在使用 avro 库从 avro 文件读取一些数据。从文件加载 33K 对象大约需要一分钟。这对我来说似乎很慢，特别是 Java 版本在大约 1 秒内读取相同的文件。
这是代码，我做错了什么吗？
import avro.datafile
import avro.io
from time import time
def load(filename):
fo = open(filename, "rb")
reader = avro.datafile.DataFileReader(fo, avro.io.DatumReader())
for i, record in enumerate(reader):
pass
return i + 1
def main(argv=None):
import sys
from argparse import ArgumentParser
argv = argv or sys.argv
parser = ArgumentParser(description="Read avro file")
start = time()
num_records = load("events.avro")
end = time()
print("{0} records in {1} seconds".format(num_records, end - start))
if __name__ == "__main__":
main()
I'm reading some data from avro file using the avro library. It takes about a minute to load 33K objects from the file. This seem very slow to me, specially with the Java version reading the same file in about 1sec.
Here is the code, am I doing something wrong?
import avro.datafile
import avro.io
from time import time
def load(filename):
fo = open(filename, "rb")
reader = avro.datafile.DataFileReader(fo, avro.io.DatumReader())
for i, record in enumerate(reader):
pass
return i + 1
def main(argv=None):
import sys
from argparse import ArgumentParser
argv = argv or sys.argv
parser = ArgumentParser(description="Read avro file")
start = time()
num_records = load("events.avro")
end = time()
print("{0} records in {1} seconds".format(num_records, end - start))
if __name__ == "__main__":
main()
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
似乎有一个名为
fastavro的 Python 包，它是一个快速的 Cython 实现，但功能不太完整。https://github.com/fastavro/fastavro
It appears there is a python package called
fastavrothat is a fast Cython implementation, but is less feature-complete.https://github.com/fastavro/fastavro
PyPI 上提供的 avro Python 包 是纯 Python，所以如果它速度较慢我并不感到惊讶比 Java 高一个数量级或更多。
有一个 Avro C 实现，但据我所知，还没有人基于它创建 Python 扩展。
The avro Python package available on PyPI is pure Python, so I'm not surprised if it is slower than Java by an order of magniture or more.
There is an Avro C implementation, but to my knowledge nobody has yet created a Python extension based on it.