如何使用MDFReader从AWS S3读取.DAT文件
我正在使用Python 3.7，并尝试从AWS S3读取.DAT文件，并在某些逻辑上转换为一个或多个CSV。我们正在使用 mdfreader python中的库。
import mdfreader
import pandas as pd
def convert_mdf_to_csvs(file_name, output_file_loc) :
yop=mdfreader.Mdf(file_name)
yop.convert_to_pandas()
# print(list(yop.keys()))
# print([keys for keys in list(yop.keys()) if keys.endswith("group")])
all_groups_keys = [keys for keys in list(yop.keys()) if keys.endswith("group")]
for keys in all_groups_keys :
print(yop[keys])
timeframe = keys.split("group")[0]
yop[keys].to_csv(str(output_file_loc) + str(timeframe) + ".csv" )
以上代码在本地机器中工作正常，但是由于AWS S3是对象存储“ yop = mdfreader.mdf（file_name）”功能？ MDF功能似乎接受完整的文件路径。我知道我可以将其复制到lambda的TMP并使用它，但是由于那个黑客，我不想这样做。
在SO Q/A上进行了很多搜索，但没有从AWS S3读取的.DAT文件类型的清晰度。
另外，是否有更好的方法来解决此问题，也许使用简单的CSV库或其他任何方法？
有帮助吗？
I'm using Python 3.7 and trying to read a .dat file from AWS S3 and convert it to one or more CSV on certain logic. We're using mdfreader library in Python.
import mdfreader
import pandas as pd
def convert_mdf_to_csvs(file_name, output_file_loc) :
yop=mdfreader.Mdf(file_name)
yop.convert_to_pandas()
# print(list(yop.keys()))
# print([keys for keys in list(yop.keys()) if keys.endswith("group")])
all_groups_keys = [keys for keys in list(yop.keys()) if keys.endswith("group")]
for keys in all_groups_keys :
print(yop[keys])
timeframe = keys.split("group")[0]
yop[keys].to_csv(str(output_file_loc) + str(timeframe) + ".csv" )
This above code is working fine in a local machine, but since AWS S3 is object storage so the read will be using boto3, but due to lack of documentation on the mdfreader library side, am not very sure how to pass this read stream into the "yop=mdfreader.Mdf(file_name)" function? Mdf function seems to accept a full file path. I know I can copy that to Lambda's tmp and use it, but since that a hack, I do not want to do that.
Searched quite a bit on SO Q/A but didn't get this clarity for .dat file type read from AWS S3.
Also, is there a better way to solve this, maybe using simple csv library or anything else?
Any help?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
最简单的方法是使用
download_file（）在本地磁盘上将文件从Amazon S3下载到/tmp/code>。然后，您可以使用现有代码来处理文件。这绝对是不是 a'hack' - 它是一种常用的技术。当然，它比流式传输文件更可靠。
可以重复使用可用存储量的限制，AWS lambda容器可以重复使用，因此使用后删除临时文件，或者使用相同的文件名（例如
/tmp/temp.dat）使其覆盖先前的版本。The easiest method would be to use
download_file()to download the file from Amazon S3 to/tmp/on the local disk.Then, you can use your existing code to process the file. This is definitely not a 'hack' -- it is a commonly used technique. It's certainly more reliable than streaming the file.
There is a limit on the amount of storage available and AWS Lambda containers can be reused, so either delete the temporary file after use, or use the same filename (eg
/tmp/temp.dat) each time so that it overwrites the previous version.