Jupyter Lab实例与502错误崩溃
我正在使用GCP顶点AI工作台的Jupyterlab虚拟笔记本实例。
我正在阅读20亿行数据，其中每一行由3列组成 每个字节每个。
我一次读取1亿行数据，并将其连接到Pandas DataFrame。
突然之间，笔记本变得无反应，有502个错误。
我意识到虚拟机崩溃了。
这是虚拟机的规格： N1标准64 240GB RAM 100 GB开车
一次，我成功达到了20亿行。 但是突然间，令我沮丧的是，这是错误的。
Google Doc只是提到重新启动内核。 当读取20亿行数据时，这并不容易。 这意味着超过1小时的工作刚刚浪费了。
是什么导致此错误？ 为什么错误如此不一致？ 错误消息在哪里崩溃？ 还是这是与Pandas DataFrame有关的错误？ 我正在创建一个具有20亿行的数据框架。 如果熊猫无法处理如此大小的行，它应该简单地 导致运行的时间错误，不会崩溃虚拟机。
提前致谢
I am using a JupyterLab virtual notebook instance from GCP Vertex AI Workbench.
I am reading 2 billion rows of data where each row is comprised of 3 columns of
8 bytes each.
I am reading 100 million rows of data at a time and concatenating it to Pandas dataframe.
All of sudden, the notebook becomes unresponsive with 502 error.
I realize that the virtual machine crashed.
Here is the spec to the virtual machine:
n1-standard 64 240GB RAM
100 GB drive
One time, I was successful to reach 2 billion rows.
But all of sudden, to my dismay, it crashed with that error.
Google doc just mentions to restart the kernel.
That is not so easy when it took more than 1 hour to read 2 billion rows of data.
This means more than 1 hour of work just got wasted.
What is causing this error?
Why the error occurs so inconsistently?
Where is the error message for this to crash?
Or is this an error related to pandas dataframe?
I am creating a dataframe that have 2 billion rows.
If pandas cannot handle rows of this magnitude, it should simply
cause a run time error, not crashing a virtual machine.
Thanks in advance
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
由于代码运行到端口重叠，因此发生了此错误。由于停止内核的一部分，因此它应该是修复的=“ nofollow noreferrer”> github 更改将
restart_kernel替换为shutdown_kernel。我们还需要确保在关闭内核时清理容器。
为此，您可以按照以下步骤进行操作：
This error happens because the code runs into ports overlaps. It is supposed to be fixed since the part of the code that stops the kernel it is changed in github The change was replacing
restart_kerneltoshutdown_kernel.We also need to be sure that the container is cleaned up when shutting down the kernel.
For this you can follow these steps: