如何完成将损坏的数据上传到生产 Google App Engine 服务器？

发布于 2024-08-27 11:32:11 字数 1133 浏览 10 评论 0原文

我将数据上传到 App Engine （不是开发服务器）通过加载器类和远程 API，我在 CSV 文件中间达到了配额。根据日志和进度sqllite db，如何选择要上传的剩余部分数据？

通过数十条记录来确定哪些已传输、哪些未传输，这并不是一项有吸引力的任务，因此我寻找某种方法来限制需要检查的记录数量。

这是相关的（IMO）日志部分，如何解释工作项编号？

[DEBUG    2010-03-30 03:22:51,757 bulkloader.py] [Thread-2] [1041-1050] Transferred 10 entities in 3.9 seconds
[DEBUG    2010-03-30 03:22:51,757 adaptive_thread_pool.py] [Thread-2] Got work item [1071-1080]
<cut>
[DEBUG    2010-03-30 03:23:09,194 bulkloader.py] [Thread-1] [1141-1150] Transferred 10 entities in 4.6 seconds
[DEBUG    2010-03-30 03:23:09,194 adaptive_thread_pool.py] [Thread-1] Got work item [1161-1170]
<cut>
[DEBUG    2010-03-30 03:23:09,226 bulkloader.py] [Thread-3] [1151-1160] Transferred 10 entities in 4.2 seconds
[DEBUG    2010-03-30 03:23:09,226 adaptive_thread_pool.py] [Thread-3] Got work item [1171-1180]
[ERROR    2010-03-30 03:23:10,174 bulkloader.py] Retrying on non-fatal HTTP error: 503 Service Unavailable

原文

I was uploading the data to App Engine (not dev server) through loader class and remote api, and I hit the quota in the middle of a CSV file. Based on logs and progress sqllite db, how can I select remaining portion of data to be uploaded?

Going through tens of records to determine which was and which was not transfered, is not appealing task, so I look for some way to limit the number of record I need to check.

Here's relevant (IMO) log portion, how to interpret work item numbers?

[DEBUG    2010-03-30 03:22:51,757 bulkloader.py] [Thread-2] [1041-1050] Transferred 10 entities in 3.9 seconds
[DEBUG    2010-03-30 03:22:51,757 adaptive_thread_pool.py] [Thread-2] Got work item [1071-1080]
<cut>
[DEBUG    2010-03-30 03:23:09,194 bulkloader.py] [Thread-1] [1141-1150] Transferred 10 entities in 4.6 seconds
[DEBUG    2010-03-30 03:23:09,194 adaptive_thread_pool.py] [Thread-1] Got work item [1161-1170]
<cut>
[DEBUG    2010-03-30 03:23:09,226 bulkloader.py] [Thread-3] [1151-1160] Transferred 10 entities in 4.2 seconds
[DEBUG    2010-03-30 03:23:09,226 adaptive_thread_pool.py] [Thread-3] Got work item [1171-1180]
[ERROR    2010-03-30 03:23:10,174 bulkloader.py] Retrying on non-fatal HTTP error: 503 Service Unavailable

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

水溶 2024-09-03 11:32:11

您可以恢复损坏的上传：

如果传输中断，您
可以从原来的位置恢复传输
停止使用 --db_filename=...
争论。该值是名称
该工具创建的进度文件，
这是您提供的名称
与 --db_filename 参数时
您开始传输，或者默认
包含时间戳的名称。这
假设你已经安装了 sqlite3，
并且没有禁用进度文件
与--db_filename=skip。