执行自定义批量加载程序时出现服务器错误 (Google Apps Engine)
我没有成功使用官方提供的bulkloader选项，所以我编写了自己的bulkloader脚本（实际上是后处理程序，它将增量地将实体从csv加载到数据存储中）。
该解决方案的工作原理如下：
1.我将从 csv 文件复制数据
2. 将其粘贴到应用程序上表单的文本区域
3. 发布表格
4. 处理程序解析传入文本的标题（列名称）
5. 将其余行存储在列表中
6. 从列表中增量获取 100 行，并为每行创建并存储一个实体（实体的类型是根据表单中选择字段指定的类型解析的）
现在，此技术可以在我的开发服务器上用于小输入最多 1000 行，超出会显示以下错误：
回溯（最近一次调用最后一次）：
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/ext/webapp/init.py”，第 513 行，call< br> 文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/myApps/ugvotes/ugvotes.py”，第 241 行，帖子
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/ext/db/init.py”，第 893 行，位于 put
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore.py”，第 291 行，位于 Put
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore.py”，第 195 行，位于 _MakeSyncCall
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_stub_map.py”，第 499 行，位于 check_success
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_rpc.py”，第 149 行，位于 _WaitImpl
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py”，第 863 行，位于 MakeSyncCall
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_stub.py”，第 80 行，位于 MakeSyncCall
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py”，第 933 行，位于 _Dynamic_Put
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py”，第 806 行，位于 __WriteDatastore
文件“/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py”，第 836 行，位于 __WritePickled
IOError: [Errno 24] 打开文件太多: '/tmp/tmpOfgvm3'
起初我认为这是由于开发服务器的限制而导致的错误，但是当我尝试从 生产服务器，我收到以下错误：
错误：服务器错误 服务器遇到错误，无法完成您的请求。
如果问题仍然存在，请报告您的问题并提及此错误消息以及导致该问题的查询。
谁知道可能出了什么问题，我能做些什么呢？
谢谢。
I had not succeeded in using the bulkloader option that is officially provided, so i wrote my own bulkloader script (actually post handler that would incrementally load entities from csv into the datastore).
The solution worked as follows:
1. I would copy data from a csv file
2. Paste it into a text area in a form on the app
3. Post the form
4. The handler parses the incoming text for headers (column names)
5. Stores rest of lines in list
6. incrementally fetches 100 lines from the list and for each line creates and stores an entity (the type of entity is resolved from the type specified from a select field from the form)
Now, this technique would work on my dev server for small inputs of up to 1000 lines, beyond that it would show the following error:
Traceback (most recent call last):
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/ext/webapp/init.py", line 513, in call
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/myApps/ugvotes/ugvotes.py", line 241, in post
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/ext/db/init.py", line 893, in put
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore.py", line 291, in Put
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore.py", line 195, in _MakeSyncCall
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 499, in check_success
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py", line 863, in MakeSyncCall
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/apiproxy_stub.py", line 80, in MakeSyncCall
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py", line 933, in _Dynamic_Put
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py", line 806, in __WriteDatastore
File "/host/_Hive/Lab/ACTIVE WORKS/UG VOTES/google_appengine/google/appengine/api/datastore_file_stub.py", line 836, in __WritePickled
IOError: [Errno 24] Too many open files: '/tmp/tmpOfgvm3'
At first I thought this was an error due to the limitations of the development server, but when i tried to do the same task from the
production server, i got the following error:
Error: Server Error
The server encountered an error and could not complete your request.If the problem persists, please report your problem and mention this error message and the query that caused it.
Who knows what could have gone wrong, and what i can do about it?
thanks.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
这是一个已知问题：Python dev SDK 1.3.8 datastore_file_stub.py 似乎泄漏文件句柄。
有一个非官方的补丁可用。
This is a known issue: Python dev SDK 1.3.8 datastore_file_stub.py appears to leak file handles.
There's an unofficial patch available.