使用 20.000 个字符串填充 Google App Engine 应用程序的数据存储区

发布于 2024-12-11 00:55:52 字数 992 浏览 0 评论 0原文

我正在尝试在本地数据存储中创建并存储 20000 个随机代码，然后在 appspot 中尝试此操作...这是模型

class PromotionCode (db.Model):
  code = db.StringProperty(required=True)

这是处理填充请求的类（只有登录的管理员可以使用它）。它创建随机字母数字代码并尝试在数据存储中存储 20000 个代码：

class Populate(webapp.RequestHandler):

  def GenerateCode(self):
    chars = string.letters + string.digits
    code = ""
    for i in range(8):
      code = code + choice(chars)
    return code.upper()

  def get(self):
    codes = "";
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
      codes = codes + "<br>" + new_code
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write(codes)

我想我可以尝试对所有这些 put() 进行批处理，因此我创建了一个代码列表 (code_list)。本地完成需要2-5分钟。

不使用bulkuploader 选项是否可以更快地完成此操作？因为显然我收到了 500 服务器错误。或者也许可以通过连续的调用或步骤来完成......

原文

I'm trying to create and store 20000 random codes in my local datastore, before trying this in appspot... This is the model

class PromotionCode (db.Model):
  code = db.StringProperty(required=True)

And this is the class that handles the populate request (only a logged admin may use it). It creates random alphanumeric codes and tries to store 20000 of them in the datastore:

class Populate(webapp.RequestHandler):

  def GenerateCode(self):
    chars = string.letters + string.digits
    code = ""
    for i in range(8):
      code = code + choice(chars)
    return code.upper()

  def get(self):
    codes = "";
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
      codes = codes + "<br>" + new_code
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write(codes)

I thought I could try batching all those put(), so I created a list of codes (code_list). It takes 2-5 minutes to do it locally.

Is it possible to do it faster without using the bulkuploader option? Because I'm getting the 500 server error, obviously. Or maybe doing it in consecutive calls or steps...

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

你的呼吸 2024-12-18 00:55:52

为什么不直接更改上面的代码以一次插入 100 个，然后

for i in {1..200}
  do
    curl --cookie "ACSID=your-acsid-cookie" http://your-app-id.appspot.com/populatepath
    sleep(5)
  done

从命令行运行类似以下内容？无论如何，条目都是随机的，您不需要记住任何状态。

您可以通过手动登录并从浏览器检查 cookie 来获取 ACSID cookie。

请求之间的睡眠将阻止您启动大量实例或达到短期配额。

如果这是您需要自动化的事情，那么任务队列建议很好，但如果这是一次性的事情，您最好保持简单。

Why not just change your code above to insert 100 at a time, and just run something like:

for i in {1..200}
  do
    curl --cookie "ACSID=your-acsid-cookie" http://your-app-id.appspot.com/populatepath
    sleep(5)
  done

from your command line? The entries are random anyway, you don't need to remember any state.

You can get the ACSID cookie by logging in manually and inspecting the cookies from your browser.

The sleep between requests will prevent you from spinning up a gigantic number of instances or hitting short-term quotas.

The task queue suggestion is good if this is something you need to automate, but if it's a one-time thing you might as well keep it simple.

回复收藏 0 原文

深白境迁sunset 2024-12-18 00:55:52

您可以在任务队列中对进程进行批处理吗？

将批处理大小设置为较高的任务队列...

您可以更快地归档它

回复收藏 0 原文

雨的味道风的声音 2024-12-18 00:55:52

我不明白为什么你必须提前创建 20,000 个，而不是根据需要即时创建每个，但我敢打赌你可以大大加快你的代码速度。像这样的事情（未经测试）：

class Populate(webapp.RequestHandler):

  chars = "AB...Z01...9"

  def GenerateCode(self):
    return ''.join(choice(chars) for _ in xrange(8))

  def get(self):
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write("done")

不打印代码可能会节省时间。

我相信这里的其他人可以做得更好......

I don't understand why you have to create 20,000 in advance as opposed to creating each as needed on the fly, but I bet you could speed up your code quite a bit. Something like this (untested):

class Populate(webapp.RequestHandler):

  chars = "AB...Z01...9"

  def GenerateCode(self):
    return ''.join(choice(chars) for _ in xrange(8))

  def get(self):
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write("done")

Not printing out the codes may save time.

I'm sure others here can do better...

回复收藏 0 原文