我想使用 GAE 抓取网站并将结果发布到 Google 实体中
进入每个链接并提取各种信息，例如权限、prims 等，然后将结果发布到 google 应用引擎上的实体中。
我想知道最好的方法是什么？
克里斯
Go into each of the links and extract out various pieces of information e.g. permissions, prims etc then post the results into a Entity on google app engine.
I want to know the best way to go about it?
Chris
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
为了使用纯 Python 库规范化 HTML，我在 html5lib 方面获得了比 BeautifulSoup 更好的体验。
然而，您只想提取简单的结构化信息，这实际上并不需要规范化 HTML。我在 Google App Engine 上有一些抓取应用程序，它们使用我自己的 xpath 库 适用于原始 HTML。
或者您可以使用正则表达式来完成一次性工作。
For normalizing HTML using a pure Python library I have had better experiences with html5lib than BeautifulSoup.
However you just want to extract simply structured information, which doesn't actually require normalizing the HTML. I have a few scraping apps on Google App Engine which use my own xpath library that works with raw HTML.
Or you can use regular expressions for one off jobs.
您可以在 Python 中使用几个不错的屏幕抓取库。
也许最简单的高级抓取工具是 scrapy。它依赖 Twisted 来实现主引擎，但提供了一个非常易于使用的界面来实现自定义抓取代码。
否则，您可以考虑使用 BeautifulSoup 或 Mechanize 提供“机械”浏览器实现。
BeautifulSoup 和 Mechanize 都应该在 App Engine 上开箱即用 - 它提供了一个围绕 httplib 和 urllib 的包装器，使用 urlfetch 作为后端。只有 scrapy 会出现问题，因为它使用了twisted。 [感谢尼克·约翰逊的更新]。
There are several nice screen scraping libraries you can use in Python.
Perhaps the easiest to knock up an advanced scraper with is scrapy. It relies on Twisted to implement the main engine but provides a very easy to use interface for implementing custom scraping code.
Otherwise you can look at doing it more manually with something like BeautifulSoup, or Mechanize which provides a "mechanical" browser implementation.
BeautifulSoup and Mechanize should both work out of the box on App Engine - it provides a wrapper around httplib and urllib that uses urlfetch as a backend. Only scrapy will be problematic, due to its use of twisted. [thanks to Nick Johnson for the update].