抓取 Ajax - 使用 python
我正在尝试用 python 废弃 youtube 中的一个页面，其中有很多 ajax
我每次都必须调用 java 脚本来获取信息。 但我不太确定该怎么做。 我正在使用 urllib2 模块来打开 URL。 任何帮助，将不胜感激。
I'm trying to scrap a page in youtube with python which has lot of ajax in it
I've to call the java script each time to get the info. But i'm not really sure how to go about it. I'm using the urllib2 module to open URLs. Any help would be appreciated.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
YouTube（以及 Google 制作的其他所有产品）已经拥有广泛的 API，可让您访问您可能想要的几乎所有数据。
有关详细信息，请参阅 Youtube 数据 API。
我使用 urllib 发出 API 请求，并使用 ElementTree 解析返回的 XML。
Youtube (and everything else Google makes) have EXTENSIVE APIs already in place for giving you access to just about any and all data you could possibly want.
Take a look at The Youtube Data API for more information.
I use urllib to make the API requests and ElementTree to parse the returned XML.
您可以使用 Wireshark 之类的东西嗅探网络流量，然后通过足够强大的抓取框架重放 HTTP 调用处理 AJAX，例如 scraPY。
You could sniff the network traffic with something like Wireshark then replay the HTTP calls via a scraping framework that is robust enough to deal with AJAX, such as scraPY.
主要问题是，您违反了 YouTube 网站的 TOS（服务条款）。 如果您坚持不懈，Youtube 工程师和律师将尽其专业的最大努力追踪您，并为您树立榜样。 如果你对这个前景感到满意，那么，就你的想法而言——从技术上讲，你最好的选择是 python-spidermonkey 和 selenium。 我想将技术提示记录下来，以防将来任何人有像您的问题标题所示的需求，如果您继续这项特定的努力，没有您显然会遇到的法律问题。
Main problem is, you're violating the TOS (terms of service) for the youtube site. Youtube engineers and lawyers will do their professional best to track you down and make an example of you if you persist. If you're happy with that prospect, then, on you head be it -- technically, your best bet are python-spidermonkey and selenium. I wanted to put the technical hints on record in case anybody in the future has needs like the ones your question's title indicates, without the legal issues you clearly have if you continue in this particular endeavor.
我的做法如下：在 Firefox 上安装 Firebug，然后在 Firebug 中打开 NET，然后单击 YouTube 上所需的链接。 现在看看发生了什么以及请求了哪些页面。 找到负责页面 AJAX 部分的那个。 现在您可以使用 urllib 或 Mechanize 来获取链接。 如果您可以通过这种方式提取相同的内容，那么您就拥有了所需的内容，然后只需解析该内容即可。 如果您不能以这种方式提取内容，那么这表明所请求的页面可能正在查看用户登录凭据、会话信息或其他标头字段，例如 HTTP_REFERER ... 等。然后您可能想查看更多内容广泛，如 scrapy ...等。我建议您始终首先遵循简单的路径。 祝你好运并快乐“负责任地”抓取！ :)
Here is how I would do it: Install Firebug on Firefox, then turn the NET on in firebug and click on the desired link on YouTube. Now see what happens and what pages are requested. Find the one that are responsible for the AJAX part of page. Now you can use urllib or Mechanize to fetch the link. If you CAN pull the same content this way, then you have what you are looking for, then just parse the content. If you CAN'T pull the content this way, then that would suggest that the requested page might be looking at user login credentials, sessions info or other header fields such as HTTP_REFERER ... etc. Then you might want to look at something more extensive like the scrapy ... etc. I would suggest that you always follow the simple path first. Good luck and happy "responsibly" scraping! :)
根据建议，您应该使用 YouTube API 来访问合法提供的数据。
关于抓取 AJAX 的一般问题，您可能需要考虑 scrapy 框架。 它为爬行和抓取网站提供了广泛的支持，并在底层使用 python-spidermonkey 来访问 javascript 链接。
As suggested, you should use the YouTube API to access the data made available legitimately.
Regarding the general question of scraping AJAX, you might want to consider the scrapy framework. It provides extensive support for crawling and scraping web sites and uses python-spidermonkey under the hood to access javascript links.