支持 JS 的 Html 抓取
我正在尝试抓取公司网页以实现自动化目的，但页面中的嵌入式脚本阻止我完全复制请求。最大的痛苦在于脚本生成的cookie。
我想过使用 Watin 自动化 IE，但我对服务应用程序下的这个解决方案感到不舒服。
在这种情况下你有什么建议？
提前致谢。
I am trying to scrape a company web page for automation purposes but the embedded scripts in the page prevent me to fully replicate the request. The biggest pain is in the script generated cookies.
I thought of automating IE with Watin but I am not comfortable with this solution under a service application.
What are your advices in this situation?
Thanks in advance.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
screen-scraper 是另一个旨在易于使用的工具（基于 java）。
基本思想正如 Byron 所说 - 你必须弄清楚正在设置哪些 cookie（Web 代理工具，例如 Fiddler< /a>、Charles 或 Firebug 和 Chrome 的开发工具会派上用场）。
因此，您不必阅读甚至执行页面上的 javascript 来模仿相同的请求。只需使用代理工具查看您的浏览器正在向服务器发送哪些 cookie，一旦您知道网站期望接收哪些 cookie，就可以在您用于抓取的任何脚本或工具中手动设置它们，这样您就成功了。
screen-scraper is another tool (java based) that aims at being easy to use.
The basic idea is as Byron said- you will have to figure out what cookies are getting set (web proxy tools like Fiddler, Charles, or browser extensions like Firebug and Chrome's dev tools will come in handy).
So, you don't necessarily have to read or even execute the javascript on the page to imitate the same requests. Just use a proxy tool to see what cookies your browser is sending to the server, and once you know what cookies the site expects to receive, set them manually in whatever script or tool you use to do your scraping and you'll be golden.
您有多种选择。
最简单的方法是在脚本中生成 cookie。您必须自己阅读 JavaScript 代码并弄清楚它在做什么并进行复制。 Fiddler 永远是您抓取时的朋友。
Htmlunit 是一个支持 JavaScript 的 Java Web 浏览器库。它没有 GUI，专为测试 Web 应用程序而设计。
Selenium 驱动浏览器的方式与 watir 非常相似，但它对大多数主要语言都有丰富的 api 支持。
You have several options.
The easiest is to generate the cookies in your script. You will have to read the javascript code yourself and figure out what it is doing and duplicate. Fiddler is always your friend when scraping.
Htmlunit is a java web browser library with JavaScript support. It has no gui and is made for testing web applications.
Selenium will driver a browser much the same way watir does, but it has rich api support for most major languages.