You may consider using HTMLunit It's a java class library made to automate browsing without having to control a browser, and it integrates the Mozilla Rhino Javascript engine to process javascript on the pages it loads. There's also a JRuby wrapper for that, named Celerity. Its javascript support is not really perfect right now, but if your pages don't use many hacks things should work fine the performance should be way better than controlling a browser. Furthermore, you don't have to worry about cookies being persisted after your scraping is over and all the other nasty things connected to controlling a browser (history, autocomplete, temp files etc).
Since you say that no AJAX is used, then all the info is present at the HTML source. The javascript just renders it based on user clicks. So you need to reverse engineer the way the application works, parse the html and the javascript code and extract the useful information. It is strictly business of text parsing - you shouldn't deal with running javascript and producing a new DOM. This would be much more difficult to do.
If AJAX was used, your job would be easier. You could easily find out how the AJAX services work (probably by receiving JSON and XML) and extract the information.
You could consider using a greasemonkey JS. greasemonkey is a very powerful Firefox add on that allows you to run your own script alongside that of specific web sites. This allows you to modify how the web site is displayed, add or remove content. You can even use it to do AJAX style lookups and add dynamic content.
If your tool is for in house use, and users are all happy to use Firefox then this could be a winner.
I suggest IRobotSoft web scraper. It is a dedicated free software for screen scraping with the best javascript support. You can create and test a robot with its visual interface. You can also embed it into your own application using its ActiveX control and hide the browser window.
I'd go with Perl's Win32::IE::Mechanize which lets you automate Internet Explorer. You should be able to click on icons and extract text while letting MSIE do the annoying tasks of processing all the JS.
I agree with kgiannakakis' answer. I'd be suprised if you couldn't reverse engineer the javascript to identify where the information comes from and then write some simple Python scripts using Urllib2 and the Beautiful Soup library to scrape the same information.
If Python and scraping are a new idea, there's some excellent tutorials available on how to get going.
[Edit] Looks like there's a Python version of mechanize too. Time to re-write some scrapers I developed a while back! :-)
I created a project site2archive that uses phantomJs to render including JS stuff and wget to scrape. phantomJs is based on Webkit, that delivers a similar browsing environment as Safari and Google Chrome.
发布评论
评论（7）
您可以考虑使用 HTMLunit
它是一个 Java 类库，用于自动浏览而无需控制浏览器，并且它集成了 Mozilla Rhino Javascript 引擎来处理其加载的页面上的 javascript。 还有一个 JRuby 包装器，名为 Celerity。 它的 javascript 支持现在还不是很完美，但是如果你的页面不使用很多 hacks，事情应该可以正常工作，性能应该比控制浏览器好得多。 此外，您不必担心抓取结束后 cookie 会被保留，也不必担心与控制浏览器相关的所有其他令人讨厌的事情（历史记录、自动完成、临时文件等）。
You may consider using HTMLunit
It's a java class library made to automate browsing without having to control a browser, and it integrates the Mozilla Rhino Javascript engine to process javascript on the pages it loads. There's also a JRuby wrapper for that, named Celerity. Its javascript support is not really perfect right now, but if your pages don't use many hacks things should work fine the performance should be way better than controlling a browser. Furthermore, you don't have to worry about cookies being persisted after your scraping is over and all the other nasty things connected to controlling a browser (history, autocomplete, temp files etc).
既然你说没有使用 AJAX，那么所有信息都存在于 HTML 源中。 JavaScript 只是根据用户的点击来呈现它。 因此，您需要对应用程序的工作方式进行逆向工程，解析 html 和 javascript 代码并提取有用的信息。 这完全是文本解析的事情——你不应该处理运行 javascript 和生成新的 DOM。 这样做会困难得多。
如果使用 AJAX，您的工作会更容易。 您可以轻松了解 AJAX 服务的工作原理（可能通过接收 JSON 和 XML）并提取信息。
Since you say that no AJAX is used, then all the info is present at the HTML source. The javascript just renders it based on user clicks. So you need to reverse engineer the way the application works, parse the html and the javascript code and extract the useful information. It is strictly business of text parsing - you shouldn't deal with running javascript and producing a new DOM. This would be much more difficult to do.
If AJAX was used, your job would be easier. You could easily find out how the AJAX services work (probably by receiving JSON and XML) and extract the information.
你可以考虑使用greasemonkey JS。 Greasemonkey 是一个非常强大的 Firefox 插件，它允许您运行自己的脚本以及特定网站的脚本。 这允许您修改网站的显示方式、添加或删除内容。 您甚至可以使用它进行 AJAX 风格的查找并添加动态内容。
如果您的工具供内部使用，并且用户都乐意使用 Firefox，那么这可能是一个赢家。
问候
You could consider using a greasemonkey JS. greasemonkey is a very powerful Firefox add on that allows you to run your own script alongside that of specific web sites. This allows you to modify how the web site is displayed, add or remove content. You can even use it to do AJAX style lookups and add dynamic content.
If your tool is for in house use, and users are all happy to use Firefox then this could be a winner.
Regards
我建议 IRobotSoft 网络抓取工具。 它是一款专门用于屏幕抓取的免费软件，具有最好的 JavaScript 支持。 您可以使用其可视化界面创建和测试机器人。 您还可以使用其 ActiveX 控件将其嵌入到您自己的应用程序中并隐藏浏览器窗口。
I suggest IRobotSoft web scraper. It is a dedicated free software for screen scraping with the best javascript support. You can create and test a robot with its visual interface. You can also embed it into your own application using its ActiveX control and hide the browser window.
我会选择 Perl 的 Win32 ::IE::Mechanize 可让您自动化 Internet Explorer。 您应该能够单击图标并提取文本，同时让 MSIE 执行处理所有 JS 的烦人任务。
I'd go with Perl's Win32::IE::Mechanize which lets you automate Internet Explorer. You should be able to click on icons and extract text while letting MSIE do the annoying tasks of processing all the JS.
我同意 kgiannakakis 的回答。 如果您无法对 javascript 进行逆向工程来识别信息的来源，然后使用 Urllib2 和 Beautiful Soup 库 抓取相同的信息。
如果 Python 和抓取是一个新想法，那么有一些很棒的 教程 提供如何开始使用的教程。
[编辑] 看起来还有机械化的Python 版本。 是时候重写我不久前开发的一些抓取工具了！ :-)
I agree with kgiannakakis' answer. I'd be suprised if you couldn't reverse engineer the javascript to identify where the information comes from and then write some simple Python scripts using Urllib2 and the Beautiful Soup library to scrape the same information.
If Python and scraping are a new idea, there's some excellent tutorials available on how to get going.
[Edit] Looks like there's a Python version of mechanize too. Time to re-write some scrapers I developed a while back! :-)
我创建了一个项目 site2archive ，它使用 phantomJs 来渲染，包括 JS 内容和 wget 进行抓取。 phantomJs 基于 Webkit，提供与 Safari 和 Google Chrome 类似的浏览环境。
I created a project site2archive that uses phantomJs to render including JS stuff and wget to scrape. phantomJs is based on Webkit, that delivers a similar browsing environment as Safari and Google Chrome.