想要抓取一个网站
我正在寻找像 yelp.com 这样的网站，以获取他们在那里拥有的所有酒吧的列表。 有没有任何工具或脚本可以帮助我做到这一点。
I am looking to scrape a website like yelp.com, to get a listing of all the bars they have there.
Are there any tools or scripts out there which would help me do this.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
从 Python 角度来看
阅读同情屏幕抓取简介很好的教程，可以让您开始使用这两种工具。
From a Python perspective
Read An Introduction to Compassionate Screen Scraping for good tutorial to get you started that uses both tools.
如果您了解 Python，我觉得 pyQuery 模块很方便。与 jQuery 一样，它允许您使用增强的 CSS 选择器来选择 DOM 对象，我发现它比使用解析器容易得多。
If you know Python, there the pyQuery module that I find handy. Like jQuery, it lets you use enhanced CSS selectors to select DOM objects, I find it far easier than using a parser.
我使用 screen-scraper 在超级页面和 citySearch 上完成了类似的工作。您可以从那里将结果写入 CSV、数据库或其他文件。
I've done work like this on Superpages and citySearch using screen-scraper. From there you can write your results to a CSV, database, or whatever.
HTTrack - 它的跨平台，多年来一直使用它
HTTrack - its cross platform, been using this for years
我在互联网时代写了一个爬虫程序来从一些电子商务网站获取信息。我使用 Perl，每个站点都有两层代码。 “发现”层将解析并查找项目列表，“处理”层将读取产品页面和单独的数据字段并将它们输入数据库。
从您想做的事情来看，我认为推出自己的解决方案可能是最好的方法，因为它并不复杂。使用 Perl 或具有良好字符串处理和正则表达式支持的类似解释语言。
分离页面非常容易。忘记解析树（我采用了这种方式并放弃了它），手动识别与您想要的每条信息接壤的模板文本块并将其放在正则表达式上以提取数据要容易得多且直接。
把它们放在一个列表上，散列，无论什么，然后用它做你想做的事。
I wrote a scraper back in the dot-com era to suck info from a few e-commerce websites. I used perl and for each site had two tiers of code. The "discover" tier would parse and find lists of items and the "process" tier would read product pages and separate fields of data and feed them into a database.
From the looks of what you want to do I think rolling your own solution is probably the best approach as it's not really complicated. Use Perl or a similar interpreted language with good string processing and regex support.
Separating the pages is really easy. Forget about parse trees (I went that way and gave up on it), it's much easier and straightforward to manually identify the clumps of text of the template bordering each piece of info you want and put that on a regex to extract the data.
Put them on a list, hash, whatever and then do what you want with it.