阻止网络抓取工具
网站可以通过哪些方式阻止网络抓取工具？如何确定您的服务器是否被机器人访问？
What are ways that websites can block web scrapers? How can you identify if your server is being accessed by a bot?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（6）
简单的机器人无法从 Flash、图像或声音中抓取文本。
Simple bots can not scrap text from flash, images or sound.
不幸的是，您的问题类似于人们问如何阻止垃圾邮件。没有固定的答案，并且它不会阻止持续存在的某人/机器人。
然而，这里有一些可以实现的方法：
Unfortunately your question is similar to people asking how do you block spam. There's no fixed answer, and it won't stop someone/bot which is persistent.
However, here are some methods that can be implemented:
您可以使用 robots.txt 来阻止注意到它的机器人（但仍然允许通过其他已知实例，例如谷歌等） - 但这不会阻止那些忽略它的机器人。您也许可以从 Web 服务器日志中获取用户代理，或者您可以更新代码以将其记录在某处。如果您希望阻止特定用户代理访问您的网站，只需返回空/默认屏幕和/或特定服务器代码即可。
You can use robots.txt to block bots that take notice of it (but still let through other known instances such as google, etc) - but that won't stop those that ignore it. You may be able to get the user agent from your web server logs, or you could update your code to record it somewhere. If you then wanted you could block particular user agents from accessing your website, just be returning either a empty/default screen and/or a particular server code.
我不认为有一种方法可以完全满足您的需要，因为在网站爬虫/抓取器中，您可以在请求页面时编辑所有标头，例如用户代理，并且您将无法识别是否存在来自 Mozilla Firefox 的用户或只是一个爬虫/爬虫...
I don't think there is a way of doing exactly what you need, because in websites crawlers/scrapers you can edit all headers when requesting a page, like User-Agent, and you won't be able to identify if there is a user from Mozilla Firefox or just a scraper/crawler...
抓取器在某种程度上依赖于页面加载之间标记的一致性。如果您想让他们的日子不好过，请想出一种方法来提供不同请求之间更改的标记。
Scrapers rely to some extent on the consistency of markup from page load to page load. If you want to make life difficult for them, come up with a means of serving altered markup from request to request.
像“不良行为”这样的内容可能会有所帮助： http://www.bad-behavior.ioerror.us/
来自他们的网站：
Bad Behaviour 旨在集成到您基于 PHP 的网站中，尽早运行以在垃圾邮件机器人有机会用垃圾邮件破坏您的网站之前将其抛出，甚至抓取您的页面以获取电子邮件地址和要填写的表格。
不良行为不仅会阻止对您网站的实际破坏，还会阻止许多电子邮件地址收集者，从而导致电子邮件数量减少垃圾邮件和许多自动化网站破解工具，有助于提高网站的安全性。
Something like "Bad Behavior" might help: http://www.bad-behavior.ioerror.us/
From their site:
Bad Behavior is designed to integrate into your PHP-based Web site, running as early as possible to throw out spam bots before they have the opportunity to vandalize your site with their junk, or even to scrape your pages for e-mail addresses and forms to fill out.
Not only does Bad Behavior block actual vandalism to your site, it also blocks many e-mail address harvesters, resulting in less e-mail spam, and many automated Web site cracking tools, helping to improve your Web site’s security.