Modernizr 和/或 yepnope 可以对机器人和蜘蛛做出反应吗?
我在页面上运行一些 JS,弹出一个模式本地化选择框。我想防止机器人/爬虫发生这种情况。有没有办法使用 Modernizr 和/或 yepnope.js 来做到这一点?
I have some JS running on a page which pops up a modal localisation select box. I would like to prevent this from happening for bots /crawlers. Is there a way to do this using Modernizr and / or yepnope.js?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
大多数机器人根本不会运行任何 Javascript 代码。他们会下载您的代码并检查它以了解它的功能或内容,或者特定机器人的功能是什么,但他们通常不需要实际运行任何 JS 代码来执行此操作。
那些运行 JS 代码的机器人将在不同的环境中运行。有些可能会模拟浏览器环境,有些可能会在实际的浏览器中运行。但是,如果不了解您可能遇到的所有机器人的内部结构,您实际上无法使用 Modernizr 这样的工具来确定解决它。
当谈到查看最终用户的浏览器时,进行浏览器检测被认为是一件坏事,而功能检测(即 Modernizr)被认为是更好的解决方案。
然而,由于我上面讨论的因素,当涉及到机器人时,规则会发生变化,并且根据用户代理字符串对它们进行检测确实变得合法。
任何合法的机器人都会提供一个 UA 字符串来标识自己,并且应该很容易被发现。
如果机器人不提供可识别的 UA 字符串,那么它就不是合法的机器人,因此您没有理由对其进行任何特殊处理以使其生活更轻松。
Most bots won't run any Javascript code at all. They'll download your code and examine it to see what it does or what it's about, or whatever that particular bot's function is, but they generally don't need to actually run any JS code to do this.
Those that do run JS code will do so in an environment that will differ from bot to bot. Some may simulate a browser environment, others may run in an actual brower. But without knowing about the innards of all the bots you're likely to encounter, you really won't be able to solve it for sure with a tool like Modernizr.
When it comes to looking at your end users's browser, doing browser detection is considered a bad thing, and feature detection (ie Modernizr) is considered a better solution.
However the rules change when it comes to bots, because of the factors I've discussed above, and it does indeed become legitimate to do detection for them based on their user-agent string.
Any legitimate bot will provide a UA string that identifies itself as such, and should be fairly easy to spot.
If a bot isn't providing an identifiable UA string, then it isn't a legitimate bot, so there's no reason for you to give it any special treatment to make life easier for it.