googlebot 是否会抓取 jQuery $.get() 调用中的网址并且可以阻止吗？
我有一个页面，其中有一个使用 这个 ajaxForm jQuery 插件 的表单。表单提交，完成后，将使用 $.get() 调用将一些新内容加载到页面。
我的问题是，Googlebot“似乎”正在 $.get() 方法中对网址进行索引。
我的第一个问题是，这可能吗？我的印象是 Googlebot 大部分情况下都不会评估 javascript（我读过一些关于它能够使用 !# 对 url 上的内容进行索引的内容）。
我的第二个问题是，如果 Google 将此调用索引到该 url，有没有办法阻止它？
提前致谢。
I have a page that has a form using this ajaxForm jQuery plugin. The form submits, and when it's complete, there is a call using $.get() to load some new content to the page.
My problem is, the Googlebot "appears" to be indexing the url in the $.get() method.
My first question is, is that even possible? I was under the impression the Googlebot didn't evaluate javascript for the most part (I read something about it being able to index content on urls with !#).
My second question is, if Google is indexing this call to that url, is there a way to prevent it?
Thanks in advance.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
您可以专门
robots.txt该文件，googlebot将应该尊重它。来自robotstxt.org：
您还可以查看 Google 的网站站长中心从列表中删除该文件。
You could
robots.txtthe file specifically, googlebotwillshould honor it.From robotstxt.org:
You can also look at Google's Webmaster Central to remove the file from the listing.
首先，您需要检查这是否真的是 GoogleBot，因为任何人都可以假装是 GoogleBot，甚至是合法用户。
来源自Google 网站站长中心官方博客：如何验证 Googlebot。
First of all you need to check that that is really the GoogleBot because anyone can pretend being GoogleBot, even a legitimate user.
Sourced from Official Google Webmaster Central Blog: How to verify Googlebot.
googlebot 将 inline-javascript 中的几乎每个字符串解释为包含“/”或常见文件扩展名（“.html”、“.php”）的 URL...尤其是第一个非常非常烦人。
混淆内联 JS 中您不希望被抓取的每个 URL。即：将“/”替换为“|”在服务器端并在 JS 中创建一个包装方法来替换“|”再次到“/”。
是的，这很烦人，还有更好的方法，即：将所有 js 放在不可抓取的外部文件中。
robots.txt 解决方案并不是真正的解决方案。因为 URL 仍然会被发现，被推送到发现（谷歌使用管道来确定下一步要抓取的内容），但随后抓取会被阻止，这基本上是一个错失的机会。
googlebot interprets pretty much every string in inline-javascript as an URL that contains a "/" or a common file extenstion (".html", ".php") ... especially the first one is very very annoying.
confuscate every URL in inline JS that you do not want to get crawled. i.e.: replace "/" with '|' on the server side and make a wrapper method in JS that replaces "|" to "/" again.
yes, thats majorly annoying and there are better ways i.e.: having all your js in an external file that is not crawlable.
the robots.txt solution is not really a solution. because the URLs still get found, pushed to discovery (the pipe google uses to determine what to crawl next) but then the crawling is blocked, which is basically one missed opportunity.