为了获得这种细粒度的控制，您最好在 HTML 中使用 robots 元标记。假设问题中的文件都是 HTML。
<meta name="robots" content="noindex" />
这应该放在文档的头部。
我发现这些标签也比 robots.txt 更容易维护。
To get that sort of fine grained control, you might be better off using a robots meta tag in your HTML. That is assuming the files in questions are all HTML.
<meta name="robots" content="noindex" />
This should be placed in the head of your document.
I find these tags easier to maintain than robots.txt as well.
There is a non-standard extension to the robots.txt format for specifying "Allow" rules. Not every bot honors it, and some bots process them differently than others.
发布评论
评论（2）
为了获得这种细粒度的控制，您最好在 HTML 中使用 robots 元标记。假设问题中的文件都是 HTML。
这应该放在文档的头部。
我发现这些标签也比 robots.txt 更容易维护。
To get that sort of fine grained control, you might be better off using a robots meta tag in your HTML. That is assuming the files in questions are all HTML.
This should be placed in the head of your document.
I find these tags easier to maintain than robots.txt as well.
robots.txt 格式有一个非标准扩展，用于指定“允许”规则。并非每个机器人都会尊重它，并且某些机器人对它们的处理方式与其他机器人不同。
您可以在这篇维基百科文章中阅读更多相关信息： http://en.wikipedia.org/wiki /Robots_exclusion_standard#Allow_directive
There is a non-standard extension to the robots.txt format for specifying "Allow" rules. Not every bot honors it, and some bots process them differently than others.
You can read more about it in this Wikipedia article: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive