That's normal. Mine was indexed a few months ago too. I think it's an issue with the crawler. edit: btw, you can delete the link on the webmaster tools website.
I wouldn't recommend disallowing the robots.txt file from itself. Instead you can use the X-Robots-Tag http header with a value of noindex which will prevent the search engines from Indexing the file.
This will allow them access to your robots.txt but prevent it from showing up in the search results.
发布评论
评论（4）
这很正常。我的也于几个月前被索引。我认为这是爬虫的问题。 编辑：顺便说一句，您可以删除网站站长工具网站上的链接。
That's normal. Mine was indexed a few months ago too. I think it's an issue with the crawler. edit: btw, you can delete the link on the webmaster tools website.
我不建议禁止 robots.txt 文件本身。相反，您可以使用带有值的 X-Robots-Tag http 标头 noindex 这将阻止搜索引擎索引该文件。
这将允许他们访问您的 robots.txt，但会阻止其显示在搜索结果中。
I wouldn't recommend disallowing the robots.txt file from itself. Instead you can use the X-Robots-Tag http header with a value of noindex which will prevent the search engines from Indexing the file.
This will allow them access to your robots.txt but prevent it from showing up in the search results.
我认为禁止您自己的 robots.txt 文件（如果可能）将导致其中写入的所有其他命令也被机器人忽略。
I Think that disallowing your own robots.txt file (if possible) will result in all the other commands written there to be ignored by bots also.
您可以将
Disallow: /robots.txt添加到 robots.txt。我认为它应该避免 robots.txt 本身被索引。You can add
Disallow: /robots.txtto your robots.txt. I think it should avoid the robots.txt itself from being indexed.