you won't stop the crawler from indexing robots.txt because its a chicken and the egg situation, however, if you aren't specifying google and other search engines to look directly at the sitemap, you could lose some indexing weight from denying your sitemap.xml. Is there a particular reason why you would want to not have users be able to see the sitemap? I actually do this which is specific just for the google crawler:
发布评论
评论（1）
您不会阻止爬虫对 robots.txt 进行索引，因为这是先有鸡还是先有蛋的情况，但是，如果您没有指定 google 和其他搜索引擎直接查看站点地图，则可能会因拒绝您的内容而损失一些索引权重。站点地图.xml。
您不希望用户看到站点地图是否有特殊原因？
我实际上这样做是专门针对谷歌爬虫的：
you won't stop the crawler from indexing robots.txt because its a chicken and the egg situation, however, if you aren't specifying google and other search engines to look directly at the sitemap, you could lose some indexing weight from denying your sitemap.xml.
Is there a particular reason why you would want to not have users be able to see the sitemap?
I actually do this which is specific just for the google crawler: