当前位置：文江博客话题详情

我可以阻止 Apache Web 服务器上每个站点的搜索爬虫吗？

发布于 2024-07-07 22:19:06 字数 215 浏览 12 评论 0原文

我在公共互联网上有一个临时服务器，运行一些网站的生产代码副本。如果暂存站点被索引，我真的不喜欢它。

有没有办法修改临时服务器上的 httpd.conf 以阻止搜索引擎爬虫？

更改 robots.txt 并不会真正起作用，因为我使用脚本将相同的代码库复制到两台服务器。另外，我也不想更改虚拟主机配置文件，因为有很多站点，并且我不想在创建新站点时必须记住复制某个设置。

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

滥情哥ㄟ 2024-07-14 22:19:06

创建包含以下内容的 robots.txt 文件：

User-agent: *
Disallow: /

将该文件放在临时服务器上的某个位置；您的目录根目录是一个很好的地方（例如/var/www/html/robots.txt）。

将以下内容添加到您的 httpd.conf 文件中：

# Exclude all robots
<Location "/robots.txt">
    SetHandler None
</Location>
Alias /robots.txt /path/to/robots.txt

SetHandler 指令可能不是必需的，但如果您使用像 mod_python 这样的处理程序，则可能需要它。

该 robots.txt 文件现在将为服务器上的所有虚拟主机提供服务，从而覆盖您可能为各个主机拥有的任何 robots.txt 文件。

（注意：我的答案本质上与 ceejayoz 的答案建议你做的事情相同，但我不得不额外花几分钟弄清楚所有细节才能使其发挥作用。为了其他人的利益，我决定将这个答案放在这里谁可能会偶然发现这个问题。）

Create a robots.txt file with the following contents:

User-agent: *
Disallow: /

Put that file somewhere on your staging server; your directory root is a great place for it (e.g. /var/www/html/robots.txt).

Add the following to your httpd.conf file:

# Exclude all robots
<Location "/robots.txt">
    SetHandler None
</Location>
Alias /robots.txt /path/to/robots.txt

The SetHandler directive is probably not required, but it might be needed if you're using a handler like mod_python, for example.

That robots.txt file will now be served for all virtual hosts on your server, overriding any robots.txt file you might have for individual hosts.

(Note: My answer is essentially the same thing that ceejayoz's answer is suggesting you do, but I had to spend a few extra minutes figuring out all the specifics to get it to work. I decided to put this answer here for the sake of others who might stumble upon this question.)

回复收藏 0 原文

终难愈 2024-07-14 22:19:06

您可以使用 Apache 的 mod_rewrite 来完成此操作。假设您的真实主机是 www.example.com，您的临时主机是 staging.example.com。创建一个名为“robots-staging.txt”的文件，并有条件地重写请求以转到该文件。

这个例子适合保护单个临时站点，比您所要求的用例稍微简单一些，但这对我来说已经可靠地工作了：

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Dissuade web spiders from crawling the staging site
  RewriteCond %{HTTP_HOST}  ^staging\.example\.com$
  RewriteRule ^robots.txt$ robots-staging.txt [L]
</IfModule>

您可以尝试将蜘蛛重定向到不同的主robots.txt服务器，但是
有些蜘蛛在从 HTTP 请求中获得除“200 OK”或“404 not found”返回代码之外的任何内容后可能会犹豫不决，并且它们可能不会读取重定向的 URL。

这样做的方法如下：

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Redirect web spiders to a robots.txt file elsewhere (possibly unreliable)
  RewriteRule ^robots.txt$ http://www.example.com/robots-staging.txt [R]
</IfModule>

You can use Apache's mod_rewrite to do it. Let's assume that your real host is www.example.com and your staging host is staging.example.com. Create a file called 'robots-staging.txt' and conditionally rewrite the request to go to that.

This example would be suitable for protecting a single staging site, a bit of a simpler use case than what you are asking for, but this has worked reliably for me:

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Dissuade web spiders from crawling the staging site
  RewriteCond %{HTTP_HOST}  ^staging\.example\.com$
  RewriteRule ^robots.txt$ robots-staging.txt [L]
</IfModule>

You could try to redirect the spiders to a master robots.txt on a different server, but
some of the spiders may balk after they get anything other than a "200 OK" or "404 not found" return code from the HTTP request, and they may not read the redirected URL.

Here's how you would do that:

<IfModule mod_rewrite.c>
  RewriteEngine on

  # Redirect web spiders to a robots.txt file elsewhere (possibly unreliable)
  RewriteRule ^robots.txt$ http://www.example.com/robots-staging.txt [R]
</IfModule>

回复收藏 0 原文