Please add robots.txt to your web site root to prevent most automatic crawlers from accessing you site.
You could also come up with a rule that redirects access to error page if Referrer is set (client comes through a link, search engine or such). This only allows direct access to the web site.
The real answer: do not bother
Trying to limit unwanted access will most likely just hurt legitimate visitors.
Setting up the limitations will cost more in time than the saved bandwidth is worth
If you bandwidth is really so expensive, change service provider
发布评论
评论（1）
请将 robots.txt 添加到您的网站根目录，以防止大多数自动抓取工具访问您的网站。
您还可以制定一条规则，如果 Referrer 已设置（客户端通过链接、搜索引擎等到达）。这仅允许直接访问该网站。
真正的答案：不要打扰
Please add robots.txt to your web site root to prevent most automatic crawlers from accessing you site.
You could also come up with a rule that redirects access to error page if Referrer is set (client comes through a link, search engine or such). This only allows direct access to the web site.
The real answer: do not bother