当我收到一条错误消息（如下）时，我正在开发一个简单的应用程序来从网站提取一些货币转换，说明他们没有自动提取政策。
禁止自动解压
<强>
禁止自动提取我们的内容。请参阅http://www.xe.com/errors/noautoextract.htm。
我真的无意违反他们的政策，但我很好奇他们是如何知道的。谁能启发我吗？
I was working on a simple application to pull some currency conversions from a website, when I received an error message (below) stating they had a no automated extraction policy.
Autoextraction Prohibited
Automated extraction of our content is prohibited. See http://www.xe.com/errors/noautoextract.htm.
I don't really have an intention of breaking their policy but I am curious as to how they can tell. Can anyone enlighten me?
发布评论
评论（4）
1) 用户代理
2) 引入 Javascript 弹出窗口。类似
Click OK to Enter的内容。3) 如果您不在 NAT 后面，则计算来自特定 IP 地址的每小时请求数。
有关更多详细信息，请查看此 Pycon 演讲 web-strategies-for-programming-websites-that-don-t-expected-it 作者：asheesh laroia。
另请参阅机器人排除标准。
有些网站还使用
4) 验证码和重新验证码
5) 重定向，这意味着您需要添加
HTTP Referrer来获取数据。1) User-Agent
2) Introducing a Javascript pop-up.Something like
Click OK to enter.3) Calculating number of request/hour from a particular ip address if you are not behind NAT.
For more detail take a look at this Pycon talk web-strategies-for-programming-websites-that-don-t-expected-it by asheesh laroia.
Also take a look at A Standard for Robot Exclusion.
Some web-sites also use
4) Captchas and Re-Captchas
5) Redirection which means you need to add a
HTTP Referrerto get your data.这是通过实施 机器人排除协议 在 HTTP 服务器级别完成的。
来自机器人排除标准
It is done at the HTTP Server level by implementing Robot Exclusion protocol.
From Robots exclusion standard
我认为他们至少会观察两个参数：
I think they watch at least two parameters :
基本上，如果您请求一个 URL 并且返回 HTML 页面，则该网站几乎对此无能为力 - 这就是网络服务器的用途。
但与人类请求页面相比，有几种技术可以阻止机器人。其中一些是对机器人“行为”的提示，另一些则试图检测机器人并阻止它。
Basically, if you request an URL and you get the HTML page back, there's pretty much nothing the site can do about it - and well, that's just what a webserver is for.
But there are several techniques to stop bots in contrast of a human being requesting the page. Some of them are hints for bots which "behave", others try to detect a bot and stop it.