python urllib2 超时

Question

好吧，我已经在 google 和 stackoverflow 中搜索了这个答案，几个小时后没有看到执行此操作的工作脚本的正确答案....

这里我粘贴了 4 个假定的 python 工作脚本示例来设置不存在的 url 的默认超时，并使用套接字和/或超时参数设置超时。

没有人工作，因此永远不会触发超时。

有什么想法吗？

第一个例子：

import urllib2

try:                
    header_s = {"User-Agent":"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"}

    req = urllib2.Request("http://www.nonexistantdomainurl.com/notexist.php",headers = header_s)


    print urllib2.urlopen(req, None, 5.0).read()

except urllib2.URLError, e:
    print "Url Error: %r" % e

except Exception,e:
  print "Fallo de tipo ",e

else: 
    print "all ok!"

第二个例子：

import urllib2

try:
    response = urllib2.urlopen("http://www.nonexistantdomainurl.com/notexist.php", None, 2.5)
except urllib2.URLError, e:
    print "Oops, timed out?"

第三个例子：

from urllib2 import Request, urlopen, URLError, HTTPError
import base64


req = Request('http://www.nonexistantdomainurl.com/notexist.php')

try:
    response = urlopen(req,timeout=5.0)   

except HTTPError, e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError, e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason

第四个例子：

import urllib2
import socket


socket.setdefaulttimeout(5)

try:
    response = urllib2.urlopen("http://www.faluquito.com/equipo.php",timeout=5.0).read()   


except urllib2.URLError, e:
    print "Url Error: %r" % e

Answer 1

>>> import urllib2
>>> import time
>>> import contextlib
>>>
>>> def timeit():
...   s = time.time()
...   try:
...     yield
...   except urllib2.URLError:
...     pass
...   print 'took %.3f secs' % (time.time() - s)
...
>>> timeit = contextlib.contextmanager(timeit)
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 2)
...
took 2.002 secs
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 5)
...
took 5.003 secs

Answer 2

如果您的机器有 unix 程序 dig，您可能能够识别不存在的 url，如下所示：

import logging
import subprocess
import shlex

logging.basicConfig(level = logging.DEBUG,
                    format = '%(asctime)s %(module)s %(levelname)s: %(message)s',
                    datefmt = '%M:%S')
logger = logging.getLogger(__name__)

urls = ['http://1.2.3.4',
       "http://www.nonexistantdomainurl.com/notexist.php",
       "http://www.faluquito.com/equipo.php",
        'google.com']

nonexistent = ['63.251.179.13', '8.15.7.117']
for url in urls:
    logger.info('Trying {u}'.format(u=url))

    proc = subprocess.Popen(shlex.split(
        'dig +short +time=1 +retry=0 {u}'.format(u = url)),
                            stdout = subprocess.PIPE, stderr = subprocess.PIPE)
    out, err = proc.communicate()
    out = out.splitlines()
    logger.info(out)
    if any(addr in nonexistent for addr in out):
        logger.info('nonexistent\n')
    else:
        logger.info('success\n')

在我的机器上，这会产生：

00:57 test INFO: Trying http://1.2.3.4
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.nonexistantdomainurl.com/notexist.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.faluquito.com/equipo.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying google.com
00:58 test INFO: ['72.14.204.113', '72.14.204.100', '72.14.204.138', '72.14.204.102', '72.14.204.101']
00:58 test INFO: success

注意 dig 返回 ['63.251.179.13', '8.15.7.117']< /code> 对于不存在的 url。

我相信我的 ISP 正在将不存在的地址更改为 63.251.179.13 或 8.15.7.117。您的 ISP 可能会采取不同的措施。在这种情况下，您可能必须将 nonexistent 更改为其他内容。