仅使用 python、html 下载 amazon.co.uk 网页，就像 firebug 看到的那样
我注意到使用 urllib 下载网页：
http:// /www.amazon.co.uk/Darkness-II-Limited-PC-DVD/dp/B005ULLEX6
我使用 urlopen( url ).read() 返回的内容和firebug看到的不一样。
示例：
如果你将firebug指向页面的图像区域，它会告诉你存在一个div id="prodImageCell"，但是当查看python打开的内容时，没有这样的事情，因此 beautifulsoup 没有找到任何东西。
这是因为图像是使用 javascript 生成的吗？
问题：
如果是这样，有没有一种方法可以使用 urllib 下载与 firebug 看到的几乎完全相同的东西（而不是使用像 Selenium 这样的东西）。
我正在尝试以编程方式获取其中一张图像的源网址，这里的示例是带有 prodImageCell 的 div 具有 src=http://ecx.images-amazon.com/images/I/51uPDvJGS3L。AA300.jpg 这确实是图像的 url。
答案：
无法正确回答，因为我没有声誉:(
感谢@huelbois为我指明了正确的方向，找到了解决方案，需要使用用户代理标头
之前
>>> import urllib2
>>> import re
>>> site = urllib2.urlopen('http://www.amazon.co.uk/\
Darkness-II-Limited-PC-DVD/dp/B005ULLEX6').read()
>>> re.search( 'prodImageCell', site )
>>>
之后
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101\
Firefox/7.0.1"
>>> headers = {'User-Agent':user_agent}
>>> req = urllib2.Request(url=url,headers=headers)
>>> site = urllib2.urlopen(req).read()
>>> re.search( 'prodImageCell', site )
<_sre.SRE_Match object at 0x01487DB0>
万岁！
I noticed that using urllib to download a webpage:
http://www.amazon.co.uk/Darkness-II-Limited-PC-DVD/dp/B005ULLEX6
the content that I get back using urlopen( url ).read() is different from what firebug sees.
Example:
If you point firebug to the page's image area, it tells you a div id="prodImageCell" exists, however when looking through what python has opened, there is no such thing, therefore beautifulsoup doesn't find anything.
Is this because the images are generated using javascript?
Question:
If so is there a way of downloading pretty much the exact same thing firebug sees using urllib (and not using something like Selenium instead).
I am trying to fetch the source url of one of the images programmatically, example here is the div with prodImageCell has src=http://ecx.images-amazon.com/images/I/51uPDvJGS3L.AA300.jpg which is indeed the url to the image.
Answer:
can't answer properly because I don't have the reputation :(
Found the solution thanks to @huelbois for pointing me in the right direction, one needs to use user-agents headers.
Before
>>> import urllib2
>>> import re
>>> site = urllib2.urlopen('http://www.amazon.co.uk/\
Darkness-II-Limited-PC-DVD/dp/B005ULLEX6').read()
>>> re.search( 'prodImageCell', site )
>>>
After
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101\
Firefox/7.0.1"
>>> headers = {'User-Agent':user_agent}
>>> req = urllib2.Request(url=url,headers=headers)
>>> site = urllib2.urlopen(req).read()
>>> re.search( 'prodImageCell', site )
<_sre.SRE_Match object at 0x01487DB0>
hurrah!
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
刚刚用 wget 测试了它（将像 urrlib 一样工作）。
您必须包含 User-Agent 标头才能获取请求的部分：
wget -O- --header='User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1' http://www.amazon.co.uk/Darkness-II-Limited-PC-DVD/dp/B005ULLEX6
返回包含请求部分的 html 页面。
哎呀：刚刚看到你按照我之前的建议成功了。伟大的！
Just tested it right now with wget (will work like urrlib).
You HAVE to include User-Agent header to get the requested part:
wget -O- --header='User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1' http://www.amazon.co.uk/Darkness-II-Limited-PC-DVD/dp/B005ULLEX6
returns the html page with the requested part.
oops: just saw right now you succeeded with my previous advice. Great!