Python请求无法获取网站的所有链接

Question

我正在学习如何使用 Python urllib.requests 模块，并且我一直在尝试从网站获取所有链接，尽管它适用于大多数链接，但我在打开 这个。

我为此链接获得的输出只是： # python teosa.py []

整个代码如下所示：

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import re

req = Request("https://randomtube.xyz/")
html_page = urlopen(req)

soup = BeautifulSoup(html_page, "lxml")

links = []
for link in soup.findAll('a'):
    links.append(link.get('href'))

print(links) 

有谁知道可能是什么问题？

Answer 1

问题是 url 中没有给出链接。

您的脚本应该按原样运行。尝试一个带有链接的网站。

注意：我认为您不需要“导入重新”。

#!/usr/bin/python3
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
#import re

req = Request("https://google.com/")
html_page = urlopen(req)

soup = BeautifulSoup(html_page, "lxml")

links = []
for link in soup.findAll('a'):
    links.append(link.get('href'))

print(links)

['https://www.google.com/imghp?hl=en&tab=wi', 'https://maps.google.com/maps?hl=en&tab=wl', 'https://play.google.com/?hl=en&tab=w8', 'https://www.youtube.com/?gl=US&tab=w1', 'https://news.google.com/?tab=wn', 'https://mail.google.com/mail/?tab=wm', 'https://drive.google.com/?tab=wo', 'https://www.google.com/intl/en/about/products?tab=wh', 'http://www.google.com/history/optout?hl=en', '/preferences?hl=en', 'https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ', '/advanced_search?hl=en&authuser=0', '/intl/en/ads/', '/services/', '/intl/en/about.html', '/intl/en/policies/privacy/', '/intl/en/policies/terms/']