Web刮擦在Python中赢得了整个列表

Question

我是从Python开始的，在Python进行网络刮擦时，它不会显示整个列表，我将留在那里，我试图将A24的A24电影排在IMDB中

from cmath import e
from pydoc import synopsis
from bs4 import BeautifulSoup
import requests


try:
    source =requests.get('https://www.imdb.com/list/ls024372673/')
    source.raise_for_status()  

    soup=BeautifulSoup(source.text,'html.parser')
    movies=soup.find('div',class_="lister-list").find_all('div')
   
    for movie in movies :
        name= movie.find('h3',class_="lister-item-header").a.text

        rank= movie.find('span',class_="lister-item-index unbold text-primary").text
        
        year= movie.find('span',class_="lister-item-year text-muted unbold").text

        star= movie.find('span',class_="ipl-rating-star__rating").text
        
        metascore= movie.find('div',class_="inline-block ratings-metascore").span.text

        score=movie.find('div',class_="list-description").text

        genre=movie.find('span',class_="genre").text
        
        runtime=movie.find('span',class_="runtime").text

        about=movie.find('p',class_="").text
       
        elements = movie.findAll('span', attrs = {'name':'nv'})
        votes = elements[0]['data-value']
        gross = elements[1]['data-value']

    print(name,rank,year,star,metascore,score,genre,runtime,about,votes,gross)
except Exception as e:
         print(e) 

Answer 1

您最好检查尝试/除块中发生的情况，并处理异常，例如如果语句：

'score': movie.find('div',class_="list-description").text if movie.find('div',class_="list-description") else None,

示例

您还可以使用更结构化的方法来保持结果：

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0"}
page = requests.get('https://www.imdb.com/list/ls024372673/', headers=headers)
soup = BeautifulSoup(page.content)

data = []
for movie in soup.select('.lister-item'):
    data.append({
        'name': movie.find('h3',class_="lister-item-header").a.text,
        'rank': movie.find('span',class_="lister-item-index unbold text-primary").text,
        'year': movie.find('span',class_="lister-item-year text-muted unbold").text,
        'star': movie.find('span',class_="ipl-rating-star__rating").text,
        'metascore': movie.find('div',class_="inline-block ratings-metascore").span.text,
        'score': movie.find('div',class_="list-description").text if movie.find('div',class_="list-description") else None,
        'genre': movie.find('span',class_="genre").text.strip(),
        'runtime': movie.find('span',class_="runtime").text,
        'about': movie.find('p',class_="").text,
        'elements': movie.find_all('span', attrs = {'name':'nv'}),
        'votes': elements[0]['data-value'],
        'gross': elements[1]['data-value']
    })
data

Answer 2

电影这不是列表。您正在使用.find（）返回第一个找到元素。您必须使用.find_all（）返回列表。

另外，您正在寻找元素内部的所有项目，其中class =“ lister-list”，但是这样，您只会得到一个元素，而不是电影列表。您应该使用class =“ lister-item-content”搜索所有元素。

source = requests.get("https://www.imdb.com/list/ls024372673/")
source.raise_for_status()  

soup = BeautifulSoup(source.text, "html.parser")
movies = soup.find_all("div", class_="lister-item-content")

for movie in movies:
    name      = (movie.find("h3", class_="lister-item-header").find("a").text).strip()
    rank      = (movie.find("span", class_="lister-item-index unbold text-primary").text).strip()
    year      = (movie.find("span", class_="lister-item-year text-muted unbold").text).strip()
    stars     = (movie.find("span", class_="ipl-rating-star__rating").text).strip()
    metascore = (movie.find("div", class_="inline-block ratings-metascore").find("span").text).strip()
    # score   = movie.find("div", class_="list-description").text // There isn't this class inside movie
    genre     = (movie.find("span", class_="genre").text).strip()
    runtime   = (movie.find("span", class_="runtime").text).strip()
    about     = (movie.find("p", class_="").text).strip()

    elements = movie.findAll("span", attrs = {"name":"nv"})
    votes    = elements[0]['data-value']
    gross    = elements[1]['data-value']

另一个问题是得分变量。电影元素中没有div class =“ list-description” 。您将获得一个错误，因为它将返回没有属性文本的nonepe对象。我还添加了一个.strip（）以删除空格。

编辑：我同意 hedgehog 。他的示例是这种代码结构的完美解决方案。只需记住添加.strip（）即可。