为什么当我尝试刮擦多个页面时，我无法获得所有文本

Question

我正在尝试刮擦多个IMSDB页面，以获取电影脚本以创建电影脚本的数据集。 我编写了此代码，

import pandas as pd
import numpy as np

#import seaborn as sns
import matplotlib.pyplot as plt

import requests       #to send the request to the URL
from bs4 import BeautifulSoup
import numpy as np  # to count the values (in our case)
import selenium

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
  
from time import sleep
from random import randint


driver = webdriver.Chrome(ChromeDriverManager().install())

scriptsList=[]
newScript=[]
titles=[]
movie_titles = pd.read_csv("movies.csv")
l=0;
url_list=[]


for index,row in movie_titles.iterrows():
    movieString=movie_titles.loc[index]["title"]
    count=0
    #print(row)
    #assigning the URL with variable name url
    movieString=movie_titles.loc[index]["title"]
    movieString=str(movieString)
    titles.append(movieString)
    movieString=movieString.replace(" ", "")
    url = 'https://imsdb.com/scripts/'+movieString+'.html'
    url_list.append(url)

for i in url_list:
      # Target URL
  driver.get(url)
  
  # print(driver.title)
  
  # Printing the whole body text
  jt=driver.find_element_by_xpath("/html/body").text
  jt = jt.strip('\n')
  jt = jt.strip('\t')
  print(jt)
  scriptsList.append(jt)

  # Closing the driver
driver.close()

scripts_DF = pd.DataFrame({'title': titles, 'Script': scriptsList})
scripts_DF.to_csv('NewScripts6.csv')'''

但是代码并未打印所有文本仅打印此代码，

 ALL SCRIPTS





Writers :
Genres :


User Comments



Back to IMSDb





Index    |    Submit    |    Link to IMSDb    |    Disclaimer    |    Privacy policy    |    Contact
The Internet Movie Script Database (IMSDb)

The web's largest
movie script resource!

Search IMSDb

Alphabetical
# A B C D E F G H
I J K L M N O P Q
R S T U V W X Y Z

Genre
Action Adventure Animation
Comedy Crime Drama
Family Fantasy Film-Noir
Horror Musical Mystery
Romance Sci-Fi Short
Thriller War Western

我还编写了此代码，

import pandas as pd
import numpy as np

#import seaborn as sns
import matplotlib.pyplot as plt

import requests       #to send the request to the URL
from bs4 import BeautifulSoup
import numpy as np  # to count the values (in our case)
import selenium

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
  
driver = webdriver.Chrome(ChromeDriverManager().install())


# Importing necessary modules
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
  
# WebDriver Chrome
driver = webdriver.Chrome(ChromeDriverManager().install())
  
# Target URL
#driver.get("https://www.geeksforgeeks.org/competitive-programming-a-complete-guide/")

driver.get("https://imsdb.com/scripts/Toy-Story.html")

# print(driver.title)
  
# Printing the whole body text
print(driver.find_element_by_xpath("/html/body").text)
  
# Closing the driver
driver.close() 

该代码打印了网站的所有文本，任何人都可以帮助我刮擦多个页面并从中获取所有文本。我认为我需要在程序中添加时间延迟，因为网站无法处理这么多请求

Answer 1

用这个代码给我打印所有文本

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://imsdb.com/Movie%20Scripts/Joker%20Script.html")
jt = driver.find_element(by=By.XPATH, value="/html/body").text
print(jt)

Answer 2

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
list=["Cable%20Guy%20Script","Joker%20Script","Capote%20Script"]


for film in list:
 driver.get("https://imsdb.com/Movie%20Scripts/"+film+".html")
 jt = driver.find_element(by=By.XPATH, value="/html/body").text
 print(jt.strip('\n').strip('\t'))

输出很长，所以我附上一件

Capote Script
IMSDb opinion
 None available

IMSDb rating
 Not available
Average user rating
 None available

Writers
  Dan Futterman

Genres
 Crime
Drama

Movie Release Date : February 2006

Read "Capote" Script

User Comments for Capote

添加您自己的评论
*名称：电子邮件：