如何在弹出窗口中将硒链接到webccrape的第二页。

Question

我正在尝试为各种结果覆盖。第一页工作正常，但是当我切换到下一页时，不幸的是，它只是再次将结果的第一页网络覆盖。结果不会返回新的URL，因此它不起作用，而是在URL打开页面顶部的窗口。我似乎也无法弄清楚如何附加第一页的结果以添加第二页，它们以单独的列表出现。以下是我拥有的代码。

from selenium import webdriver
import time
import requests
from bs4 import BeautifulSoup
from selenium.webdriver.common.keys import Keys

#original webscraping code to get the names of locations from page 1
url = r'https://autochek.africa/en/ng/fix-your-car/service/scheduled-car-service'
driver = webdriver.Chrome()
driver.get(url)
xpath_get_locations = r'/html/body/div[1]/div/div[2]/div/div[1]/div/div[2]/div[2]/div/div/form/div[7]/div/label'

driver.find_element_by_xpath(xpath_get_locations).click()

soup = BeautifulSoup(driver.page_source, 'html.parser')


location_results = [i.text for i in soup.find_all('div', {'class': 'jsx-1642469937 state'})]

print(location_results)
time.sleep(3)

#finished page 1, finding the next button to go to page 2
xpath_find_next_button = r'/html/body/div[1]/div/div[2]/div/div[1]/div/div[2]/div[2]/div[2]/div[2]/div/div/div[3]/ul/li[13]'
driver.find_element_by_xpath(xpath_find_next_button).click()

#getting the locations from page 2
second_page_results = [i.text for i in soup.find_all('div', {'class': 'jsx-1642469937 state'})]



print(second_page_results)
time.sleep(2)

Answer 1

加载新页面或在页面上运行一些JavaScript代码后，您必须再次运行

soup = BeautifulSoup(driver.page_source, 'html.parser')

才能使用新的HTML。

---

或Skip BeautifulSoup，并在selenium中完成所有操作。

使用find_elements _...在Word Elements中使用char s 。

items = driver.find_elements_by_xpath('//div[@class="jsx-1642469937 state"]')

location_result = [i.text for i in items]

---

一句：

（xpath不需要前缀r，因为它不使用\）

顺便说 更可读的xpath。

#xpath_get_locations = r'/html/body/div[1]/div/div[2]/div/div[1]/div/div[2]/div[2]/div/div/form/div[7]/div/label'

xpath_get_locations = '//label[text()="Drop-off at Autochek location"]'

使用按钮Next＆gt;而不是搜索按钮2，3等会更简单。

xpath_find_next_button = '//li[@class="next-li"]/a'

---

编辑：

完整的工作代码，使用时 -loop访问所有页面。

我添加了模块webdriver_manager自动下载浏览器的驱动程序（Fresh）驱动程序。

我使用find_elemens（by.xpath，...）是因为find_elemens_by_xpath（...） is deprected。

from selenium import webdriver
from selenium.webdriver.common.by import By
#from selenium.webdriver.common.keys import Keys
#from selenium.webdriver.support.ui import WebDriverWait
#from selenium.webdriver.support import expected_conditions as EC
#from selenium.common.exceptions import NoSuchElementException, TimeoutException

import time
#from bs4 import BeautifulSoup

from webdriver_manager.chrome import ChromeDriverManager
#from webdriver_manager.firefox import GeckoDriverManager

driver = webdriver.Chrome(executable_path=ChromeDriverManager().install())
#driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())

# ---

url = 'https://autochek.africa/en/ng/fix-your-car/service/scheduled-car-service'
driver.get(url)

#xpath_get_locations = r'/html/body/div[1]/div/div[2]/div/div[1]/div/div[2]/div[2]/div/div/form/div[7]/div/label'
xpath_get_locations = '//label[text()="Drop-off at Autochek location"]'
driver.find_element(By.XPATH, xpath_get_locations).click()

# ---

all_locations = []

while True:
    
    # --- get locations on page
    
    time.sleep(1) # sometimes `JavaScript` may need time to add new items (and you can't catch it with `WebDriverWait`)

    #items = soup.find_all('div', {'class': 'jsx-1642469937 state'})
    items = driver.find_elements(By.XPATH, '//div[@class="jsx-1642469937 state"]')

    #soup = BeautifulSoup(driver.page_source, 'html.parser')

    locations = [i.text for i in items]

    print(locations)
    print('-------')

    all_locations += locations
    
    # --- find button `next >` and try to click it 
    
    #xpath_find_next_button = r'/html/body/div[1]/div/div[2]/div/div[1]/div/div[2]/div[2]/div[2]/div[2]/div/div/div[3]/ul/li[13]'
    xpath_find_next_button = '//li[@class="next-li"]/a'

    try:
        driver.find_element(By.XPATH, xpath_find_next_button).click()
    except:
        break  # exit loop
    
# ---

#driver.close()