解析网页时，无法从iframe（内部HTML页面）提取/加载所有HREF

Question

我真的在为这种情况而苦苦挣扎，并且整天都在尝试。请我需要您的帮助。我正在尝试刮擦此网页： 我想获得所有137 HREF-S（137个文档）。 使用的代码L：

   def test(self):
        final_url = 'https://decisions.scc-csc.ca/scc-csc/en/d/s/index.do?cont=&ref=&d1=2012-01-01&d2=2022-01-31&p=&col=1&su=16&or='
        self.driver.get(final_url)
        soup = BeautifulSoup(self.driver.page_source, 'html.parser')
        iframes = soup.find('iframe')
        src = iframes['src']
        base = 'https://decisions.scc-csc.ca/'
        main_url = urljoin(base, src)
        self.driver.get((main_url))
        browser = self.driver
        elem = browser.find_element_by_tag_name("body")
        no_of_pagedowns = 20
        while no_of_pagedowns:
            elem.send_keys(Keys.PAGE_DOWN)
            time.sleep(0.2)
            no_of_pagedowns -= 1

问题在于它仅加载25个第一文档（HREF），并且不知道该怎么做。

Answer 1

此代码向下滚动，直到可见所有元素，然后将PDF的URL保存在列表 pdfs 中。请注意，所有工作都是用硒完成的，而无需使用美丽的套件

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome(options=options, service=Service(your_chromedriver_path))
driver.get('https://decisions.scc-csc.ca/scc-csc/en/d/s/index.do?cont=&ref=&d1=2012-01-01&d2=2022-01-31&p=&col=1&su=16&or=')

# wait for the iframe to be loaded and then switch to it
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "decisia-iframe")))

# in this case number_of_results = 137
number_of_results = int(driver.find_element(By.XPATH, "//h2[contains(., 'result')]").text.split()[0])
pdfs = []

while len(pdfs) < number_of_results:
    pdfs = driver.find_elements(By.CSS_SELECTOR, 'a[title="Download the PDF version"]')
    # scroll down to the last visible row
    driver.execute_script('arguments[0].scrollIntoView({block: "center", behavior: "smooth"});', pdfs[-1])
    time.sleep(1)

pdfs = [pdf.get_attribute('href') for pdf in pdfs]