如何使用 Selenium 和 Python 抓取 Stockrover 网站内的所有公司名称

Question

如果下面的代码从表中抓取第一个公司名称 IBM，我将如何编码以从表的第一列中抓取所有公司名称？

相关代码：

table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '#gridview-1070-record-2989')))

例如，我需要的下一个是#gridview-1070-record-2990等等。

当前结果：

IBM

期望结果：

IBM
Microsoft Corporation
Apple Corporation
Google
Tesla
etc.

完整代码：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
import pandas as pd

   
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
ser = Service("./chromedriver.exe")
browser = driver = webdriver.Chrome(service=ser)

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
wait = WebDriverWait(driver, 30)
driver.get("https://stockrover.com")
wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[1]/div/section[2]/div/ul/li[2]"))).click()
user = driver.find_element(By.NAME, "username")
password = driver.find_element(By.NAME, "password")
user.clear()
user.send_keys("vibajajo64")
password.clear()
password.send_keys("vincer64")
driver.find_element(By.NAME, "Sign In").click()
wait = WebDriverWait(driver, 30)


table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '#gridview-1070-record-2989')))
for tab in table:
  print(tab.text)

Answer 1

从网站内的所有 元素中提取并打印文本，例如 IBM、Microsoft Corporation 等stockrover，而不是 presence_of_all_elements_ located() 你需要诱导WebDriver等待 href="https://stackoverflow.com/a/64770041/7429447">visibility_of_all_elements_ located() 并且您可以使用以下任一定位器策略：

• 使用CSS_SELECTOR：

  print([WebDriverWait 中 my_elem 的 my_elem.text(driver, 20).until(EC.visibility_of_all_elements_ located((By.CSS_SELECTOR, "table[id^='gridview-1070-record']"))) ）]）
  

• 使用XPATH：

  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_ located((By.XPATH, "//table[starts-with(@id, 'gridview-1070-记录'）]”）））]）
  

• 注意：您必须添加以下导入：

  从 selenium.webdriver.support.ui 导入 WebDriverWait
  从 selenium.webdriver.common.by 导入
  从 selenium.webdriver.support 导入预期条件作为 EC
  

Answer 2

您可以将相关代码放入 for 循环中，然后格式化输入的字符串以根据索引进行搜索，这样

table = []
for i in range(2989,"""the number of the last record you need"""):
table.append(wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '#gridview-1070-record-{}'.format(i)))))

这将为您提供一个公司数组