如何使用RVEST和XML2从网页中获取所有嵌套URL？

Question

我正在尝试从下面的网页中获取所有嵌套链接。我的代码下面返回一个空的字符向量。

page1 <- "https://thrivemarket.com/c/condiments-sauces?cur_page=1"
page1 <- read_html(page1)
page1_body <- page1 %>% 
  html_node("body") %>% 
  html_children()

page1_urls <- page1 %>%
  rvest::html_nodes('body') %>%
  xml2::xml_find_all("//div[contains(@class, 'd85qmy-0 kRbsKs')]") %>%
  rvest::html_attr('href')

预先感谢您为此提供帮助。

最好，
〜梅拉

Answer 1

您要查找的链接在您正在阅读的HTML文档中不存在read_html。当您在浏览器中查看页面时，HTML文档包含浏览器运行的JavaScript代码。此JavaScript代码中的一些使您的浏览器下载更多信息，以插入您在浏览器上看到的网页。

就您而言，您要查找的额外信息是以JSON文件的形式进行的，您可以通过以下方式获取并解析：

library(httr)
library(dplyr)

url <- paste0("https://thrivemarket.com/api/v1/products",
              "?page_size=60&multifilter=1&cur_page=1")

content(GET(url))$products %>%
  lapply(function(x) data.frame(product = x$title, url = x$url)) %>%
  bind_rows() %>%
  as_tibble()
#> # A tibble: 60 x 2
#>    product                                          url                         
#>    <chr>                                            <chr>                       
#>  1 Organic Extra Virgin Olive Oil                   https://thrivemarket.com/p/~
#>  2 Grass-Fed Collagen Peptides                      https://thrivemarket.com/p/~
#>  3 Grass-Fed Beef Sticks, Original                  https://thrivemarket.com/p/~
#>  4 Organic Dry Roasted & Salted Cashews             https://thrivemarket.com/p/~
#>  5 Organic Vanilla Extract                          https://thrivemarket.com/p/~
#>  6 Organic Raw Cashews                              https://thrivemarket.com/p/~
#>  7 Organic Coconut Milk, Regular                    https://thrivemarket.com/p/~
#>  8 Organic Robust Maple Syrup, Grade A, Value Size  https://thrivemarket.com/p/~
#>  9 Organic Coconut Water                            https://thrivemarket.com/p/~
#> 10 Non-GMO Avocado Oil Potato Chips, Himalayan Salt https://thrivemarket.com/p/~
#> # ... with 50 more rows

^{在2022-06-04创建的 reprex软件包（v2.0.1）}