请求HTML没有获得完整的HTML

Question

我有以下

            response=requests.get(item_url,headers=headers).text
            soup=BeautifulSoup(response,'lxml')
            print(soup)
            
            product=soup.find_all('a',class_='shelfProductTile-descriptionLink')
            print(product)
            price_per_weight=soup.find_all('div',class_='shelfProductTile-cupPrice ng-star-inserted')
            print(price_per_weight)

来自URL的代码： https://www.woolworths.com.au/shop/shop/search/products?searchterm = uncle%20Tobys%20OATS%20OATS%20500G&； SORTBY = TRADERREELELELELELELELELELELELELELELELELELELELELELELELELELELELELELELVANCE

我已经尝试了LXML和HTML.Parsers and html.parsers'' t在请求html中获取上述变量的类。我还尝试过按美丽的汤find_all find_all return none 但是，仍然获得产品和Price_per_weight的空列表。

可以使用美丽的汤来刮擦此信息，还是需要使用其他工具等工具？ （如果可能的话，我更喜欢不使用硒）。

Answer 1

您看到的数据是通过JavaScript从外部URL加载的，因此 BeautifulSoup 看不到它。要加载数据，您可以使用下一个示例：

import json
import requests

url = "https://www.woolworths.com.au/apis/ui/Search/products"

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0",
}

payload = {
    "Filters": [],
    "IsSpecial": False,
    "Location": "/shop/search/products?searchTerm=uncle%20tobys%20oats%20500g&sortBy=TraderRelevance",
    "PageNumber": 1,
    "PageSize": 36,
    "SearchTerm": "uncle tobys oats 500g",
    "SortType": "TraderRelevance",
}


with requests.session() as s:
    # load cookies
    s.get(
        "https://www.woolworths.com.au/shop/search/products?searchTerm=uncle%20tobys%20oats%20500g&sortBy=TraderRelevance",
        headers=headers,
    )

    # load actual data
    data = s.post(url, json=payload, headers=headers).json()

    # uncomment to print all data:
    # print(json.dumps(data, indent=4))

    for p in data["Products"]:
        print("{:<60} {}$".format(p["DisplayName"], p["Products"][0]["Price"]))

打印：

Uncle Tobys Oats Traditional Rolled Oats Porridge 500g       4.5$
Uncle Tobys Oats Quick Oats Porridge Porridge 500g           4.5$