有OnClick事件按钮时刮擦数据

Question

我正在尝试从“ https://lol.inven.co.kr/dataninfo/match/teamlist.php”中刮擦数据（对不起，网站不使用英语）

当我不clikc时，该网站没有“详细图表”

单击按钮时”显示出“详细信息”，我实际上想刮擦。

使用此虚拟网页

由于它是虚拟网页，因此请求URL（POST）没有任何规则。

在这种情况下，是单击OnClick按钮（“ eftectable”）时刮擦出现的唯一方法是Selenium？无论如何，我可以使用美丽的套装和请求吗？

欢迎任何想法。先感谢您！

Answer 1

您可以从该帖子请求中获取所有数据，但是首先您需要获得MatchCode值。

然后，制作post请求，然后解析XML您回来的。

例如：

import re
import time

import requests
from bs4 import BeautifulSoup

headers = {
    "Content-Type": "application/x-www-form-urlencoded;charset=utf-8",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"
}

with requests.Session() as s:
    soup = (
        BeautifulSoup(
            s.get(
                "https://lol.inven.co.kr/dataninfo/match/teamList.php",
                headers=headers,
            ).text,
            "lxml"
        ).select("#lolDbMatchTeamList .listDetail a")
    )

match_ids = [re.search(r"(\d+)", a["onclick"]).group(1) for a in soup]

for match_id in match_ids[:1]:  # just getting one match here
    r = s.post(
        f"https://lol.inven.co.kr/dataninfo/match/match_detail.xml.php?dummy={int(time.time())}",
        data={"matchcode": match_id},
    )
    player_names = (
        "\n".join(
            p.find("playername").getText() for p
            in BeautifulSoup(r.text, features="xml").select("player")
        )
    )
    print(player_names)

这将打印第一场比赛的播放器名称。

RNG Ming
RNG Wei
RNG Xiaohu
RNG GALA
RNG Ming
T1 Zeus
T1 Oner
T1 Faker
T1 Gumayusi
T1 Keria

这是您在每个post请求上获得的完整XML。

<?xml version="1.0" encoding="UTF-8"?>
<resultdata>
    <match>7345</match>
        <player>
            <code>125062</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>155</champ><champicon>Gwen_Square_0</champicon><champname>그웬</champname><spell1icon>lol_spell_ghost</spell1icon><spell1name>유체화</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>3</kill><death>0</death><assist>5</assist><ka>53</ka><kd>3.6</kd><asd>6</asd><kda>9.6</kda>    </player>
        <player>
            <code>125063</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002581</playercode><playername>RNG Wei</playername><mname>Wei</mname><champ>34</champ><champicon>LeeSin_Square_0_2</champicon><champname>리 신</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>2</kill><death>1</death><assist>11</assist><ka>87</ka><kd>2</kd><asd>11</asd><kda>13</kda>    </player>
        <player>
            <code>125064</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>99012</playercode><playername>RNG Xiaohu</playername><mname>Xiaohu</mname><champ>113</champ><champicon>Lissandra_Square_0_1</champicon><champname>리산드라</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>2</kill><death>2</death><assist>10</assist><ka>80</ka><kd>1</kd><asd>5</asd><kda>6</kda>    </player>
        <player>
            <code>125065</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002583</playercode><playername>RNG GALA</playername><mname>GALA</mname><champ>64</champ><champicon>Tristana_Square_0_4</champicon><champname>트리스타나</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>5</kill><death>0</death><assist>3</assist><ka>53</ka><kd>6</kd><asd>3.6</asd><kda>9.6</kda>   </player>
        <player>
            <code>125066</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>135</champ><champicon>Rakan_Square_0</champicon><champname>라칸</champname><spell1icon>lol_spell_ignite</spell1icon><spell1name>점화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>3</kill><death>0</death><assist>8</assist><ka>73</ka><kd>3.6</kd><asd>9.6</asd><kda>13.2</kda> </player>
        <player>
            <code>125067</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002552</playercode><playername>T1 Zeus</playername><mname>Zeus</mname><champ>100</champ><champicon>Jayce_Square_0_1</champicon><champname>제이스</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>1</kill><death>4</death><assist>0</assist><ka>33</ka><kd>0.3</kd><asd>0</asd><kda>0.3</kda>  </player>
        <player>
            <code>125068</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002559</playercode><playername>T1 Oner</playername><mname>Oner</mname><champ>80</champ><champicon>MonkeyKing_Square_0_2</champicon><champname>오공</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>1</kill><death>2</death><assist>2</assist><ka>100</ka><kd>0.5</kd><asd>1</asd><kda>1.5</kda>   </player>
        <player>
            <code>125069</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>135</playercode><playername>T1 Faker</playername><mname>Faker</mname><champ>89</champ><champicon>Ahri_Square_0_2</champicon><champname>아리</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>0</kill><death>5</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.4</asd><kda>0.4</kda>  </player>
        <player>
            <code>125070</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002255</playercode><playername>T1 Gumayusi</playername><mname>Gumayusi</mname><champ>129</champ><champicon>Jhin_Square_0</champicon><champname>진</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>1</kill><death>1</death><assist>1</assist><ka>67</ka><kd>1</kd><asd>1</asd><kda>2</kda>  </player>
        <player>
            <code>125071</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>900280</playercode><playername>T1 Keria</playername><mname>Keria</mname><champ>144</champ><champicon>yummi_square_0</champicon><champname>유미</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_heal</spell2icon><spell2name>회복</spell2name><kill>0</kill><death>3</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.7</asd><kda>0.7</kda>    </player>
    </resultdata>