Python+webdriver访问一个js渲染的网页为什么在headless无法完全渲染

Question

Python+webdriver访问一个js渲染的网页为什么在headless无法完全渲染

如下code：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

options = webdriver.ChromeOptions()
executable_path = "/Users/shurachow/Downloads/chromedriver"
# options.add_argument('--disable-gpu')
# options.add_argument('--headless')
options.add_argument('--hide-scrollbars')

mobileEmulation = {'deviceName': 'iPhone 8'}
options.add_experimental_option('mobileEmulation', mobileEmulation)
options.add_argument('lang=zh_CN.UTF-8')
options.add_argument('Accept="application/json, text/javascript"')

prefs = {
    'profile.default_content_setting_values': {
        'images': 1,
        'javascript': 1,
        'geolocation': 1
    }
}
options.add_experimental_option("prefs", prefs)
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=options)

browser.get('https://daojia.jd.com')
browser.execute_script("""
    var __s__script = document.createElement('script');
    const scriptContent = `
    localStorage.setItem("_current_address_", '{"city":"武汉市","longitude":114.30525,"latitude":30.59276,"areaCode":1381,"districtCode":1386,"address":"湖北省武汉市江岸区沿江大道188号","district":"江岸区","title":"武汉市","adcode":"420102","requestId":"92746b90-d3d0-11e8-8978-246e96285542","poiId":"","cityId":1381,"_hasLogin_":false}');
     `;
    __s__script.innerHTML = scriptContent;
    document.head.prepend(__s__script)
""")
try:
    WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.index-shop-wrap')))
    print 'good'
except:
    print 'bad'

browser.close()

如果把--headless的注释打开
页面一直渲染不出.index-shop-wrap
最终就得不到print 'good'的结果

反之在有chrome GUI的情况下，是可以的，是我忽视了哪儿了么？

望指点。

Answer 1

我也遇到这个问题了. 在爬天眼查的时候不加headless数据可以正常显示, 但是加了之后就不行了,会丢失一部分数据.并且数据也不全