ubuntu 下 python 使用 selenium + PhantomJS 时出错

Question

因为爬取的网站用到了AJAX，所以我想要使用selenium抓取数据。但是用selenium速度太慢，于是使用PhantomJS来加快速度。但是使用时会报错，各位帮忙看看吧。
爬取的网站是这个http://www.ncbi.nlm.nih.gov/pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)
爬的内容是每篇论文的链接。

ps:selenium用的是pip install -U selenium安装的，应该是最新版吧。Phantomjs用的是1.9.7版

#coding=utf-8
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
import re

domain = "http://www.ncbi.nlm.nih.gov/"
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)"
url = domain + url_tail

browser = webdriver.PhantomJS()
browser.get(url)


def extract_data(browser):
    links = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a")
    return [link.get_attribute("href") for link in links]

page_start, page_end = 1, 3

'''每次都从page_start开始'''
page_number_box = browser.find_element_by_xpath("//*[@id='pageno']").clear()
page_number_box_cleared = browser.find_element_by_xpath("//*[@id='pageno']")
page_number_box_cleared.send_keys(str(page_start) + Keys.RETURN)

'''建立文件links.txt,模式为append'''
with open('links.txt', 'a') as f:
    for page in range(page_start, page_end+1):
        f.write("page" + str(page) + '\n')
        WebDriverWait(browser, 20).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div.rprt p.title"))
        )
        '''写入链接尾部的数字'''
        for line in extract_data(browser):
            line_re = re.findall('\d+', line)
            f.write(str(line_re) + '\n')

        print "page %d" % page


        time.sleep(5)
        '''点击下一页'''
        browser.find_element_by_css_selector("div.pagination a.next").click()

browser.close()

报错：

Traceback (most recent call last):
  File "scrape_url.py", line 25, in <module>
    page_number_box = browser.find_element_by_xpath("//*[@id='pageno']").clear()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 232, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[@id='pageno']'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"101","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:46413","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"sessionId\": \"99f43900-cb27-11e4-99bd-f371f24cc826\", \"value\": \"//*[@id='pageno']\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/99f43900-cb27-11e4-99bd-f371f24cc826/element"}}
Screenshot: available via screen

先谢过各位了。

Answer 1

我最近也遇到，我认为是动态js还没解析，所以没获取到网页代码。异常是NoSuchElementException，很明显了。

Answer 2

还有一种可能，是由于phantomjs属于headless browser，是没有窗口的，那么可能所有元素都没有绘制。所以这时候你find任何元素都是NoSuchElementException异常。
可以尝试如下步骤：

browser = webdriver.PhantomJS()
browser.set_window_size(800, 600) # set browser size.
browser.get("http\:example.com") # Load page

参考：https://github.com/ariya/phantomjs/issues/11637

Answer 3

自己回答一下吧。
在stackoverflow上找到了一个解决方法。
屏蔽掉css，图片和js，以提高速度。
虽然PhantomJS还是不能用，但是确实速度变快了，目的达到就好。

firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.download.folderList", 2)
firefox_profile.set_preference("permissions.default.stylesheet", 2)
firefox_profile.set_preference("permissions.default.image", 2)
firefox_profile.set_preference("javascript.enable", False)

browser = webdriver.Firefox(firefox_profile=firefox_profile)

http://stackoverflow.com/questions/20892768/how-to-speed-up-browsing-in-selenium-firefox
http://stackoverflow.com/questions/17462884/is-selenium-slow-or-is-my-code-wrong

Answer 4

这样的话，岂不是js也无法解析了，为什么不用其他的更快的工具呢？

Answer 5

要加路径的 我也是折腾了一会才弄出来的 路径还要加\(下面复制出来就一个了)
browser = webdriver.PhantomJS(executable_path='D:\Program Files\Python35\Scripts\phantomjs\bin\phantomjs.exe')