Javascript 的 Python Scraper？

发布于 2024-09-03 03:17:43 字数 487 浏览 6 评论 0原文

谁能指导我找到一个好的 Python 屏幕抓取库来获取 JavaScript 代码（希望有好的文档/教程）？我想看看有哪些选择，但最重要的是最容易学习且效果最快......想知道是否有人有经验。我听说过一些关于蜘蛛猴的东西，但也许还有更好的？

具体来说，我使用 BeautifulSoup 和 Mechanize 到达这里，但需要一种方法来打开 javascript 弹出窗口、提交数据并下载/解析 javascript 弹出窗口中的结果。

<a href="javascript:openFindItem(12510109)" onclick="s_objectID=&quot;javascript:openFindItem(12510109)_1&quot;;return this.s_oc?this.s_oc(e):true">Find Item</a>

我想用 Google App 引擎和 Django 来实现这个。谢谢！

原文

Can anyone direct me to a good Python screen scraping library for javascript code (hopefully one with good documentation/tutorials)? I'd like to see what options are out there, but most of all the easiest to learn with fastest results... wondering if anyone had experience. I've heard some stuff about spidermonkey, but maybe there are better ones out there?

Specifically, I use BeautifulSoup and Mechanize to get to here, but need a way to open the javascript popup, submit data, and download/parse the results in the javascript popup.

<a href="javascript:openFindItem(12510109)" onclick="s_objectID="javascript:openFindItem(12510109)_1";return this.s_oc?this.s_oc(e):true">Find Item</a>

I'd like to implement this with Google App engine and Django. Thanks!

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

极致的悲 2024-09-10 03:17:43

在这些情况下，我通常做的是自动化实际的浏览器，并从那里获取处理后的 HTML。

编辑：

以下是在页面加载后自动将 InternetExplorer 导航至 URL 并获取标题和位置的示例。

from win32com.client import Dispatch

from ctypes import Structure, pointer, windll
from ctypes import c_int, c_long, c_uint
import win32con
import pywintypes

class POINT(Structure):
    _fields_ = [('x', c_long),
                ('y', c_long)]
    def __init__( self, x=0, y=0 ):
        self.x = x
        self.y = y

class MSG(Structure):
    _fields_ = [('hwnd', c_int),
                ('message', c_uint),
                ('wParam', c_int),
                ('lParam', c_int),
                ('time', c_int),
                ('pt', POINT)]

def wait_until_ready(ie):
    pMsg = pointer(MSG())
    NULL = c_int(win32con.NULL)

    while True:

        while windll.user32.PeekMessageW(pMsg, NULL, 0, 0, win32con.PM_REMOVE) != 0:
            windll.user32.TranslateMessage(pMsg)
            windll.user32.DispatchMessageW(pMsg)

        if ie.ReadyState == 4:
            break


ie = Dispatch("InternetExplorer.Application")

ie.Visible = True

ie.Navigate("http://google.com/")

wait_until_ready(ie)

print "title:", ie.Document.Title
print "location:", ie.Document.location

What I usually do is automate an actual browser in these cases, and grab the processed HTML from there.

Edit:

Here's an example of automating InternetExplorer to navigate to a URL and grab the title and location after the page loads.

from win32com.client import Dispatch

from ctypes import Structure, pointer, windll
from ctypes import c_int, c_long, c_uint
import win32con
import pywintypes

class POINT(Structure):
    _fields_ = [('x', c_long),
                ('y', c_long)]
    def __init__( self, x=0, y=0 ):
        self.x = x
        self.y = y

class MSG(Structure):
    _fields_ = [('hwnd', c_int),
                ('message', c_uint),
                ('wParam', c_int),
                ('lParam', c_int),
                ('time', c_int),
                ('pt', POINT)]

def wait_until_ready(ie):
    pMsg = pointer(MSG())
    NULL = c_int(win32con.NULL)

    while True:

        while windll.user32.PeekMessageW(pMsg, NULL, 0, 0, win32con.PM_REMOVE) != 0:
            windll.user32.TranslateMessage(pMsg)
            windll.user32.DispatchMessageW(pMsg)

        if ie.ReadyState == 4:
            break


ie = Dispatch("InternetExplorer.Application")

ie.Visible = True

ie.Navigate("http://google.com/")

wait_until_ready(ie)

print "title:", ie.Document.Title
print "location:", ie.Document.location

回复收藏 0 原文