Python lxml/beautiful soup 查找网页上的所有链接

发布于 2024-11-10 01:30:21 字数 285 浏览 1 评论 0原文

我正在编写一个脚本来读取网页，并构建符合特定条件的链接数据库。现在我陷入了 lxml 并理解如何从 html 中获取所有 's ...

result = self._openurl(self.mainurl)
content = result.read()
html = lxml.html.fromstring(content)
print lxml.html.find_rel_links(html,'href')

原文

I am writing a script to read a web page, and build a database of links that matches a certain criteria. Right now I am stuck with lxml and understanding how to grab all the <a href>'s from the html...

result = self._openurl(self.mainurl)
content = result.read()
html = lxml.html.fromstring(content)
print lxml.html.find_rel_links(html,'href')

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

冷︶言冷语的世界 2024-11-17 01:30:21

使用 XPath。像这样的东西（无法从这里测试）：

urls = html.xpath('//a/@href')

Use XPath. Something like (can't test from here):

urls = html.xpath('//a/@href')

回复收藏 0 原文

赠佳期 2024-11-17 01:30:21

通过 iterlinks，lxml 为此提供了一个出色的功能任务。

这会为操作、存档、背景、引用、classid、代码库、数据、href、longdesc、配置文件、src、usemap、dynsrc 中的每个链接生成（元素、属性、链接、pos），或 lowsrc 属性。

回复收藏 0 原文

时光倒影 2024-11-17 01:30:21

我想提供一种替代的基于 lxml 的解决方案。

解决方案使用lxml.cssselect中提供的函数

    import urllib
    import lxml.html
    from lxml.cssselect import CSSSelector
    connection = urllib.urlopen('http://www.yourTargetURL/')
    dom =  lxml.html.fromstring(connection.read())
    selAnchor = CSSSelector('a')
    foundElements = selAnchor(dom)
    print [e.get('href') for e in foundElements]

I want to provide an alternative lxml-based solution.

The solution uses the function provided in lxml.cssselect

    import urllib
    import lxml.html
    from lxml.cssselect import CSSSelector
    connection = urllib.urlopen('http://www.yourTargetURL/')
    dom =  lxml.html.fromstring(connection.read())
    selAnchor = CSSSelector('a')
    foundElements = selAnchor(dom)
    print [e.get('href') for e in foundElements]

回复收藏 0 原文

一页 2024-11-17 01:30:21

你可以使用这个方法：

from urllib.parse import urljoin, urlparse
from lxml import html as lh
class Crawler:
     def __init__(self, start_url):
         self.start_url = start_url
         self.base_url = f'{urlparse(self.start_url).scheme}://{urlparse(self.start_url).netloc}'
         self.visited_urls = set()

     def fetch_urls(self, html):
         urls = []
         dom = lh.fromstring(html)
         for href in dom.xpath('//a/@href'):
              url = urljoin(self.base_url, href)
              if url not in self.visited_urls and url.startswith(self.base_url):
                   urls.append(url)
         return urls

You can use this method:

from urllib.parse import urljoin, urlparse
from lxml import html as lh
class Crawler:
     def __init__(self, start_url):
         self.start_url = start_url
         self.base_url = f'{urlparse(self.start_url).scheme}://{urlparse(self.start_url).netloc}'
         self.visited_urls = set()

     def fetch_urls(self, html):
         urls = []
         dom = lh.fromstring(html)
         for href in dom.xpath('//a/@href'):
              url = urljoin(self.base_url, href)
              if url not in self.visited_urls and url.startswith(self.base_url):
                   urls.append(url)
         return urls

回复收藏 0 原文

~没有更多了~