使用 xpath 在 clojure 中解析 rss feed

Question

我正在尝试解析这段 rss

<rss xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        <title>
            Signal RSS - full
        </title>
        <link>
            https://www.mystery.com
        </link>
        <description>
            null
        </description>
        <pubDate>
            Wed, 09 Mar 2022 14:07:31 GMT
        </pubDate>
        <lastBuildDate>
            Wed, 09 Mar 2022 14:07:31 GMT
        </lastBuildDate>
        <item>
            <guid isPermaLink="false">
                someid
            </guid>
            <description>
                -- other text
            </description>
            <text>
                BC-AT&amp;T-Discovery-Start-Mega-Bond-Sale-in-Test-of-Uneasy-Market
            </text>
            <content medium="document" expression="custom" type="text/vnd.IPTC.NewsML" lang="EN" url="https://api.com/syndication/newsml/v12/news/R8FRGG3/a715dac7-5282-4422-be8e" />
        </item>
    </channel>
</rss>

相当标准，对吧？

使用 https://kyleburton.github.io/clj-xpath/site/ 我将其修改为：

(ns clj-xpath-examples.core
  (:require
   [clojure.string :as string]
   [clojure.pprint :as pp])
  (:use
   clj-xpath.core))

(def input (slurp '.pathToXml.xml'))

(xml->doc input)

这给了我这个我无法理解的错误：

; IllegalAccessException class clojure.lang.Reflector cannot access class com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl (in module java.xml) because module java.xml does not export com.sun.org.apache.xerces.internal.jaxp to unnamed module @689eb690  jdk.internal.reflect.Reflection.newIllegalAccessException (Reflection.java:392)

我哪里出错了？如果我可以使用 xpath 来实现这一点，那么我的解决方案就会更加简洁。

Answer 1

这是一种方法：

(ns tst.demo.core
  (:use tupelo.core tupelo.test)
  (:require
    [clojure.walk :as walk]
    [tupelo.forest :as forest]
    [tupelo.parse.xml :as xml]
    [tupelo.string :as str]
    ))

(def xml-str
  (str/quotes->double "
      <rss xmlns:media='http://search.yahoo.com/mrss/' version='2.0'>
        <channel>
            <title>
                Signal RSS - full
            </title>
            <link>
                https://www.mystery.com
            </link>
            <description>
                null
            </description>
            <pubDate>
                Wed, 09 Mar 2022 14:07:31 GMT
            </pubDate>
            <lastBuildDate>
                Wed, 09 Mar 2022 14:07:31 GMT
            </lastBuildDate>
            <item>
                <guid isPermaLink='false'>
                    someid
                </guid>
                <description>
                    -- other text
                </description>
                <text>
                    BC-AT&T-Discovery-Start-Mega-Bond-Sale-in-Test-of-Uneasy-Market
                </text>
                <content medium='document' expression='custom' type='text/vnd.IPTC.NewsML' lang='EN' url='https://api.com/syndication/newsml/v12/news/R8FRGG3/a715dac7-5282-4422-be8e' />
            </item>
        </channel>
    </rss> "))

使用单元测试：

(dotest
  (let [enlive-raw  (xml/parse xml-str)
        enlive-nice (walk/postwalk (fn [item]
                                     (if (string? item)
                                       (str/trim item)
                                       item))
                      enlive-raw)]
    (is= enlive-nice
      {:attrs   {:version "2.0" :xmlns:media "http://search.yahoo.com/mrss/"}
       :content [{:attrs   {}
                  :content [{:attrs {} :content ["Signal RSS - full"] :tag :title}
                            {:attrs {} :content ["https://www.mystery.com"] :tag :link}
                            {:attrs {} :content ["null"] :tag :description}
                            {:attrs   {}
                             :content ["Wed, 09 Mar 2022 14:07:31 GMT"]
                             :tag     :pubDate}
                            {:attrs   {}
                             :content ["Wed, 09 Mar 2022 14:07:31 GMT"]
                             :tag     :lastBuildDate}
                            {:attrs   {}
                             :content [{:attrs {:isPermaLink "false"} :content ["someid"] :tag :guid}
                                       {:attrs {} :content ["-- other text"] :tag :description}
                                       {:attrs   {}
                                        :content ["BC-AT&T-Discovery-Start-Mega-Bond-Sale-in-Test-of-Uneasy-Market"]
                                        :tag     :text}
                                       {:attrs   {:expression "custom"
                                                  :lang       "EN"
                                                  :medium     "document"
                                                  :type       "text/vnd.IPTC.NewsML"
                                                  :url        "https://api.com/syndication/newsml/v12/news/R8FRGG3/a715dac7-5282-4422-be8e"}
                                        :content []
                                        :tag     :content}]
                             :tag     :item}]
                  :tag     :channel}]
       :tag :rss})))

使用 我最喜欢的模板项目 进行构建。

---

PS您可能还对 Tupelo Forest 库：

(forest/enlive->hiccup enlive-nice) => 
[:rss
 {:version "2.0", :xmlns:media "http://search.yahoo.com/mrss/"}
 [:channel
  [:title "Signal RSS - full"]
  [:link "https://www.mystery.com"]
  [:description "null"]
  [:pubDate "Wed, 09 Mar 2022 14:07:31 GMT"]
  [:lastBuildDate "Wed, 09 Mar 2022 14:07:31 GMT"]
  [:item
   [:guid {:isPermaLink "false"} "someid"]
   [:description "-- other text"]
   [:text
    "BC-AT&T-Discovery-Start-Mega-Bond-Sale-in-Test-of-Uneasy-Market"]
   [:content
    {:expression "custom",
     :lang "EN",
     :medium "document",
     :type "text/vnd.IPTC.NewsML",
     :url
     "https://api.com/syndication/newsml/v12/news/R8FRGG3/a715dac7-5282-4422-be8e"}]]]]

---

PPS 您可能还对

• clojure.data.xml
• Hickory （和 博客)

Answer 2

首先，您可以使用 Clojure 的内置 clojure.xml/parse 将 RSS 信息获取到数据结构中。

repl> (require '[clojure.xml :as xml])
nil

repl> (require '[clojure.java.io :as io])
nil

repl> (xml/parse (io/file "search.yahoo.co.xml"))
{:tag :rss, :attrs {:version "2.0", :xmlns:media "http://search....

repl> (clojure.pprint/pprint *1)
{:tag :rss,
 :attrs {:version "2.0", :xmlns:media "http://search.yahoo.com/mrss/"},
 :content
 [{:tag :channel,
   :attrs nil,
   :content
   [{:tag :title,
     :attrs nil,
     :content ["\n            Signal RSS - full\n        "]}
    {:tag :link,
     :attrs nil,
     :content ["\n            https://www.mystery.com\n        "]}
    {:tag :description,
...

您对数据结构的处理方式取决于您想要从 RSS 中提取的内容。 clojure.xml/parse 中的结构是一棵树。如果您想要“所有链接”而不考虑树，那么有一个非常有趣的 Clojure 核心函数，它将树转换为节点序列，然后可以使用 map 或 <代码>过滤器。

repl> (xml/parse (io/file "search.yahoo.co.xml"))

repl> (def s (xml-seq *1))
#'rssp.core/s

repl> (count s)
20

repl> (map :tag s)
(:rss :channel :title nil :link nil :description nil :pubDate ...

repl> (filter #(= :link (:tag %)) s)
({:tag :link, :attrs nil, :content ["\n https://www.mystery.com\n  "]})

如果您想驾驶乌龟绕着树转，在每个节点上上下左右查看，那么请查看内置函数 clojure.zip/xml-zip。

所有这些功能的文档可以在 https://clojure.github.io 找到/clojure/clojure.core-api.html