如何在Python中使用gensim进行字符串语义匹配？

Question

在Python中，我们如何确定一个字符串与我们的短语是否有语义关系？

示例：

我们的短语是：

'Fruit and Vegetables'

我们想要检查语义关系的字符串是：

'I have an apple in my basket', 'I have a car in my house'

结果：

正如我们所知的第一项I have an apple in my篮子与我们的短语有关。

Answer 1

您可以使用 gensim 库来实现 MatchSemantic 并将这样的代码编写为函数（在此处查看完整代码）：

初始化

---

1. 安装gensim和numpy：

pip install numpy
pip install gensim

代码

---

1. 首先，我们必须实现要求，

from re import sub
import numpy as np
from gensim.utils import simple_preprocess
import gensim.downloader as api
from gensim.corpora import Dictionary
from gensim.models import TfidfModel
from gensim.similarities import SparseTermSimilarityMatrix, WordEmbeddingSimilarityIndex, SoftCosineSimilarity

2. 使用这个函数来检查字符串和句子是否与你想要的短语匹配。

def MatchSemantic(query_string, documents):
    stopwords = ['the', 'and', 'are', 'a']

    if len(documents) == 1: documents.append('')

    def preprocess(doc):
        # Tokenize, clean up input document string
        doc = sub(r'<img[^<>]+(>|$)', " image_token ", doc)
        doc = sub(r'<[^<>]+(>|$)', " ", doc)
        doc = sub(r'\[img_assist[^]]*?\]', " ", doc)
        doc = sub(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', " url_token ", doc)
        return [token for token in simple_preprocess(doc, min_len=0, max_len=float("inf")) if token not in stopwords]

    # Preprocess the documents, including the query string
    corpus = [preprocess(document) for document in documents]
    query = preprocess(query_string)

    # Load the model: this is a big file, can take a while to download and open
    glove = api.load("glove-wiki-gigaword-50")
    similarity_index = WordEmbeddingSimilarityIndex(glove)

    # Build the term dictionary, TF-idf model
    dictionary = Dictionary(corpus + [query])
    tfidf = TfidfModel(dictionary=dictionary)

    # Create the term similarity matrix.
    similarity_matrix = SparseTermSimilarityMatrix(similarity_index, dictionary, tfidf)

    query_tf = tfidf[dictionary.doc2bow(query)]

    index = SoftCosineSimilarity(
        tfidf[[dictionary.doc2bow(document) for document in corpus]],
        similarity_matrix)

    return index[query_tf]

注意：
如果第一次运行代码，进程栏将从 0% 变为 100% 以下载 glove-wiki-gigaword-50 gensim ，之后一切都将被设置，您可以简单地运行代码。

例如，用法

---

中的任何句子或项目匹配，

，我们想要查看 Fruit and Vegetables 是否与 documents Test:

query_string = 'Fruit and Vegetables'
documents = ['I have an apple on my basket', 'I have a car in my house']
MatchSemantic(query_string, documents)

---

因此我们知道第一个项目我的篮子里有一个苹果与水果和蔬菜有语义关系，因此它的分数将为0.189，而第二个项目则没有关系会被发现，所以它的分数将是0

输出：

0.189    # I have an apple in my basket
0.000    # I have a car in my house