为什么我的话语lemmatization无法正常工作？

Question

嗨，Stackoverflow社区！ 长期读者，但首次海报。我目前正在尝试NLP的手，并且在阅读了一些有关此主题的论坛帖子后，我似乎无法让Lemmatizer正常工作（下面的功能）。比较我的原始文本与预处理的文本，除了柠檬水之外，所有清洁步骤都可以按预期工作。我什至尝试过指定语音的一部分：“ v”不要将单词默认为名词，并且仍然获得动词的基本形式（ex：ext：thew - ＆gt; turn，是 - 阅读）...但是这似乎不起作用。

欣赏另一套眼睛和反馈 - 谢谢！

# key imports

import pandas as pd
import numpy as np
from nltk import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from string import punctuation
from nltk.stem import WordNetLemmatizer
import contractions


# cleaning functions

def to_lower(text):
    '''
    Convert text to lowercase
    '''
    return text.lower()

def remove_punct(text):
    return ''.join(c for c in text if c not in punctuation)

def remove_stopwords(text):
    '''
    Removes stop words which don't have meaning (ex: is, the, a, etc.)
    '''
    additional_stopwords = ['app']

    stop_words = set(stopwords.words('english')) - set(['not','out','in']) 
    stop_words = stop_words.union(additional_stopwords)
    return ' '.join([w for w in nltk.word_tokenize(text) if not w in stop_words])

def fix_contractions(text):
    '''
    Expands contractions
    '''
    return contractions.fix(text)



# preprocessing pipeline

def preprocess(text):
    # convert to lower case
    lower_text = to_lower(text)
    sentence_tokens = sent_tokenize(lower_text)
    word_list = []      
            
    for each_sent in sentence_tokens:
        # fix contractions
        clean_text = fix_contractions(each_sent)
        # remove punctuation
        clean_text = remove_punct(clean_text)
        # filter out stop words
        clean_text = remove_stopwords(clean_text)
        # get base form of word
        wnl = WordNetLemmatizer()
        for part_of_speech in ['v']:
            lemmatized_word = wnl.lemmatize(clean_text, part_of_speech)
        # split the sentence into word tokens
        word_tokens = word_tokenize(lemmatized_word)
        for i in word_tokens:
            word_list.append(i)                     
    return word_list

# lemmatize not properly working to get base form of word
# ex: 'turned' still remains 'turned' without returning base form 'turn'
# ex: 'running' still remains 'running' without getting base form 'run'



sample_data = posts_with_text['post_text'].head(5)
print(sample_data)
sample_data.apply(preprocess)

Answer 1

柠檬酸与茎非常相似，因为它将一组变形的单词降低到一个普通词。不同之处在于，狐猴化将其变为其真实的根词，这称为引理。如果我们以“ Amaze”，“ Amazing”，“惊人的”为单词，那么所有这些的引理都是“ Amaze”。与通常会返回“惊人”的茎相比。通常，诱饵比茎更先进。

words = ['amaze', 'amazed', 'amazing']

我们将再次使用nltk进行诱饵。我们还需要确保下载了WordNet数据库，该数据库将作为我们的Lemmatizer的查找，以确保它产生了真正的引理。

import nltk

nltk.download('wordnet')

from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print([lemmatizer.lemmatize(word) for word in words] )
#outputs ['amaze', 'amazed', 'amazing']

显然没有发生任何事情，那是因为lemmatization要求我们还提供词性（pos）标签，这是基于语法的单词类别。例如名词，形容词或动词。在我们的情况下，我们可以将每个单词作为动词，然后可以这样实现：

from nltk.corpus import wordnet

print([lemmatizer.lemmatize(word, wordnet.VERB) for word in words])
#outputs['amaze', 'amaze', 'amaze']