获取不带标点符号的句子中的字数NLTK python

Question

我正在尝试使用 python 中的 nltk 获取句子中的字数

这是我编写的代码

import nltk

data = "Sample sentence, for checking. Here is an exclamation mark! Here is a question? This isn't an easy-task."

for i in nltk.sent_tokenize(data):
    print(nltk.word_tokenize(i))

这是输出

['Sample', 'sentence', ',', 'for', 'checking', '.']
['Here', 'is', 'an', 'exclamation', 'mark', '!']
['Here', 'is', 'a', 'question', '?']
['This', 'is', "n't", 'an', 'easy-task', '.']

有没有办法删除标点符号，防止 isn't 分成两部分单词并将 easy-task 分成两部分？

我需要的答案是这样的：

['Sample', 'sentence', 'for', 'checking']
['Here', 'is', 'an', 'exclamation', 'mark']
['Here', 'is', 'a', 'question']
['This', "isn't", 'an', 'easy', 'task']

我可以通过使用像输出这样的停用词来管理标点符号

import nltk

data = "Sample sentence, for checking. Here is an exclamation mark! Here is a question? This isn't an easy-task."

stopwords = [',', '.', '?', '!']

for i in nltk.sent_tokenize(data):
    for j in nltk.word_tokenize(i):
        if j not in stopwords:
            print(j, ', ', end="")
    print('\n')

：

Sample , sentence , for , checking , 

Here , is , an , exclamation , mark , 

Here , is , a , question , 

This , is , n't , an , easy-task , 

但这并不能解决 isn't 和 easy-task 问题。有办法做到这一点吗？ 谢谢

Answer 1

您可以使用不同的标记器来满足您的要求。

import nltk
import string
tokenizer = nltk.TweetTokenizer()

for i in nltk.sent_tokenize(data):
    print(i)
    print([x for x in tokenizer.tokenize(i) if x not in string.punctuation])

#op
['Sample', 'sentence', 'for', 'checking']
['Here', 'is', 'an', 'exclamation', 'mark']
['Here', 'is', 'a', 'question']
['This', "isn't", 'an', 'easy-task']