RegEx Tokenizer 将文本拆分为单词、数字和标点符号

发布于 2024-10-21 03:00:09 字数 632 浏览 4 评论 0原文

我想做的就是把一个文本拆分成他的终极元素。

例如：

from nltk.tokenize import *
txt = "A sample sentences with digits like 2.119,99 or 2,99 are awesome."
regexp_tokenize(txt, pattern='(?:(?!\d)\w)+|\S+')
['A','sample','sentences','with','digits','like','2.199,99','or','2,99','are','awesome','.']

您可以看到它运行良好。我的问题是：如果数字位于文本末尾会发生什么？

txt = "Today it's 07.May 2011. Or 2.999."
regexp_tokenize(txt, pattern='(?:(?!\d)\w)+|\S+') 
['Today', 'it', "'s", '07.May', '2011.', 'Or', '2.999.']

结果应该是： ['Today', 'it', "s", '07.May', '2011','.', 'Or', '2.999','.']

我必须做什么才能得到上面的结果？

原文

What I want to do is to split a text into his ultimate elements.

For example:

from nltk.tokenize import *
txt = "A sample sentences with digits like 2.119,99 or 2,99 are awesome."
regexp_tokenize(txt, pattern='(?:(?!\d)\w)+|\S+')
['A','sample','sentences','with','digits','like','2.199,99','or','2,99','are','awesome','.']

You can see it works fine. My Problem is: What happens if the digit is at the end of a text?

txt = "Today it's 07.May 2011. Or 2.999."
regexp_tokenize(txt, pattern='(?:(?!\d)\w)+|\S+') 
['Today', 'it', "'s", '07.May', '2011.', 'Or', '2.999.']

The result should be:
['Today', 'it', "'s", '07.May', '2011','.', 'Or', '2.999','.']

What I have to do to get the result above?

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

欲拥i 2024-10-28 03:00:09

我创建了一个模式来尝试包含单词、数字中出现的句号和逗号。希望这有帮助：

txt = "Today it's 07.May 2011. Or 2.999."
regexp_tokenize(txt, pattern=r'\w+([.,]\w+)*|\S+')
['Today', 'it', "'s", '07.May', '2011', '.', 'Or', '2.999', '.']

I created a pattern to try to include periods and commas occurring inside words, numbers. Hope this helps:

txt = "Today it's 07.May 2011. Or 2.999."
regexp_tokenize(txt, pattern=r'\w+([.,]\w+)*|\S+')
['Today', 'it', "'s", '07.May', '2011', '.', 'Or', '2.999', '.']

回复收藏 0 原文

~没有更多了~