比较文档中的文本频率与语料库中的频率
我想分析文档中的字母、二元组、单词等项目，并将它们在我的文档中出现的频率与它们在大型文档语料库中出现的频率进行比较。
这个想法是，诸如“if”、“and”、“the”之类的单词在所有文档中都很常见，但某些单词在本文档中比语料库中的典型单词更常见。
这一定是非常标准的。它叫什么？以显而易见的方式做这件事时，我总是对文档中的新词有问题，但在语料库评级中却没有遇到无限重要的问题。此事如何处理？
I want to analyse a document for items such as letters, bigrams, words, etc and compare how frequent they are in my document to how frequent they were over a large corpus of documents.
The idea is that words such as "if", "and", "the" are common in all documents but some words will be much more common in this document than is typical for the corpus.
This must be pretty standard. What is it called? Doing it the obvious way I always had a problem with novel words in my document but not in the corpus rating infinitely significant. How is this dealt with?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
您很可能已经检查过 tf-idf
或 okapi_bm25 系列的其他一些指标。
您也可以检查 自然语言处理工具包 nltk 以获得一些现成的解决方案
更新：
对于新词，应该进行平滑处理：Good-Turing、Laplace ， ETC。
most likely you've already checked the tf-idf
or some other metrics from okapi_bm25 family.
also you can check natural language processing toolkit nltk for some ready solutions
UPDATE:
as for novel words, smoothing should be applied: Good-Turing, Laplace, etc.
它属于线性分类器，其中朴素贝叶斯分类器是最著名的形式（因为它在解决现实世界的分类问题时非常简单和鲁棒）。
It comes under the heading of linear classifiers, with Naive Bayesian classifiers being the most well-known form (due to its remarkably simplicity and robustness in attacking real-world classification problems).