row_sums vs findfreqterms用于子集termdocmatrix以包含具有给定最小频率的单词

Question

我的问题很简单。我有一个（二进制）TDM，我想减少至少两个文档中出现的行的行数：

我认为这两种方法会在二进制矩阵中产生相同的结果：

> rowTotals = row_sums(tdm)
> dtm2 <- tdm[which(rowTotals > 2),]
> dtm2
<<TermDocumentMatrix (terms: 208361, documents: 763717)>>
Non-/sparse entries: 34812736/159094025101
Sparsity           : 100%
Maximal term length: 154
Weighting          : binary (bin)

> #alternative probably faster:
> atleast2 <- findFreqTerms(tdm, lowfreq = 2)
> dtm2 <- tdm[atleast2,]
> dtm2
<<TermDocumentMatrix (terms: 340436, documents: 763717)>>
Non-/sparse entries: 35076886/259961683726
Sparsity           : 100%
Maximal term length: 308
Weighting          : binary (bin)

但是事实并非如此..您能帮助弄清楚为什么不是吗？

Answer 1

它们产生完全相同的结果。您的第二部分有一个错误。您的频率为2及以上，而在第一部分中，您的单词频率为3及以上。如果确保两个选择标准都是相同的，则会发现它们将产生相同的结果。请参见下面的代码示例。也要速度比较。

library(tm)

data("crude")
crude <- as.VCorpus(crude)
crude <- tm_map(crude, stripWhitespace)
crude <- tm_map(crude, content_transformer(tolower))
crude <- tm_map(crude, removePunctuation)
crude <- tm_map(crude, removeNumbers)
crude <- tm_map(crude, removeWords, stopwords("english"))

tdm <- TermDocumentMatrix(crude)

# via row_totals
row_totals <- slam::row_sums(tdm)
dtm_via_rowtotals <- tdm[which(row_totals > 2),]

<<TermDocumentMatrix (terms: 237, documents: 20)>>
Non-/sparse entries: 864/3876
Sparsity           : 82%
Maximal term length: 13
Weighting          : term frequency (tf)

# via findFreqTerms
freq_terms <- findFreqTerms(tdm, lowfreq = 3)
dtm_via_freq_terms <- tdm[freq_terms, ]

<<TermDocumentMatrix (terms: 237, documents: 20)>>
Non-/sparse entries: 864/3876
Sparsity           : 82%
Maximal term length: 13
Weighting          : term frequency (tf)

他们一样吗？

all.equal(dtm_via_rowtotals, dtm_via_freq_terms)
[1] TRUE

速度：

microbenchmark::microbenchmark(row_totals = {rowtotals <- slam::row_sums(tdm); dtm_via_rowtotals <- tdm[which(rowtotals > 2),]},
                               freq_terms = {freq_terms <- findFreqTerms(tdm, lowfreq = 3); dtm_via_freq_terms <- tdm[freq_terms, ]},
                               times = 1000L)

Unit: milliseconds
       expr    min     lq     mean median      uq     max neval
 row_totals 1.5039 1.6347 1.885161 1.7106 1.86085  9.3405  1000
 freq_terms 1.5696 1.6895 2.039345 1.7760 1.93525 99.0942  1000

通过ROW_TOTALS选择的速度稍快。但这是因为findfreqterms实际上使用row_sums获取信息，并且有一些额外的代码行检查是否通过文档术语矩阵，以及您请求的频率是否是实际数字。