library("textcat")
textcat(c(
"This is an English sentence.",
"Das ist ein deutscher Satz.",
"Esta es una frase en espa~nol."))
[1] "english" "german" "spanish"
The textcat package does this. It can detect 74 'languages' (more properly, language/encoding combinations), more with other extensions. Details and examples are in this freely available article:
Identifying the language used will typically be the first step in most
natural language processing tasks. Among the wide variety of language
identification methods discussed in the literature, the ones employing
the Cavnar and Trenkle (1994) approach to text categorization based on
character n-gram frequencies have been particularly successful. This
paper presents the R extension package textcat for n-gram based text
categorization which implements both the Cavnar and Trenkle approach
as well as a reduced n-gram approach designed to remove redundancies
of the original approach. A multi-lingual corpus obtained from the
Wikipedia pages available on a selection of topics is used to
illustrate the functionality of the package and the performance of the
provided language identification methods.
And here's one of their examples:
library("textcat")
textcat(c(
"This is an English sentence.",
"Das ist ein deutscher Satz.",
"Esta es una frase en espa~nol."))
[1] "english" "german" "spanish"
if (!require("pacman")) install.packages("pacman") # for package manangement
pacman::p_load("tidyverse")
pacman::p_load("textcat")
pacman::p_load("cld2")
pacman::p_load("cld3")
pacman::p_load("rtweet")
punk <- rtweet::search_tweets(q = "punk") %>% mutate(textcat = textcat(x = text), cld2 = cld2::detect_language(text = text, plain_text = FALSE), cld3 = cld3::detect_language(text = text)) %>% select(text, textcat, cld2, cld3)
View(punk)
# Only English tweets
punk %>% filter(cld2 == "en" & cld3 == "en")
最后，如果这个问题与推文特别相关，我也许应该补充一点：Twitter 通过 API 提供了自己的推文语言检测，而且它似乎相当准确（可以理解，对于非常短的推文来说，准确度较低）。因此，如果您运行 rtweet::search_tweets(q = "punk")，您将看到生成的 data.frame 包含一个“lang”列。如果您通过 API 获取推文，那么您可能更信任 Twitter 自己的检测系统，而不是上面建议的替代解决方案（对其他文本仍然有效）。
The cldr package in a previous answer is not any more available on CRAN and may be difficult to install. However, Google's (Chromium's) cld libraries are now available in R through other dedicated packages, cld2 and cld3.
After testing with some thousands of tweets in multiple European languages, I can say that among available options, textcat is by far the least reliable. With textcat I also get quite frequently tweets wrongly detected as "middle_frisian", "rumantsch", "sanskrit", or other unusual languages. It may be relatively good with other types of texts, but I think textcat is pretty bad for tweets.
cld2 seems to be in general still better than cld3. If you want a safe way to include only tweets in English, you can still run both cld2 and cld3 and keep only tweets that are recognised as English by both.
Here's an example based on a Twitter search which usually offers result in many different languages, but always including some tweets in English.
if (!require("pacman")) install.packages("pacman") # for package manangement
pacman::p_load("tidyverse")
pacman::p_load("textcat")
pacman::p_load("cld2")
pacman::p_load("cld3")
pacman::p_load("rtweet")
punk <- rtweet::search_tweets(q = "punk") %>% mutate(textcat = textcat(x = text), cld2 = cld2::detect_language(text = text, plain_text = FALSE), cld3 = cld3::detect_language(text = text)) %>% select(text, textcat, cld2, cld3)
View(punk)
# Only English tweets
punk %>% filter(cld2 == "en" & cld3 == "en")
Finally, I should perhaps add the obvious if this question is specifically related to tweets: Twitter provides via API its own language detection for tweets, and its seems to be pretty accurate (understandably less so with very short tweets). So if you run rtweet::search_tweets(q = "punk"), you will see that the resulting data.frame includes a "lang" column. If you get your tweets via API, then you can probably trust Twitter's own detection system more than the alternative solutions suggested above (which remain valid for other texts).
tl;dr: cld2 is the fastest by far (cld3 x22, textcat x118, handmade solution x252)
There's been a lot of discussion about accuracy here, which is understandable for tweets. But what about speed ?
Here's a benchmark of cld2, cld3 and textcat.
I threw in also some naïve function I wrote, it's counting occurences of stopwords in the text (uses tm::stopwords).
I thought for long texts I may not need a sophisticated algorithm, and testing for many languages might be detrimental. In the end my approach ends up being the slowest (most likely because the packaged approaches are looping in C.
I leave it here so I can spare time to those that would have the same idea. I expect the Englishinator solution of Tyler Rinker would be slow as well (testing for only one language, but much more words to test and similar code).
I can't comment on the accuracy of cld2 vs cld3 (@giocomai claimed cld2 was better in his answer), but I confirm that textcat seems very unreliable (metionned in several places on this page). All texts were classified correctly by all methods above except this one classified as Spanish by textcat:
"Argentine crude oil production was \ndown 10.8 pct in January 1987 to
12.32 mln barrels, from 13.81 \nmln barrels in January 1986, Yacimientos Petroliferos Fiscales \nsaid. \n January 1987 natural
gas output totalled 1.15 billion cubic \nmetrers, 3.6 pct higher than
1.11 billion cubic metres produced \nin January 1986, Yacimientos Petroliferos Fiscales added. \n Reuter"
EnglishWordComparisonList<-as.vector(source(path to the list you downloaded above))
Englishinator<-function(tweet, threshold = .06) {
TWTS <- which((EnglishWordComparisonList %in% tweet)/length(tweet) > threshold)
tweet[TWTS]
#or tweet[TWTS,] if the original tweets is a data frame
}
lapply(tweets, Englishinator)
我实际上并没有使用这个，因为我在研究中使用英语单词列表的方式有很大不同，但我认为这会起作用。
An approach in R would be to keep a text file of English words. I have several of these including one from http://www.sil.org/linguistics/wordlists/english/. After sourcing the .txt file you can use this file to match against each tweet. Something like:
You'd want to have some threshold percentage to cut off to determine if it's English (I arbitrarily chose .06).
EnglishWordComparisonList<-as.vector(source(path to the list you downloaded above))
Englishinator<-function(tweet, threshold = .06) {
TWTS <- which((EnglishWordComparisonList %in% tweet)/length(tweet) > threshold)
tweet[TWTS]
#or tweet[TWTS,] if the original tweets is a data frame
}
lapply(tweets, Englishinator)
I haven't actually used this because I use the English word list much differently in my research but I think this would work.
还有一个运行良好的 R 包，名为“franc” 。虽然它比其他的慢，但我对它的体验比 cld2 尤其是 cld3 更好。
There is also a pretty well working R package called "franc". Though, it is slower than the others, I had a better experience with it than with cld2 and especially cld3.
发布评论
评论（7）
textcat包执行此操作。它可以检测 74 种“语言”（更准确地说，是语言/编码组合），还有更多其他扩展。详细信息和示例可在这篇免费文章中找到：Hornik, K.、Mair, P.、Rauch, J.、Geiger, W.、Buchta, C., & Feinerer, I. R. Journal of Statistical Software, 52, 1 中的基于 n-Gram 的文本分类的 textcat 包 -17。
摘要如下：
这是他们的例子之一：
The
textcatpackage does this. It can detect 74 'languages' (more properly, language/encoding combinations), more with other extensions. Details and examples are in this freely available article:Hornik, K., Mair, P., Rauch, J., Geiger, W., Buchta, C., & Feinerer, I. The textcat Package for n-Gram Based Text Categorization in R. Journal of Statistical Software, 52, 1-17.
Here's the abstract:
And here's one of their examples:
之前答案中的
cldr软件包在 CRAN 上不再可用，并且可能难以安装。不过，Google（Chromium 的）cld库现在可以通过其他专用包cld2和cld3在 R 中使用。在对多种欧洲语言的数千条推文进行测试后，我可以说，在可用选项中，
textcat是迄今为止最不可靠的。使用textcat，我还经常收到被错误检测为“middle_frisian”、“rumantsch”、“sanskrit”或其他不常见语言的推文。对于其他类型的文本来说，它可能相对较好，但我认为textcat对于推文来说相当糟糕。总的来说，
cld2似乎仍然比cld3更好。如果您想要一种安全的方式仅包含英文推文，您仍然可以运行cld2和cld3并仅保留被两者识别为英文的推文。这是一个基于 Twitter 搜索的示例，该搜索通常提供多种不同语言的结果，但始终包含一些英文推文。
最后，如果这个问题与推文特别相关，我也许应该补充一点：Twitter 通过 API 提供了自己的推文语言检测，而且它似乎相当准确（可以理解，对于非常短的推文来说，准确度较低）。因此，如果您运行 rtweet::search_tweets(q = "punk")，您将看到生成的 data.frame 包含一个“lang”列。如果您通过 API 获取推文，那么您可能更信任 Twitter 自己的检测系统，而不是上面建议的替代解决方案（对其他文本仍然有效）。
The
cldrpackage in a previous answer is not any more available on CRAN and may be difficult to install. However, Google's (Chromium's)cldlibraries are now available in R through other dedicated packages,cld2andcld3.After testing with some thousands of tweets in multiple European languages, I can say that among available options,
textcatis by far the least reliable. WithtextcatI also get quite frequently tweets wrongly detected as "middle_frisian", "rumantsch", "sanskrit", or other unusual languages. It may be relatively good with other types of texts, but I thinktextcatis pretty bad for tweets.cld2seems to be in general still better thancld3. If you want a safe way to include only tweets in English, you can still run bothcld2andcld3and keep only tweets that are recognised as English by both.Here's an example based on a Twitter search which usually offers result in many different languages, but always including some tweets in English.
Finally, I should perhaps add the obvious if this question is specifically related to tweets: Twitter provides via API its own language detection for tweets, and its seems to be pretty accurate (understandably less so with very short tweets). So if you run
rtweet::search_tweets(q = "punk"), you will see that the resulting data.frame includes a "lang" column. If you get your tweets via API, then you can probably trust Twitter's own detection system more than the alternative solutions suggested above (which remain valid for other texts).尝试 http://cran.r-project.org/web/packages/cldr/< /a> 将 Google Chrome 的语言检测功能引入 R。
Try http://cran.r-project.org/web/packages/cldr/ which brings Google Chrome's language detection to R.
这里有很多关于准确性的讨论，这对于推文来说是可以理解的。但速度呢？
以下是
cld2、cld3和textcat的基准测试。我还添加了一些我编写的简单函数，它计算文本中停用词的出现次数（使用 tm::stopwords ）。
我认为对于长文本，我可能不需要复杂的算法，并且对多种语言进行测试可能是有害的。最后我的方法最终是最慢的（很可能是因为打包的方法在 C 中循环。
我把它留在这里，这样我就可以腾出时间给那些有相同想法的人。我希望 Tyler Rinker 的
Englishinator解决方案也会很慢（仅测试一种语言，但需要测试更多单词和类似的代码）。>注意事项
textcat的不准确性我无法评论
cld2与cld3的准确性（@giocomai 声称cld2< /code> 在他的回答中更好），但我确认textcat似乎非常不可靠（在本页的几个地方提到过），所有文本都通过上述所有方法正确分类，除了这个被分类为西班牙语的文本。经过文本猫：There's been a lot of discussion about accuracy here, which is understandable for tweets. But what about speed ?
Here's a benchmark of
cld2,cld3andtextcat.I threw in also some naïve function I wrote, it's counting occurences of stopwords in the text (uses
tm::stopwords).I thought for long texts I may not need a sophisticated algorithm, and testing for many languages might be detrimental. In the end my approach ends up being the slowest (most likely because the packaged approaches are looping in
C.I leave it here so I can spare time to those that would have the same idea. I expect the
Englishinatorsolution ofTyler Rinkerwould be slow as well (testing for only one language, but much more words to test and similar code).The benchmark
Note on
textcat's inaccuracyI can't comment on the accuracy of
cld2vscld3(@giocomai claimedcld2was better in his answer), but I confirm thattextcatseems very unreliable (metionned in several places on this page). All texts were classified correctly by all methods above except this one classified as Spanish bytextcat:R 中的一种方法是保留英语单词的文本文件。我有其中几个，其中一个来自 http://www.sil.org/linguistics/wordlists /英语/。获取 .txt 文件后，您可以使用该文件来匹配每条推文。比如：
你想要有一些阈值百分比来切断以确定它是否是英语（我随意选择了0.06）。
我实际上并没有使用这个，因为我在研究中使用英语单词列表的方式有很大不同，但我认为这会起作用。
An approach in R would be to keep a text file of English words. I have several of these including one from http://www.sil.org/linguistics/wordlists/english/. After sourcing the .txt file you can use this file to match against each tweet. Something like:
You'd want to have some threshold percentage to cut off to determine if it's English (I arbitrarily chose .06).
I haven't actually used this because I use the English word list much differently in my research but I think this would work.
还有一个运行良好的 R 包，名为“franc” 。虽然它比其他的慢，但我对它的体验比 cld2 尤其是 cld3 更好。
There is also a pretty well working R package called "franc". Though, it is slower than the others, I had a better experience with it than with cld2 and especially cld3.
我不确定 R，但有几个其他语言的库。您可以在此处找到其中收集的一些内容：
http://www.detectlanguage.com/
另外还有一个最近有趣的项目：
http://blog.mikemccandless.com/2011/10/language-detection-with-googles-compact.html
使用此库生成了 Twitter 语言地图：
http://www.flickr.com/photos/walkingsf/6277163176/in/photostream
如果您找不到 R 库，我建议考虑通过 Web 服务使用远程语言检测器。
I'm not sure about R, but there are several libraries for other languages. You can find some of them collected here:
http://www.detectlanguage.com/
Also one recent interesting project:
http://blog.mikemccandless.com/2011/10/language-detection-with-googles-compact.html
Using this library Twitter languages map was produced:
http://www.flickr.com/photos/walkingsf/6277163176/in/photostream
If you will not find a library for R, I suggest to consider using remote language detector through webservice.