（人类）文档的语言
有没有一种方法（程序、库）可以大致了解文档是用哪种语言编写的？
我有一堆混合语言的文本文档（~500K）要导入到支持 i18n 的 CMS（Drupal）中。
我不需要完美的匹配，只需要一些猜测。
Is there a way (a program, a library) to approximately know which language a document is written in?
I have a bunch of text documents (~500K) in mixed languages to import in a i18n enabled CMS (Drupal)..
I don't need perfect matches, only some guess.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
似乎有一个 Perl 模块可以实现此目的：Lingua::Identify
Paul。
There seems to be a Perl module for this: Lingua::Identify
Paul.
我想说你最好的选择是寻找关键词——文章之类的东西——对于你正在寻找的语言来说是独一无二的。 例如，“Un”将以西班牙语和法语显示，但“une”可识别为法语，而“unos”可识别为西班牙语。 变音符号也很有用——你会在西班牙语和葡萄牙语中看到“ñ”，在法语和其他一些语言中看到“ç”……诸如此类的东西。
编辑 - 保罗的解决方案可能是最好的； 看起来它使用了我概述的方法，再加上一些额外的方法。
I'd say your best bet is to look for key words - articles, that kind of thing - that are unique to the languages you're looking for. "Un" will show up in both Spanish and French, for example, but "une" is identifiably French whereas "unos", for example, is identifiably Spanish. Diacritics are useful too - you'll see "ñ" in Spanish and possibly Portuguese, "ç" in French and a few others... that kind of thing.
edit - Paul's solution is probably the best; looks like it uses methods like what I outlined, plus a few extra.
通过运行 Google 搜索“确定文档的语言" 我发现许多不同的网站可以帮助您。 第一页上的第三个链接最终将我引向了Google Code API 正是您所需要的。
By running a Google search for "determine language of document" I found many different sites that will help you. The third link on the first page eventually led me to a function in the Google Code API that is exactly what you need.
考虑到您拥有需要识别的所有不同语言的语料库数据，有一种非常简单的方法可以做到这一点。 这称为 n 元模型。 我认为 Lingua::Identify 确实如此不过，这已经是你最好的选择，而不是实现你自己的。
There is a pretty easy way to do this, given that you have corpus data in all the different languages you'll need to identify. It's called n-gram modeling. I think Lingua::Identify does this already, though, so that is your best bet rather than implementing your own.
Google Translation API 很酷，并且有一个 REST 接口。 但我需要向它发送大量的大文档（是的，我可以使用摘录），即使谷歌是谷歌，我也不认为这是
公平的。
文件也不是我的，我会询问我的客户是否可以将它们发送给第三方（即使，迟早，G会得到它们；））。
我想我会走 Perl 道路......
Google Translation API is cool, and has a REST interface. But I need to send it a LOT of BIG document (yes, I could use an excerpt) and, even if Google is Google, I don't think this
fair.
Document are also not mine, and Id ask my client if it is ok to send them to a third party (even if, soon or later, G will get them ;)).
I think I'll go trough the Perl path...