javascript 和带有 utf-16 代理项对的字符串操作
我正在开发一个 Twitter 应用程序,刚刚偶然进入了 utf-8(16) 的世界。似乎大多数 javascript 字符串函数都像我一样对代理对视而不见。我必须重新编码一些东西以使其具有广泛的字符识别能力。
我有这个函数可以将字符串解析为数组,同时保留代理对。然后我将重新编码几个函数来处理数组而不是字符串。
function sortSurrogates(str){
var cp = []; // array to hold code points
while(str.length){ // loop till we've done the whole string
if(/[\uD800-\uDFFF]/.test(str.substr(0,1))){ // test the first character
// High surrogate found low surrogate follows
cp.push(str.substr(0,2)); // push the two onto array
str = str.substr(2); // clip the two off the string
}else{ // else BMP code point
cp.push(str.substr(0,1)); // push one onto array
str = str.substr(1); // clip one from string
}
} // loop
return cp; // return the array
}
我的问题是,我缺少一些更简单的东西吗?我看到很多人重申 javascript 本身就处理 utf-16,但我的测试让我相信,这可能是数据格式,但函数还不知道。我错过了一些简单的事情吗?
编辑: 为了帮助说明这个问题:
var a = "0123456789"; // U+0030 - U+0039 2 bytes each
var b = "
I'm working on a twitter app and just stumbled into the world of utf-8(16). It seems the majority of javascript string functions are as blind to surrogate pairs as I was. I've got to recode some stuff to make it wide character aware.
I've got this function to parse strings into arrays while preserving the surrogate pairs. Then I'll recode several functions to deal with the arrays rather than strings.
function sortSurrogates(str){
var cp = []; // array to hold code points
while(str.length){ // loop till we've done the whole string
if(/[\uD800-\uDFFF]/.test(str.substr(0,1))){ // test the first character
// High surrogate found low surrogate follows
cp.push(str.substr(0,2)); // push the two onto array
str = str.substr(2); // clip the two off the string
}else{ // else BMP code point
cp.push(str.substr(0,1)); // push one onto array
str = str.substr(1); // clip one from string
}
} // loop
return cp; // return the array
}
My question is, is there something simpler I'm missing? I see so many people reiterating that javascript deals with utf-16 natively, yet my testing leads me to believe, that may be the data format, but the functions don't know it yet. Am I missing something simple?
EDIT:
To help illustrate the issue:
var a = "0123456789"; // U+0030 - U+0039 2 bytes each
var b = "????????????????????????????????????????"; // U+1D7D8 - U+1D7E1 4 bytes each
alert(a.length); // javascript shows 10
alert(b.length); // javascript shows 20
Twitter sees and counts both of those as being 10 characters long.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
![扫码二维码加入Web技术交流群](/public/img/jiaqun_03.jpg)
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(5)
Javascript 内部使用 UCS-2,而不是 UTF-16。因此,在 Javascript 中处理 Unicode 非常困难,我不建议尝试这样做。
至于 Twitter 的做法,您似乎是在说它是按代码点进行理智计数,而不是按代码单元进行疯狂计数。
除非别无选择,否则应该使用实际支持 Unicode 的编程语言,并且该语言具有代码点接口,而不是代码单元接口。正如您所发现的,Javascript 还不够好。
它有UCS-2 诅咒,这甚至比UTF-16 诅咒还要糟糕,后者已经够糟糕了。我在 OSCON 演讲中谈论了所有这些,
Javascript uses UCS-2 internally, which is not UTF-16. It is very difficult to handle Unicode in Javascript because of this, and I do not suggest attempting to do so.
As for what Twitter does, you seem to be saying that it is sanely counting by code point not insanely by code unit.
Unless you have no choice, you should use a programming language that actually supports Unicode, and which has a code-point interface, not a code-unit interface. Javascript isn't good enough for that as you have discovered.
It has The UCS-2 Curse, which is even worse than The UTF-16 Curse, which is already bad enough. I talk about all this in OSCON talk, ???? Unicode Support Shootout: ???? The Good, the Bad, & the (mostly) Ugly ????.
Due to its horrible Curse, you have to hand-simulate UTF-16 with UCS-2 in Javascript, which is simply nuts.
Javascript suffers from all kinds of other terrible Unicode troubles, too. It has no support for graphemes or normalization or collation, all of which you really need. And its regexes are broken, sometimes due to the Curse, sometimes just because people got it wrong. For example, Javascript is incapable of expressing regexes like
[????-????]
. Javascript doesn’t even support casefolding, so you can’t write a pattern like/ΣΤΙΓΜΑΣ/i
and have it correctly match στιγμας.You can try to use the XRegEXp plugin, but you won’t banish the Curse that way. Only changing to a language with Unicode support will do that, and ???????????????????????????????????????? just isn’t one of those.
我已经拼凑出了 Unicode 字符串处理对象的起点。它创建一个名为
UnicodeString()
的函数,该函数接受 JavaScript 字符串或表示 Unicode 代码点的整数数组,并提供length
和codePoints
属性以及toString()
和slice()
方法。添加正则表达式支持会非常复杂,但是像indexOf()
和split()
(没有正则表达式支持)这样的东西应该很容易实现。I've knocked together the starting point for a Unicode string handling object. It creates a function called
UnicodeString()
that accepts either a JavaScript string or an array of integers representing Unicode code points and provideslength
andcodePoints
properties andtoString()
andslice()
methods. Adding regular expression support would be very complicated, but things likeindexOf()
andsplit()
(without regex support) should be pretty easy to implement.以下是一些在处理 JavaScript 中的代理项对时可能会有所帮助的脚本:
ES6 Unicode 垫片ES3+ 添加了 ECMAScript 6 中的
String.fromCodePoint
和String.prototype.codePointAt
方法。ES3/5fromCharCode
和charCodeAt
方法不考虑代理对,因此会给出错误的结果。XRegExp 中与
\u{10FFFF} 的完整 21 位 Unicode 代码点匹配
允许匹配 XRegExp 正则表达式中的任何单个代码点。
Here are a couple scripts that might be helpful when dealing with surrogate pairs in JavaScript:
ES6 Unicode shims for ES3+ adds the
String.fromCodePoint
andString.prototype.codePointAt
methods from ECMAScript 6. The ES3/5fromCharCode
andcharCodeAt
methods do not account for surrogate pairs and therefore give wrong results.Full 21-bit Unicode code point matching in XRegExp with
\u{10FFFF}
allows matching any individual code point in XRegExp regexes.Javascript 字符串迭代器可以为您提供实际字符而不是代理代码点:
Javascript string iterators can give you the actual characters instead of the surrogate code points:
这符合我正在寻找的内容。它需要更好地支持不同的字符串函数。当我添加它时,我将更新这个答案。
测试结果:
This is along the lines of what I was looking for. It needs better support for the different string functions. As I add to it I will update this answer.
Test results: