什么是“代理对”?在Java中?
我正在阅读 StringBuffer
的文档,特别是 reverse() 方法。该文档提到了一些关于代理对的内容。在这种情况下,代理对是什么?什么是低和高代理?
I was reading the documentation for StringBuffer
, in particular the reverse() method. That documentation mentions something about surrogate pairs. What is a surrogate pair in this context? And what are low and high surrogates?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(7)
术语“代理对”是指在 UTF-16 编码方案中对具有高代码点的 Unicode 字符进行编码的方法。
在 Unicode 字符编码中,字符映射到 0x0 到 0x10FFFF 之间的值。
在内部,Java 使用 UTF-16 编码方案来存储 Unicode 文本的字符串。在 UTF-16 中,使用 16 位(两字节)代码单元。由于 16 位只能包含从 0x0 到 0xFFFF 的字符范围,因此需要使用一些额外的复杂性来存储高于此范围(0x10000 到 0x10FFFF)的值。这是通过使用称为代理的代码单元对来完成的。
代理代码单元处于称为“高代理”和“低代理”的两个范围内,具体取决于它们是否被允许出现在两个代码单元序列的开头或结尾。
The term "surrogate pair" refers to a means of encoding Unicode characters with high code-points in the UTF-16 encoding scheme.
In the Unicode character encoding, characters are mapped to values between 0x0 and 0x10FFFF.
Internally, Java uses the UTF-16 encoding scheme to store strings of Unicode text. In UTF-16, 16-bit (two-byte) code units are used. Since 16 bits can only contain the range of characters from 0x0 to 0xFFFF, some additional complexity is used to store values above this range (0x10000 to 0x10FFFF). This is done using pairs of code units known as surrogates.
The surrogate code units are in two ranges known as "high surrogates" and "low surrogates", depending on whether they are allowed at the start or end of the two-code-unit sequence.
早期的 Java 版本使用 16 位 char 数据类型表示 Unicode 字符。这种设计在当时是有意义的,因为所有 Unicode 字符的值都小于 65,535 (0xFFFF),并且可以用 16 位表示。但后来,Unicode 将最大值增加到 1,114,111 (0x10FFFF)。由于 16 位值太小,无法表示 Unicode 版本 3.1 中的所有 Unicode 字符,因此 UTF-32 编码方案采用了 32 位值(称为代码点)。
但为了有效使用内存,16 位值优于 32 位值,因此 Unicode 引入了新设计以允许继续使用 16 位值。该设计采用UTF-16编码方案,将1,024个值分配给16位高代理项(在U+D800到U+DBFF范围内),并将另外1,024个值分配给16位低代理项(在U+DC00范围内)到 U+DFFF)。它使用高代理项后跟低代理项(代理对)来表示(1,024 和 1,024 的乘积)1,048,576 (0x100000) 65,536 (0x10000) 和 1,114,111 (0x10FFFF) 之间的值。
Early Java versions represented Unicode characters using the 16-bit char data type. This design made sense at the time, because all Unicode characters had values less than 65,535 (0xFFFF) and could be represented in 16 bits. Later, however, Unicode increased the maximum value to 1,114,111 (0x10FFFF). Because 16-bit values were too small to represent all of the Unicode characters in Unicode version 3.1, 32-bit values — called code points — were adopted for the UTF-32 encoding scheme.
But 16-bit values are preferred over 32-bit values for efficient memory use, so Unicode introduced a new design to allow for the continued use of 16-bit values. This design, adopted in the UTF-16 encoding scheme, assigns 1,024 values to 16-bit high surrogates(in the range U+D800 to U+DBFF) and another 1,024 values to 16-bit low surrogates(in the range U+DC00 to U+DFFF). It uses a high surrogate followed by a low surrogate — a surrogate pair — to represent (the product of 1,024 and 1,024)1,048,576 (0x100000) values between 65,536 (0x10000) and 1,114,111 (0x10FFFF) .
在这篇文章的上述答案中添加更多信息。
在 Java-12 中测试,应该适用于 5 以上的所有 Java 版本。
如下所述:https://stackoverflow.com/a/47505451 /2987755,
无论哪个字符(其 Unicode 高于 U+FFFF)都表示为代理对,Java 将其存储为一对 char 值,即单个 Unicode 字符表示为两个相邻的 Java 字符。
正如我们在下面的示例中看到的。
长度:
Adding some more info to the above answers from this post.
Tested in Java-12, should work in all Java versions above 5.
As mentioned here: https://stackoverflow.com/a/47505451/2987755,
whichever character (whose Unicode is above U+FFFF) is represented as a surrogate pair, which Java stores as a pair of char values, i.e. the single Unicode character is represented as two adjacent Java characters.
As we can see in the following example.
Length:
Equality:
Represent "????" to String using Unicode
\ud83c\udf09
as below and check equality.Java does not support UTF-32
You can convert Unicode character to Java String
String.substring() does not consider supplementary characters
To solve this we can use
String.offsetByCodePoints(int index, int codePointOffset)
Iterating Unicode string with BreakIterator
Sorting Strings with Unicode java.text.Collator
Character's
toUpperCase()
,toLowerCase()
, methods should not be used, instead, use String uppercase and lowercase of particular locale.Character.isLetter(char ch)
does not support, better usedCharacter.isLetter(int codePoint)
, for eachmethodName(char ch)
method in the Character class there will be type ofmethodName(int codePoint)
which can handle supplementary characters.Specify charset in
String.getBytes()
, converting from Bytes to String,InputStreamReader
,OutputStreamWriter
New Methods are added in Java-21,
java.lang.Character.isEmoji
and new Regex related patterns, emoji data from here, This new functions can be helpful if you are using any library as mentioned hereRef:
https://coolsymbol.com/emojis/emoji-for-copy-and-paste.html#objects
https://www.online-toolz.com/tools/text-unicode-entities-convertor.php
https://www.ibm.com/developerworks/library/j-unicode/index.html
https://www.oracle.com/technetwork/articles/javaee/supplementary-142654.html
More info on example image1 image2
Other terms worth to explore: Normalization, BiDi
该文档所说的是,无效的 UTF-16 字符串在调用
reverse
方法后可能会变得有效,因为它们可能是有效字符串的反转。代理对(此处讨论)是 UTF-16 格式的一对 16 位值编码单个 Unicode 代码点;低代理和高代理是该编码的两半。What that documentation is saying is that invalid UTF-16 strings may become valid after calling the
reverse
method since they might be the reverses of valid strings. A surrogate pair (discussed here) is a pair of 16-bit values in UTF-16 that encode a single Unicode code point; the low and high surrogates are the two halves of that encoding.小前言
Unicode 表示代码点。每个代码点都可以根据 Unicode 标准编码为 8 位、16 位或 32 位块。
在版本 3.1 之前,主要使用的是 8 位编码(称为 UTF-8)和 16 位编码(称为 UCS-2 或“以 2 个八位位组编码的通用字符集”)。 UTF-8 将 Unicode 点编码为 1 字节块的序列,而 UCS-2 始终占用 2 字节:
A = 41 - UTF-8 的一个 8 位块
A = 0041 - 采用 UCS-2 的 16 位块
Ω = CE A9 - 两个 8 位 UTF-8 块
Ω = 03A9 - 采用 UCS-2 的 16 位块
问题
该联盟认为 16 位足以覆盖任何人类可读的语言,这给出了 >2^16 = 65536 个可能的代码值。对于 Plane 0(也称为 BMP 或基本多语言平面)来说,情况就是如此,它目前包含 65536 个代码点中的 55,445 个。 BMP 几乎涵盖了世界上所有人类语言,包括中日韩符号 (CJK)。
随着时间的流逝,新的亚洲字符集被添加,仅中文符号就花费了七万多点。现在,甚至有 表情符号点 作为标准的一部分
Small preface
Unicode represents code points. Each code point can be encoded in 8-, 16,- or 32-bit blocks according to the Unicode standard.
Prior to the Version 3.1, mostly in use was 8-bit enconding, known as UTF-8, and 16-bit encoding, known as UCS-2 or “Universal Character Set coded in 2 octets”. UTF-8 encodes Unicode points as a sequence of 1-byte blocks, while UCS-2 always takes 2 bytes:
A = 41 - one block of 8-bits with UTF-8
A = 0041 - one block of 16-bits with UCS-2
Ω = CE A9 - two blocks of 8-bits with UTF-8
Ω = 03A9 - one block of 16-bits with UCS-2
Problem
The consortium thought that 16 bits would be enough to cover any human-readable language, which gives 2^16 = 65536 possible code values. This was true for the Plane 0, also known as BMP or Basic Multilingual Plane, that includes 55,445 of 65536 code points today. BMP covers almost every human language in the world, including Chinese-Japanese-Korean symbols (CJK).
The time passed and new Asian character sets were added, Chinese symbols took more than 70,000 points alone. Now, there are even Emoji points as part of the standard ????. New 16 "additional" Planes were added. The UCS-2 room was not enough to cover anything bigger than Plane-0.
Unicode decision
With those changes, BMP is covered with 1 block of 16 bits in UTF-16, while all "Supplementary characters" are covered with Surrogate Pairs presenting 2 blocks by 16 bits each, totally 1024x1024 = 1 048 576 points.
A high surrogate precedes a low surrogate. Any deviation from this rule is considered as a bad encoding. For example, a surrogate without a pair is incorrect, a low surrogate standing before a high surrogate is incorrect.
????, 'MUSICAL SYMBOL G CLEF', is encoded in UTF-16 as a pair of surrogates 0xD834 0xDD1E (2 by 2 bytes),
in UTF-8 as 0xF0 0x9D 0x84 0x9E (4 by 1 byte),
in UTF-32 as 0x0001D11E (1 by 4 bytes).
Current situation
To support legacy applications with incorrect UTF-8/UTF-16 encodings, a new standard WTF-8, Wobbly Transformation Format, was created. It supports arbitrary surrogate points, such as a non-paired surrogate or an incorrect sequence. Today, some products do not comply with the standard and treat UTF-8 as WTF-8.
Many historic details were suppressed to follow the topic ⚖.
The latest Unicode Standard can be found at http://www.unicode.org/versions/latest
代理对是指 UTF-16 对某些字符进行编码的方式,请参阅 http://en.wikipedia.org/wiki/UTF-16/UCS-2#Code_points_U.2B10000..U.2B10FFFF
Surrogate pairs refer to UTF-16's way of encoding certain characters, see http://en.wikipedia.org/wiki/UTF-16/UCS-2#Code_points_U.2B10000..U.2B10FFFF
代理对是 UTF-16 中组成一个“代码点”的两个“代码单元”。 Java 文档指出,在相反之后,这些“代码点”仍然有效,其“代码单元”顺序正确。它还指出两个不成对的代理代码单元可以颠倒并形成有效的代理对。这意味着如果存在未配对的代码单元,那么逆向的逆向可能会不一样!
但请注意,文档中没有提及任何有关字形的内容——字形是多个代码点的组合。这意味着 e 和随之而来的重音仍然可以交换,从而将重音放在 e 之前。这意味着如果 e 之前有另一个元音,它可能会得到 e 上的重音。
哎呀!
A surrogate pair is two 'code units' in UTF-16 that make up one 'code point'. The Java documentation is stating that these 'code points' will still be valid, with their 'code units' ordered correctly, after the reverse. It further states that two unpaired surrogate code units may be reversed and form a valid surrogate pair. Which means that if there are unpaired code units, then there is a chance that the reverse of the reverse may not be the same!
Notice, though, the documentation says nothing about Graphemes -- which are multiple codepoints combined. Which means e and the accent that goes along with it may still be switched, thus placing the accent before the e. Which means if there is another vowel before the e it may get the accent that was on the e.
Yikes!