如何在antlr中匹配unicode字符
我试图挑选出文本中的所有标记，并且需要匹配所有 Ascii 和 Unicode 字符，因此以下是我对它们的布局方式。
fragment CHAR : ('A'..'Z') | ('a'..'z');
fragment DIGIT : ('0'..'9');
fragment UNICODE : '\u0000'..'\u00FF';
现在，如果我将令牌规则写为：
TOKEN : (CHAR|DIGIT|UNICODE)+;
我得到“决策可以使用多种替代方案匹配输入，例如“'A'..'Z'”：1, 3 因此，该输入的替代方案 3 被禁用” " 决策可以使用多种选择来匹配输入，例如“'0'..'9'”：2、3 因此，该输入禁用了替代方案 3”
并且没有任何内容匹配： 而且如果我把它写成
TOKEN : (UNICODE)+;
没有匹配。
有没有办法做到这一点。
I am trying to pick out all tokens in a text and need to match all Ascii and Unicode characters, so here is how I have laid them out.
fragment CHAR : ('A'..'Z') | ('a'..'z');
fragment DIGIT : ('0'..'9');
fragment UNICODE : '\u0000'..'\u00FF';
Now if I write my token rule as:
TOKEN : (CHAR|DIGIT|UNICODE)+;
I get "Decision can match input such as "'A'..'Z'" using multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input"
" Decision can match input such as "'0'..'9'" using multiple alternatives: 2, 3
As a result, alternative(s) 3 were disabled for that input"
And nothing gets matched:
And also if I write it as
TOKEN : (UNICODE)+;
Nothing gets matched.
Is there a way of doing this.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果您计划使用 Unicode，需要考虑的另一件事是您应该设置
charvocabulary选项来表示您希望允许 Unicode 范围 0 到 FFFE 中的任何字符。您通常会使用默认值在示例中看到的是
为了涵盖上面提出的观点。一般来说，如果您需要 ascii 字符范围
'A'..'Z'和 unicode 范围，您将创建一个 unicode 词法分析器规则，例如：'\u0080'..'\ufffe'One other thing to consider if you are planning on using Unicode is that you should set the
charvocabularyoption to say that you want to allow any char in the Unicode range of 0 through FFFEThe default you'll usually see in the examples is
To cover the point made above. Generally if you needed both the ascii character range
'A'..'Z'and the unicode range you'd make a unicode lexer rule like:'\u0080'..'\ufffe'实际上，
TOKEN: (UNICODE)+完全没用。由于一切都是令牌字符，因此如果您尝试使用这样的规则来匹配 Java 程序，比如说，它只会匹配整个程序并将其作为一个大令牌返回给您。
如果您想将输入分成有意义的片段，您确实需要将角色分成不同的组。
了解“专业人士”是如何做到这一点可能会帮助您。这是Java 的 BNF 语法，这里是BNF 标识符，这显示了他们如何采取分组的麻烦
Practically speaking,
TOKEN: (UNICODE)+is completely useless.Since everything is a token character, if you try to use such a rule to match a Java program, say, it will simply match the whole program and return it to you as one big token.
You really do need to break your characters down into different groups if you want to split your input apart into meaningful fragments.
It might help you to take a look at how the "pros" have done it. Here is a BNF grammar for Java, and here is BNF for an identifier, which shows how they took to the trouble to group out