ANTLR 可以根据以下字符区分词法分析器规则吗？

发布于 2024-10-30 23:34:10 字数 506 浏览 0 评论 0原文

为了解析测试文件，我想允许标识符以数字开头。

我的规则是：

ID  :   ('a'..'z' | 'A'..'Z' | '0'..'9' | '_') ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')*
;

但是我还需要匹配该文件中的数字。我的规则是：

INT :   '0'..'9'+
;

显然 Antlr 不会让我这样做，因为 INT 永远不会匹配。

有办法允许吗？具体来说，我想匹配一个 INTEGER 后跟一个不带空格的 ID 作为 ID，并仅在其后跟一个空格时创建一个 INT 令牌。

例如：

3BOB -> [ID with text "3BOB"]
3 BOB -> [INT with text "3"] [ID with text "BOB"]

原文

For parsing a test file I'd like to allow identifier's to begin with a number.

my rule is:

ID  :   ('a'..'z' | 'A'..'Z' | '0'..'9' | '_') ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')*
;

However I also need to match numbers in this file as well. My rule for that is:

INT :   '0'..'9'+
;

Obviously Antlr won't let me do this as INT will never be matched.

Is there a way to allow this? Specifically I'd like to match an INTEGER followed by an ID with no spaces as just an ID and create an INT token only if it's followed by a space.

For example:

3BOB -> [ID with text "3BOB"]
3 BOB -> [INT with text "3"] [ID with text "BOB"]

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

合久必婚 2024-11-06 23:34:10

只需更改 ID 和 INT 令牌的定义顺序即可。

grammar qqq;

// Parser's rules.

root:
    (integer|identifier)+
;

integer:
    INT {System.out.println("INT with text '"+$INT.text+"'.");}
;

identifier:
    ID {System.out.println("ID with text '"+$ID.text+"'.");}
;

// Lexer's tokens.

INT:    '0'..'9'+
;

ID:    ('a'..'z' | 'A'..'Z' | '0'..'9' | '_')
       ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')*
;

WS:    ' ' {skip();}
;

UNPREDICTED_TOKEN
:
    ~(' ') {System.out.println("Unpredicted token.");}
;

语法中定义标记的顺序很重要：如果一个字符串可以归属于多个标记，则该字符串将归属于第一个定义的标记。在您的情况下，如果您希望整数“123”在仍然符合 ID 的情况下归属于 INT，请将 INT 定义放在第一位。

Antlr 的令牌匹配是贪婪的，因此它不会停止在“123BOB”中的“123”上，而是会继续，直到没有一个令牌与字符串匹配，并获取最后一个匹配的令牌。所以你的标识符现在可以以数字开头。

关于代币顺序的评论也可以在 Mark Volkmann 的这篇文章中找到。

Just change the order in which ID and INT tokens are defined.

grammar qqq;

// Parser's rules.

root:
    (integer|identifier)+
;

integer:
    INT {System.out.println("INT with text '"+$INT.text+"'.");}
;

identifier:
    ID {System.out.println("ID with text '"+$ID.text+"'.");}
;

// Lexer's tokens.

INT:    '0'..'9'+
;

ID:    ('a'..'z' | 'A'..'Z' | '0'..'9' | '_')
       ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')*
;

WS:    ' ' {skip();}
;

UNPREDICTED_TOKEN
:
    ~(' ') {System.out.println("Unpredicted token.");}
;

The order in which tokens are defined in grammar is significant: in case a string can be attributed to multiple tokens it is attributed to that one which is defined first. In your case if you want integer '123' to be attributed to INT when it still conforms to ID -- put INT definition first.

Antlr's token matching is greedy so it won't stop on '123' in '123BOB', but will continue until non of the tokens match the string and take the last token matched. So your identifiers now can start with numbers.

A remark on tokens order can also be found in this article by Mark Volkmann.

回复收藏 0 原文

泅人 2024-11-06 23:34:10

您的规则中的以下细微更改应该可以解决问题：

ID  :   ('0'..'9')* // optional numbers
        ('a'..'z' | 'A'..'Z' | '_' |  '&' | '/' | '-' | '.') // followed by mandatory character which is not a number
        ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')* // followed by more stuff (including numbers)
;

INT :   '0'..'9'+ // a number
;

您只需让标识符以可选数字开头，并将以下字符设为强制字符即可。

The following minor changes in your rules should do the trick:

ID  :   ('0'..'9')* // optional numbers
        ('a'..'z' | 'A'..'Z' | '_' |  '&' | '/' | '-' | '.') // followed by mandatory character which is not a number
        ('a'..'z' | 'A'..'Z' | '0'..'9' | '_' | '&' | '/' | '-' | '.')* // followed by more stuff (including numbers)
;

INT :   '0'..'9'+ // a number
;

You simply let allow your identifiers to start with an optional number and make the following characters mandatory.

回复收藏 0 原文

~没有更多了~