Question

This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Constructors

• new Tokenizer(Object rules, String flag)

Constructs a new tokenizer based on the given rules and flags.

rules	Object	Required. The highlighting rules
flag	String	Required. Any additional regular expression flags to pass (like "i" for case insensitive)

Methods

• Tokenizer.getLineTokens(Object line, Object startState)
• 返回值： Object

Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

line	Object	Required.
startState	Object	Required.