Hi @Mike-3, thanks for the praise, always great to hear from users! 😄

You're right, the new parser does not expose the tokens (intermediate representation) of the parsed expression in any way. This is due to this implementation relying on petitparser which does not use a traditional lexer but a parsing expression grammar (PEG) instead.

I considered the lexer an internal implementation detail of the parser, so have not maintained any sort of backwards compatibility there. However I'm curious to hear what your use case is?

In this case, I used it for a verbal arithmetic solver and used the functionality to determine the available variables. But I'm sure I can do that some other way.

https://github.com/GCWizard/GCWizard/blob/aa2d8ce3fb67be61212f6c12b2ae84edf0bbac4c/lib/tools/games/verbal_arithmetic/logic/helper.dart#L76

That makes sense. I was planning to implement #73 soon which would solve that particular issue.