Add Scanner.

[dabrahams]

This is intended to be a replacement lexer; I hope it's of some use. I'm not attached to how or whether it is integrated. Of course I didn't make any of the examples use it, nor did I include tests, which should get done someday. I suggest that if backward compatibility is important to you that you simply reimplement CitronLexer in terms of Scanner.

Thanks for this project; it has basically been a joy to work with :-)

[roop]

Thanks a lot. I'll take a closer look this weekend and figure out how to integrate it (maybe make it a separate module alongside the current lexer).

Thanks for this project; it has basically been a joy to work with :-)

Thanks. This made my day. :) 😊

[dabrahams]

A few features I forgot to describe:

1. This code labels each token with its location in the source.
2. This lexer obeys the "max munch" rule, which says that the longest text matching any pattern is the next token. That's how lexers traditionally work, so you don't have to worry about the order in which patterns are specified.
3. It uses one single regex for all tokens that are described literally rather than as patterns (e.g. keywords)

It's not the most efficient possible implementation, but it should be more efficient than what's currently in the project.

[dabrahams]

You can see an example of this feature in use here: https://github.com/dabrahams/ebnf-citron/blob/main/Sources/ebnf-citron/EBNFLexer.swift and here: https://github.com/dabrahams/ebnf-citron/blob/main/Sources/ebnf-citron/main.swift#L16-L23

[dabrahams]

This PR should be considered a draft until I report back.

[dabrahams]

OK, reporting back; it works, FWIW.

[dabrahams]

@roop Bump. I use this all the time.

[roop]

Thanks, @dabrahams. I hope to get around to integrating this in the next few days.

[dabrahams]

I'm still on my own fork because this hasn't been merged.