Make Token inherit from Generic. about lark HOT 2 OPEN

The reason the signature is the way it is is because the line attributes might be None. If you want to write typesafe code, you will need to check for this. If you are checking for this and mypy can't figure out that you are doing the check, that would be a problem on mypy's side, not something we can fix.

from lark.

I thought something like that, this particularly can happen if you inject tokens after the lexer stage. But if you are not injecting tokens, then you end with every function on a transformer following this pattern

def some_rule(...,token:Token,...)->SomeType:
line = token.line #type:ignore

Are there other cases apart from token injection that can make the line a None while column or position isn't?

Otherwise the transformers code can be wrote like:

def some_rule(...,token:Token[int],...)->SomeType:
line = token.line

When you know you are not injecting tokens, and if you are, you can explicitly state that you are generating tokens without information by using Token[None]

from lark.