[Critical Bug] The Token Counter is bugged about chess-challenge HOT 5 OPEN

Steps to reproduce:
Adding this line in the move loop of the default EvilBot incorrectly increases the token count from 142 to 151.
Adding it to the MoveIsCheckmate function instead correctly increases the token count to 153.

from chess-challenge.

This appears to have something to do with 'syntax trivia'.

If you add this code to line 57 inside the foreach loop:

var a = token.ToString().Replace("\n", "\\n");
var b = token.TrailingTrivia.ToString().Replace("\n", "\\n");
Console.WriteLine($"'{a}' has trailing trivia '{b}'");

and then run it on this code:

using ChessChallenge.API;

public class MyBot : IChessBot
{
    public Move Think(Board board, Timer timer)
    {
        bool tokenTest = Move.NullMove is { TargetSquare.Rank: 1 };
        return default;
    }
}

you can observe that somehow, it causes the line after the pattern match to count as "syntax trivia":

'}' has trailing trivia ''
';' has trailing trivia '\n        return default;\n'
'}' has trailing trivia '\n}'
'' has trailing trivia ''

which https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/work-with-syntax#syntax-trivia defines as

Syntax trivia represent the parts of the source text that are largely insignificant for normal understanding of the code, such as white space, comments, and preprocessor directives.

In this case, the return statement is very much significant to understanding the code, and it isn't whitespace, a comment, or a preprocessor directive, so I'm inclined to believe this is a bug in the parser. This should be somewhat mitigated by adding descendIntoTrivia: true to the DescendantTokens call, which should make it visit trivia tokens.

from chess-challenge.

I think it is a bug in the parser too.
The returned tokens also differ in other places that should not have changed.

See the images below; Left original source, mid in the move loop, right in the MoveIsCheckmate

You can see tokens are missing in mid compared to the original.

Here you can see tokens are inserted and missing in right compared to the original.

from chess-challenge.

Thanks for reporting, that's really strange! I will investigate (and will look into descendintotrivia as a potential solution @Algorhythm-sxv)

from chess-challenge.

In case anyone is still wondering, it turns out the issue is because I accidentally used an outdated version of Roslyn. Updating to 4.7 fixes the problem. Sorry about the blunder!

from chess-challenge.