PHPToken polyfill cannot be compatible with PHPToken::getAll on PHP-8.0-dev

[peter279k]

When looking at the php-8.0-dev tests on GitHub Action, it seems to be failed during PHPUnit test.

The PHPtoken polyfill result is as follows when running IsCheckTest class:

['T_OPEN_TAG', true],
['T_COMMENT', true],
['T_DOC_COMMENT', true],
['T_WHITESPACE', true],
['T_FUNCTION', false],
['T_WHITESPACE', true],
['T_STRING', false],
['(', false],
[')', false],
['T_WHITESPACE', true],
['{', false],

When running PhpToken::getAll on php-8.0-dev, it will be:

['T_OPEN_TAG', true],
['T_COMMENT', true],
['T_WHITESPACE', true], // This is different stuff from PHPToken polyfill
['T_DOC_COMMENT', true],
['T_WHITESPACE', true],
['T_FUNCTION', false],
['T_WHITESPACE', true],
['T_STRING', false],
['(', false],
[')', false],
['T_WHITESPACE', true],
['{', false],

[peter279k]

I suspect the token_get_all behavior is not same as PhpToken::getAll method.

[peter279k]

@Ayesh, do you have any idea about this?