✨ New Tokens\TokenHelper class

As of PHPUnit 9.3, PHPUnit started using PHP-Parser as a basis to calculate code coverage.

The problem with this is that PHP-Parser backfills new PHP native tokens, while the code in this repo contains logic which checks for the existence of tokens using `defined()`. That logic breaks with PHP-Parser in play.

An external PHPCS standard may also be using PHP-Parser, but as Parser doesn't fix the tokenization of files, the tokens polyfilled by PHP-Parser would still cause interference.

To work round this, I'm introducing a `TokenHelper::tokenExists()` method which can distinguish tokens defined by PHP and PHPCS from tokens defined by PHP-Parser.

This should solve the issue in most cases, though in edge-cases, PHP-Parser could still prevent the PHPCS native `Tokenizer\PHP` class from providing polyfills when it should.

For external standards which want to use PHPUnit 9.3+ for calculating code coverage, warming the cache is also still recommended, though it shouldn't necessarily be _needed_ anymore if the standard implements the use of the `TokenHelper::tokenExists()` method in all the appropriate places.

Includes unit tests.

Refs:
* sebastianbergmann/php-code-coverage#798
* https://github.com/nikic/PHP-Parser/blob/master/lib/PhpParser/Lexer.php

Note: while `Helper` would have been the more intuitive name for the class, as there is also a `BackCompat\Helper` class, this would mean use aliases would need to be used a lot, so instead, I've elected to name the class `TokenHelper`.

Author: jrfnl

PHPCSUtils/Tokens/TokenHelper.php

@@ -0,0 +1,55 @@
+<?php
+/**
+ * PHPCSUtils, utility functions and classes for PHP_CodeSniffer sniff developers.
+ *
+ * @package   PHPCSUtils
+ * @copyright 2019-2020 PHPCSUtils Contributors
+ * @license   https://opensource.org/licenses/LGPL-3.0 LGPL3
+ * @link      https://github.com/PHPCSStandards/PHPCSUtils
+ */
+
+namespace PHPCSUtils\Tokens;
+
+/**
+ * Helpers for working with tokens.
+ *
+ * @since 1.0.0
+ */
+class TokenHelper
+{
+
+    /**
+     * Check whether a PHP native token exists (for real).
+     *
+     * Under normal circumstances, checking whether a token exists (either defined by PHP or by PHPCS)
+     * is as straight-forward as running `defined('T_TOKEN_NAME')`.
+     *
+     * Unfortunately, this doesn't work in all circumstances, most notably, if an external standard
+     * also uses PHP-Parser or when code coverage is run on a standard using PHPUnit >= 9.3 (which uses PHP-Parser),
+     * this logic breaks because PHP-Parser also polyfills tokens.
+     * This method takes potentially polyfilled tokens from PHP-Parser into account and will regard the token
+     * as undefined if it was declared by PHP-Parser.
+     *
+     * Note: this method only _needs_ to be used for PHP native tokens, not for PHPCS specific tokens.
+     * Also, realistically, it only needs to be used for tokens introduced in PHP in recent versions (PHP 7.4 and up).
+     * Having said that, the method _will_ also work correctly when a name of a PHPCS native token is passed or
+     * of an older PHP native token.
+     *
+     * {@internal PHP native tokens have a positive integer value. PHPCS polyfilled tokens are strings.
+     * PHP-Parser polyfilled tokens will always have a negative integer value < 0, which is how
+     * these are filtered out.}
+     *
+     * @link https://github.com/sebastianbergmann/php-code-coverage/issues/798
+     * @link https://github.com/nikic/PHP-Parser/blob/master/lib/PhpParser/Lexer.php
+     *
+     * @since 1.0.0-alpha4
+     *
+     * @param string $name The token name.
+     *
+     * @return bool
+     */
+    public static function tokenExists($name)
+    {
+        return (\defined($name) && (\is_int(\constant($name)) === false || \constant($name) > 0));
+    }
+}

Tests/Tokens/TokenHelper/TokenExistsTest.php

@@ -0,0 +1,97 @@
+<?php
+/**
+ * PHPCSUtils, utility functions and classes for PHP_CodeSniffer sniff developers.
+ *
+ * @package   PHPCSUtils
+ * @copyright 2019-2020 PHPCSUtils Contributors
+ * @license   https://opensource.org/licenses/LGPL-3.0 LGPL3
+ * @link      https://github.com/PHPCSStandards/PHPCSUtils
+ */
+
+namespace PHPCSUtils\Tests\Tokens\TokenHelper;
+
+use PHPCSUtils\BackCompat\Helper;
+use PHPCSUtils\Tokens\TokenHelper;
+use PHPUnit\Framework\TestCase;
+
+/**
+ * Test class.
+ *
+ * @covers \PHPCSUtils\Tokens\TokenHelper::tokenExists
+ *
+ * @since 1.0.0
+ */
+class TokenExistsTest extends TestCase
+{
+
+    /**
+     * Add select constants to allow testing this method.
+     *
+     * @beforeClass
+     *
+     * @return void
+     */
+    public static function setUpConstants()
+    {
+        \define('T_FAKETOKEN', -5);
+    }
+
+    /**
+     * Test the method.
+     *
+     * @dataProvider dataTokenExists
+     *
+     * @param string $name     Token name.
+     * @param bool   $expected Expected function return value.
+     *
+     * @return void
+     */
+    public function testTokenExists($name, $expected)
+    {
+        $this->assertSame($expected, TokenHelper::tokenExists($name));
+    }
+
+    /**
+     * Data provider.
+     *
+     * @return array
+     */
+    public function dataTokenExists()
+    {
+        $phpcsVersion = Helper::getVersion();
+
+        return [
+            'Token which doesn\'t exist either way' => [
+                'name'     => 'T_DOESNOTEXIST',
+                'expected' => false,
+            ],
+            'PHP native token which always exists' => [
+                'name'     => 'T_FUNCTION',
+                'expected' => true,
+            ],
+            'PHPCS native token which always exists (in the PHPCS versions supported)' => [
+                'name'     => 'T_CLOSURE',
+                'expected' => true,
+            ],
+            'PHP native token which may not exist - PHP 7.4 T_FN' => [
+                'name'     => 'T_FN',
+                'expected' => (\version_compare($phpcsVersion, '3.5.3', '>=')
+                                || \version_compare(\PHP_VERSION_ID, '70399', '>=')),
+            ],
+            'PHP native token which may not exist - PHP 8.0 T_NULLSAFE_OBJECT_OPERATOR' => [
+                'name'     => 'T_NULLSAFE_OBJECT_OPERATOR',
+                'expected' => (\version_compare($phpcsVersion, '3.5.7', '>=')
+                                || \version_compare(\PHP_VERSION_ID, '79999', '>=')),
+            ],
+            'PHP native token which may not exist - PHP 8.1 T_ENUM' => [
+                'name'     => 'T_ENUM',
+                'expected' => (\version_compare($phpcsVersion, '3.7.0', '>=')
+                                || \version_compare(\PHP_VERSION_ID, '80099', '>=')),
+            ],
+            'Mocked polyfilled token' => [
+                'name'     => 'T_FAKETOKEN',
+                'expected' => false,
+            ],
+        ];
+    }
+}