Test result is different between local and CI

[saschanaz]

My local result in #33893 is:

> npm test      

> [email protected] pretest C:\Users\sasch\Documents\GitHub\typescript-f
> gulp tests

[12:55:04] Using gulpfile ~\Documents\GitHub\typescript-f\gulpfile.js
[12:55:04] Starting 'tests'...
[12:55:04] Starting 'generateLibs'...
[12:55:04] Starting 'buildScripts'...
[12:55:04] Finished 'generateLibs' after 234 ms
[12:55:05] Finished 'buildScripts' after 334 ms
[12:55:05] Starting 'generateDiagnostics'...
[12:55:05] Finished 'generateDiagnostics' after 1.08 ms
[12:55:05] Starting 'buildShims'...
[12:55:05] Starting 'buildDebugTools'...
[12:55:05] Finished 'buildShims' after 300 ms
[12:55:05] Finished 'buildDebugTools' after 301 ms
[12:55:05] Starting 'buildTests'...
[12:55:05] Starting 'buildServerLibraryOut'...
[12:55:05] Finished 'buildTests' after 377 ms
[12:55:05] Finished 'buildServerLibraryOut' after 377 ms
[12:55:05] Starting 'createServerLibraryJs'...
[12:55:05] Finished 'createServerLibraryJs' after 11 ms
[12:55:05] Starting 'createServerLibraryDts'...
[12:55:05] Finished 'createServerLibraryDts' after 5.3 ms
[12:55:05] Finished 'tests' after 1.11 s

> [email protected] test C:\Users\sasch\Documents\GitHub\typescript-f
> gulp runtests-parallel --light=false

[12:55:07] Using gulpfile ~\Documents\GitHub\typescript-f\gulpfile.js
[12:55:07] Starting 'runtests-parallel'...
[12:55:07] Starting 'generateLibs'...
[12:55:07] Starting 'buildScripts'...
[12:55:07] Finished 'generateLibs' after 281 ms
[12:55:07] Finished 'buildScripts' after 376 ms
[12:55:07] Starting 'generateDiagnostics'...
[12:55:07] Finished 'generateDiagnostics' after 891 μs
[12:55:07] Starting 'buildShims'...
[12:55:07] Starting 'buildDebugTools'...
[12:55:07] Finished 'buildShims' after 355 ms
[12:55:07] Finished 'buildDebugTools' after 349 ms
[12:55:07] Starting 'buildTests'...
[12:55:07] Starting 'buildTypescriptServicesOut'...
[12:55:07] Starting 'buildServerLibraryOut'...
[12:55:08] Finished 'buildTests' after 424 ms
[12:55:08] Finished 'buildTypescriptServicesOut' after 425 ms
[12:55:08] Starting 'createTypescriptServicesJs'...
[12:55:08] Finished 'buildServerLibraryOut' after 425 ms
[12:55:08] Starting 'createServerLibraryJs'...
[12:55:08] Finished 'createServerLibraryJs' after 13 ms
[12:55:08] Starting 'createServerLibraryDts'...
[12:55:08] Finished 'createServerLibraryDts' after 5.88 ms
[12:55:08] Finished 'createTypescriptServicesJs' after 38 ms
[12:55:08] Starting 'createTypescriptServicesDts'...
[12:55:08] Finished 'createTypescriptServicesDts' after 4.31 ms
[12:55:08] Starting 'createTypescriptJs'...
[12:55:08] Finished 'createTypescriptJs' after 24 ms
[12:55:08] Starting 'createTypescriptDts'...
[12:55:08] Finished 'createTypescriptDts' after 6.12 ms
[12:55:08] Starting 'createTypescriptStandaloneDts'...
[12:55:08] Finished 'createTypescriptStandaloneDts' after 4.06 ms
[12:55:08] Starting 'runTestsParallel'...
[12:55:08] Running tests with config: {"light":false,"workerCount":12,"taskConfigsFolder":"C:\\Users\\sasch\\AppData\\Local\\Temp/ts-tests6","noColor":false,"timeout":40000,"keepFailed":false}[12:55:08] > node built/local/run.js
Discovered 150 unittest suites and 41 new suites.
Discovering runner-based tests...
Skipped 815 expensive tests; estimated time savings of 33.21% with --skipPercent=5.00 chance of missing a test.
Discovered 13098 test files in 4594ms.
Starting to run tests using 12 threads...
Batching initial test lists...
Batched into 12 groups. Unprofiled tests including tsrunner-fourslash-server://tests/cases/fourslash/server/typeReferenceOnServer.ts will be run first.

  [▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬] √ 61798 passing (2m)


  61798 passing (2m)

[12:57:23] Finished 'runTestsParallel' after 2.25 min
[12:57:23] Starting 'postTest'...
[12:57:23] Starting 'buildEslintRules'...
[12:57:24] Finished 'buildEslintRules' after 404 ms
[12:57:24] Starting 'lintFoldStart'...
[12:57:24] Finished 'lintFoldStart' after 838 μs
[12:57:24] Starting 'lint-scripts'...
[12:57:24] Linting: node_modules/eslint/bin/eslint --cache --cache-location scripts/.eslintcache --format autolinkable-stylish --rulesdir scripts/eslint/built/rules --ext .ts scripts
[12:57:24] > C:\Program Files\nodejs\node.exe node_modules/eslint/bin/eslint --cache --cache-location scripts/.eslintcache --format autolinkable-stylish --rulesdir scripts/eslint/built/rules --ext .ts scripts
[12:57:25] Finished 'lint-scripts' after 1.52 s
[12:57:25] Starting 'lint-compiler'...
[12:57:25] Linting: node_modules/eslint/bin/eslint --cache --cache-location src/.eslintcache --format autolinkable-stylish --rulesdir scripts/eslint/built/rules --ext .ts src
[12:57:25] > C:\Program Files\nodejs\node.exe node_modules/eslint/bin/eslint --cache --cache-location src/.eslintcache --format autolinkable-stylish --rulesdir scripts/eslint/built/rules --ext .ts src
C:\Users\sasch\Documents\GitHub\typescript-f\src\compiler\checker.ts
ERROR: C:\Users\sasch\Documents\GitHub\typescript-f\src\compiler\checker.ts:10648:59  Parsing error: Expression expected.
C:\Users\sasch\Documents\GitHub\typescript-f\src\services\codefixes\inferFromUsage.ts
ERROR: C:\Users\sasch\Documents\GitHub\typescript-f\src\services\codefixes\inferFromUsage.ts:316:114  Parsing error: Expression expected.
[12:57:26] 'lint-compiler' errored after 1.11 s
[12:57:26] Error: Process exited with code: 1
    at ChildProcess.<anonymous> (C:\Users\sasch\Documents\GitHub\typescript-f\scripts\build\utils.js:54:28)
    at ChildProcess.emit (events.js:210:5)
    at ChildProcess.EventEmitter.emit (domain.js:499:23)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
[12:57:26] 'postTest' errored after 3.04 s
[12:57:26] 'runtests-parallel' errored after 2.32 min
npm ERR! Test failed.  See above for more details.

Seems that it weirdly just locally passes the tests which fail on CI, and then fails locally to parse the new optional chaining syntax. npm i && npx gulp clean doesn't fix this, any idea why this is happening?

[ajafff]

I had to git clean and npm install to make my local checkout work again.

[ajafff]

Re: different test results
That might be caused by the --skipPercent option. Locally its default is 5 to speed up tests. In CI it's set to 0 to ensure everything works as expected

[ahejlsberg]

I've gotten bitten by the --skipPercent option too. Just yesterday @sandersn and I were discussing removing it or at least having it default to --skipPercent=0.

[RyanCavanaugh]

We've removed the "only run some tests" feature