[Asking for help] Process are blocking when throw error in scenario with Retry enabled

[Horsty80]

When i run to all test (~70) in 4 workers, at the end, i get stuck with this error message

✖ "after all" hook: codeceptjs.afterSuite in "Homepage feature" in 0ms

• CodeceptJS version: 3.6.2
• NodeJS Version: 20
• playwright: 1.43.0
• allure-codeceptjs: 2.15.1
• Configuration file:

    Playwright: {
      url: testConfig.url,
      show: false,
      locale: "en-EN",
      browser: 'chromium',
      chromium: {
        args: ['--auth-server-whitelist=""'],
      },
      ignoreHTTPSErrors: true,
      fullPageScreenshots: true,
      windowSize: '1920x1080',
      waitForAction: 250,
      waitForTimeout: 10000, 
      waitForNavigation: 'load',
      restart: 'context',
      keepCookies: false,
      keepBrowserState: false
    },

I have allready open here #3640 an issue before, but no response and the probleme has gone, but it's here again :'(

[Horsty80]

This issue, append in local too but it's not on each run, so it's seems to be random...

I don't know how to fix this

[Horsty80]

I've worked on a reproductible repo, you can find here https://github.com/Horsty80/codeceptjs-debug

Working test

pnpm i
pnpm working

Throwing error in After/Before without try/catch

 pnpm error-1

Lead to error but the CI is not block

  4) Error 1 @error1
       "before each" hook: Before for "test something 6":
     Error: done() called multiple times in hook <Error 1 @error1 "before each" hook: Before for "test something 6"> of file /Users/cyril/Workspace/codeceptjs-debug/tests/error-1_test.ts
      at /Users/cyril/Workspace/codeceptjs-debug/node_modules/.pnpm/codeceptjs@3.6.2_@babel+core@7.24.5_@babel+preset-env@7.24.5_react@18.2.0_typescript@5.4.5/node_modules/codeceptjs/lib/scenario.js:156:55
      at runNextTicks (node:internal/process/task_queues:60:5)
      at processTimers (node:internal/timers:511:9)

  5) Error 1 @error1
       "before each" hook: Before for "test something 5":
     Uncaught Async error
      at Timeout._onTimeout (tests/error-1_test.ts:7:11)
      at listOnTimeout (node:internal/timers:573:17)
      at processTimers (node:internal/timers:514:7)

Throwing error in some random Scenario without try/catch

 pnpm error-2

Lead to mentioned error ✖ "after all" hook: codeceptjs.afterSuite in "Error 2 @error2" in 0ms
But the CI is block and it's impossible to take back control.

-- FAILURES:

  1) Error 2 @error2
       "before each" hook: Before for "test something 5":
     Uncaught Async error
      at Timeout._onTimeout (tests/error-2_test.ts:16:11)
      at listOnTimeout (node:internal/timers:573:17)
      at processTimers (node:internal/timers:514:7)
  
  Scenario Steps:
  - I.amOnPage("https://www.google.com") at Context.<anonymous> (./tests/error-2_test.ts:4:5)
  
  
  Scenario Steps:
  - I.amOnPage("https://www.google.com") at Context.<anonymous> (./tests/error-2_test.ts:4:5)
  

  2) Error 2 @error2
       "before each" hook: Before for "test something 5":
     Error: done() called multiple times in hook <Error 2 @error2 "before each" hook: Before for "test something 5"> of file /Users/cyril/Workspace/codeceptjs-debug/tests/error-2_test.ts
      at /Users/cyril/Workspace/codeceptjs-debug/node_modules/.pnpm/codeceptjs@3.6.2_@babel+core@7.24.5_@babel+preset-env@7.24.5_react@18.2.0_typescript@5.4.5/node_modules/codeceptjs/lib/scenario.js:156:55


  FAIL  | 1 passed, 2 failed   // 2s
    --- STARTED "before each" hook: Before for "test something 6" ---
    I am on page "https://www.google.com"
    › [Browser:Info] Autofocus processing was blocked because a document already has a focused element.
    › [Browser:Info] Autofocus processing was blocked because a document already has a focused element.
    › [Browser:Error] Permissions policy violation: unload is not allowed in this document.
    › [Browser:Error] Permissions policy violation: unload is not allowed in this document.
    --- ENDED "before each" hook: Before for "test something 6" ---
    I see element button#W0wltc
    I click button#W0wltc
    --- ENDED "before each" hook: Before for "test something 7" ---
    I see element button#W0wltc
    I click button#W0wltc
    I dont see element button#W0wltc
[03]   ✔ test something 6 in 59ms
  ✔ test something 6 in 168ms
  test something 9
    I dont see element button#W0wltc
[01]   ✔ test something 7 in 177ms
  ✔ OK in 177ms

  OK  | 3 passed   // 3s
    --- STARTED "before each" hook: Before for "test something 9" ---
    I am on page "https://www.google.com"
    › [Browser:Info] Autofocus processing was blocked because a document already has a focused element.
    › [Browser:Error] Permissions policy violation: unload is not allowed in this document.
    --- ENDED "before each" hook: Before for "test something 9" ---
    I see element button#W0wltc
    I click button#W0wltc
    I dont see element button#W0wltc
[03]   ✔ test something 9 in 168ms
  ✔ OK in 168ms


-- FAILURES:
  ✖ "after all" hook: codeceptjs.afterSuite in "Error 2 @error2" in 0ms

Throwing error in some random Scenario with try/catch

 pnpm error-3

Lead to the same error as pnpm error-2 the CI / terminal is block and impossible to take back the control

Throwing error in Before with try/catch

 pnpm error-4

Lead to the same error as pnpm error-1, test failed, but we take back the control.

[Horsty80]

So my first conclusion are something in my test code throw an async error and it's not try/catch and Codecept can not handle it correctly.
But on my workflow of test (not the linked repository) I use codeceptJs method, and I have remove all throw that I can find on my custom helper.

Tomorrow I will work on adding some try/catch on stuff I presume can lead to unhandled error, but is it possible that some error inside codeceptJs code can lead to unhandled error ?

[Horsty80]

🎉UPDATE

It's seems that if I remove retry: 3 in the codecept.conf.ts the "blocking" not happens anymore.
I will try some retry configuration to check this clue

[Horsty80]

I invok @DavertMik 🙌 you have work on the global retry feature and maybe you can help me on it or how to find the root cause

[Horsty80]

Last test and last conclusion ^^

For codeceptjs: 3.4.1 is working ✅
for 3.5.0 is not working ❌

With the same config

After checking changelog, I've found the line that cause the issue from the fix here #3667

https://github.com/codeceptjs/CodeceptJS/pull/3667/files#diff-9c88056d70f40caa621cb9519252872211ec64f80a84fa1c127c1f98880a93b8R47

I've try to replace this code block

    if (Number.isInteger(+retryConfig)) {
       if (test.retries() === -1) test.retries(retryConfig);
       return;
     }

By this:

    if (Number.isInteger(+retryConfig)) {
       if (isNotSet(test.retries())) test.retries(retryConfig);
       return;
     }

I don't really understand what is in cause, but clearly the test.retries() === -1 cause this issue

[Horsty80]

... Wrong fixed, that is not the root cause of the issue.
But strangely adding line of code here have impact on the blocking issue 🤔 even a console.log() inside the if achieve the same thing...

So I keep investigate ✌️

[andonary]

I think using a console.log inside of this force JavaScript to load this part of code. And thus import Suite from Mocha: https://github.com/Horsty80/CodeceptJS/blob/3c009e0759b3ddda7caa1984209fe248213553d2/lib/scenario.js#L175

test.retries is an inside method of Mocha:
https://github.com/mochajs/mocha/blob/master/lib/suite.js#L129

to me, test.retries should not be undefined. Not sure where should be the root cause

[Horsty80]

to me, test.retries should not be undefined. Not sure where should be the root cause

Yes it's test.retries is defined, just a wrong idea of the root cause for me ( that for i delete my PR)
I'm keep investigate, but the code base of the lib and the project are difficult to understand

test.retries for Scenario and not Suite is defined here if i understand:
https://github.com/mochajs/mocha/blob/master/lib/test.js#L75

[Horsty80]

Maybe link with #4197 ?

[kobenguyent]

We patched the retryTo issue in 3.5.11 > https://codecept.io/changelog/#_3-5-11 and the stale issue is found by you from 3.5.0. So the issue should be somewhere else, I think.