Skip to content

pkg/dartdev/test/commands/create_integration_test takes too long to fail #51037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nshahan opened this issue Jan 17, 2023 · 3 comments
Closed

pkg/dartdev/test/commands/create_integration_test takes too long to fail #51037

nshahan opened this issue Jan 17, 2023 · 3 comments
Assignees
@bkonyi
Labels
area-dart-cli Use area-dart-cli for issues related to the 'dart' command like tool.

Comments

@nshahan
Copy link
Contributor

nshahan commented Jan 17, 2023

I've noticed that breaking the test pkg/dartdev/test/commands/create_integration_test causes CI on the pkg-linux-release-try bot to run for close to an extra hour. See the results and timeline here https://ci.chromium.org/ui/p/dart/builders/try/pkg-linux-release-try/71378/overview

I'm not sure how long that link will stay active but to summarize, it takes ~17 minutes for the step titled "package unit tests" to complete. This found one test failure: pkg/dartdev/test/commands/create_integration_test. That single failure needs to be deflaked which takes another 53 minutes. And these times are the runtimes, not including time spent waiting for the resources to run.

Would it be possible for this test to fail faster?

Here is the log from the failure:

/===============================================================================================\
| pkg/dartdev/test/commands/create_integration_test broke (Pass -> RuntimeError, expected Pass) |
\===============================================================================================/

--- Command "vm" (took 11:35.000418s):
DART_CONFIGURATION=ReleaseX64 CHROME_PATH=/b/s/w/ir/cache/builder/sdk/third_party/browsers/chrome/chrome/google-chrome out/ReleaseX64/dart-sdk/bin/dart --enable_asserts -Dtest_runner.configuration=unittest-asserts-release-linux --ignore-unrecognized-flags --packages=/b/s/w/ir/cache/builder/sdk/.dart_tool/package_config.json /b/s/w/ir/cache/builder/sdk/pkg/dartdev/test/commands/create_integration_test.dart

exit code:
255

stdout:
00:00 �[32m+0�[0m: create integration console�[0m

console: creating template

console: analyzing generated project

console: checking formatting

console: running the following commands:

  [run]

[1 / 1] Running "[run]"...

[1 / 1] Done "[run]".

00:18 �[32m+1�[0m: create integration console-full�[0m

console-full: creating template

console-full: analyzing generated project

console-full: checking formatting

console-full: running the following commands:

  [run]

[1 / 1] Running "[run]"...

[1 / 1] Done "[run]".

00:34 �[32m+2�[0m: create integration package�[0m

package: creating template

package: analyzing generated project

package: checking formatting

package: running the following commands:

  [run, example/template_project_example.dart]

[1 / 1] Running "[run, example/template_project_example.dart]"...

[1 / 1] Done "[run, example/template_project_example.dart]".

00:51 �[32m+3�[0m: create integration package-simple�[0m

package-simple: creating template

package-simple: analyzing generated project

package-simple: checking formatting

package-simple: running the following commands:

  [run, example/template_project_example.dart]

[1 / 1] Running "[run, example/template_project_example.dart]"...

[1 / 1] Done "[run, example/template_project_example.dart]".

01:06 �[32m+4�[0m: create integration server-shelf�[0m

server-shelf: creating template

server-shelf: analyzing generated project

server-shelf: checking formatting

server-shelf: running the following commands:

  [run, bin/server.dart]

[1 / 1] Running "[run, bin/server.dart]"...

stdout: Server listening on port 8080


[1 / 1] Done "[run, bin/server.dart]".

01:27 �[32m+5�[0m: create integration web�[0m

web: creating template

web: analyzing generated project

web: checking formatting

web: running the following commands:

  [pub, global, activate, webdev]

  [webdev, serve]

[1 / 2] Running "[pub, global, activate, webdev]"...

[1 / 2] Done "[pub, global, activate, webdev]".

[2 / 2] Running "[webdev, serve]"...

stdout: �[2K

stdout: [INFO] Connecting to the build daemon...

stdout: �[2K
[INFO] Generating build script...

stdout: �[2K
[INFO] Generating build script completed, took 1.1s�[2K
[INFO] 

stdout: �[2K
[INFO] Precompiling build script......

stdout: �[2K
[INFO] Precompiling build script... completed, took 25.2s�[2K
[INFO] 

stdout: �[2K
[INFO] Starting daemon...

stdout: �[2K
[INFO] Initializing inputs

stdout: �[2K
[INFO] Building new asset graph...

stdout: �[2K
[INFO] Building new asset graph completed, took 4.4s


stdout: �[2K
[INFO] Checking for unexpected pre-existing outputs....

stdout: �[2K
[INFO] Checking for unexpected pre-existing outputs. completed, took 2ms


stdout: �[2K
[INFO] Initializing inputs

stdout: �[2K
[INFO] Building new asset graph...

stdout: �[2K
[INFO] Building new asset graph completed, took 4.4s

stdout: �[2K
[INFO] 

stdout: �[2K
[INFO] Checking for unexpected pre-existing outputs....

stdout: �[2K
[INFO] Checking for unexpected pre-existing outputs. completed, took 2ms

stdout: �[2K
[INFO] 

stdout: �[2K
[INFO] Setting up file watchers...

stdout: �[2K
[INFO] Setting up file watchers completed, took 7ms�[2K
[INFO] 

stdout: �[2K

stdout: [INFO] Registering build targets...

stdout: �[2K

stdout: [INFO] Starting initial build...

stdout: �[2K

stdout: [INFO] Starting resource servers...

stdout: �[2K

stdout: [INFO] Serving `web` on http://127.0.0.1:41367/


stdout: �[2K

stdout: [INFO] About to build [web]...

stdout: �[2K

stdout: [INFO] Running build...

stdout: �[2K
[INFO] Generating SDK summary...

stdout: �[2K
[INFO] 7.4s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 9.8s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 14.5s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 15.9s elapsed, 0/1 actions completed.

stdout: �[2K
[WARNING] No actions completed for 15.9s, waiting on:
  - build_web_compilers:entrypoint on web/main.dart



stdout: �[2K
[INFO] Generating SDK summary completed, took 15.9s


stdout: �[2K
[SEVERE] build_web_compilers:sdk_js on asset:build_web_compilers/$package$: Error compiling dartdevc module:build_web_compilers|lib/src/dev_compiler/dart_sdk.sound.js

Unhandled exception:
PathNotFoundException: Cannot open file, path = '/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/dart-sdk/lib/_internal/ddc_platform_sound.dill' (OS Error: No such file or directory, errno = 2)
#0      _File.throwIfError (dart:io/file_impl.dart:618:7)
#1      _File.openSync (dart:io/file_impl.dart:473:5)
#2      _File.readAsBytesSync (dart:io/file_impl.dart:533:18)
#3      loadComponentFromBinary (package:kernel/kernel.dart:27:36)
#4      compileSdkFromDill (package:dev_compiler/src/kernel/command.dart:558:19)
#5      _compile (package:dev_compiler/src/kernel/command.dart:171:12)
#6      compile (package:dev_compiler/src/kernel/command.dart:52:12)
#7      internalMain (package:dev_compiler/ddc.dart:37:24)
#8      main (file:///b/s/w/ir/cache/builder/sdk/pkg/dev_compiler/bin/dartdevc.dart:18:14)
#9      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:293:33)
#10     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:192:26)




stdout: �[2K
[SEVERE] build_web_compilers:entrypoint on web/main.dart: Bad state: Unable to locate required sdk resource build_web_compilers|lib/src/dev_compiler/dart_sdk.sound.js



stdout: �[2K
[INFO] 17.9s elapsed, 1266/1266 actions completed.

stdout: �[2K
[INFO] Running build completed, took 18.1s
�[2K
[INFO] Caching finalized dependency graph...

stdout: �[2K
[INFO] Caching finalized dependency graph completed, took 683ms


stdout: �[2K
[SEVERE] Failed after 18.9s
�[2K
[INFO] ------------------------------------------------------------------------


06:27 �[32m+5�[0m�[31m -1�[0m: create integration web �[1m�[31m[E]�[0m�[0m

  TimeoutException after 0:05:00.000000: Test timed out after 5 minutes.

  package:test_api/src/backend/invoker.dart 338:28  Invoker._handleError.<fn>
  dart:async/zone.dart 1390:47                      _rootRun
  dart:async/zone.dart 1300:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 336:10  Invoker._handleError
  package:test_api/src/backend/invoker.dart 291:9   Invoker.heartbeat.<fn>.<fn>
  dart:async/zone.dart 1398:13                      _rootRun
  dart:async/zone.dart 1300:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 290:38  Invoker.heartbeat.<fn>
  dart:async-patch/timer_patch.dart 18:15           Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 398:19         _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 429:5          _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 192:26      _RawReceivePort._handleMessage
  

06:28 �[32m+5�[0m�[31m -1�[0m: create integration web-simple�[0m

web-simple: creating template

06:28 �[32m+5�[0m�[31m -1�[0m: create integration web�[0m

stdout: �[2K
[WARNING] Build script updated. Shutting down the Build Daemon.

stdout: 
�[2K
[WARNING] Please restart WebDev.



06:39 �[32m+5�[0m�[31m -1�[0m: create integration web-simple�[0m

web-simple: analyzing generated project

web-simple: checking formatting

web-simple: running the following commands:

  [pub, global, activate, webdev]

  [webdev, serve]

[1 / 2] Running "[pub, global, activate, webdev]"...

[1 / 2] Done "[pub, global, activate, webdev]".

[2 / 2] Running "[webdev, serve]"...

stdout: �[2K

stdout: [INFO] Connecting to the build daemon...

stdout: �[2K
[INFO] Generating build script...

stdout: �[2K
[INFO] Generating build script completed, took 969ms�[2K
[INFO] 

stdout: �[2K
[INFO] Precompiling build script......

stdout: �[2K
[INFO] Precompiling build script... completed, took 23.5s�[2K
[INFO] 

stdout: �[2K
[INFO] Starting daemon...

stdout: �[2K

stdout: [INFO] Initializing inputs

stdout: �[2K
[INFO] Building new asset graph...

stdout: �[2K
[INFO] Building new asset graph completed, took 3.7s
�[2K
[INFO] Checking for unexpected pre-existing outputs....�[2K
[INFO] Checking for unexpected pre-existing outputs. completed, took 1ms


stdout: �[2K
[INFO] Initializing inputs

stdout: �[2K
[INFO] Building new asset graph...�[2K
[INFO] Building new asset graph completed, took 3.7s�[2K
[INFO] �[2K
[INFO] Checking for unexpected pre-existing outputs....�[2K
[INFO] Checking for unexpected pre-existing outputs. completed, took 1ms�[2K
[INFO] 

stdout: �[2K
[INFO] Setting up file watchers...

stdout: �[2K
[INFO] Setting up file watchers completed, took 8ms�[2K
[INFO] 

stdout: �[2K

stdout: [INFO] Registering build targets...

stdout: �[2K

stdout: [INFO] Starting initial build...

stdout: �[2K

stdout: [INFO] Starting resource servers...

stdout: �[2K
[INFO] Serving `web` on http://127.0.0.1:43421/


stdout: �[2K
[INFO] About to build [web]...

stdout: �[2K
[INFO] Running build...

stdout: �[2K
[INFO] Generating SDK summary...

stdout: �[2K
[INFO] 7.0s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 9.1s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 13.2s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] 14.3s elapsed, 0/1 actions completed.

stdout: �[2K
[INFO] Generating SDK summary completed, took 14.2s


stdout: �[2K
[SEVERE] build_web_compilers:sdk_js on asset:build_web_compilers/$package$: Error compiling dartdevc module:build_web_compilers|lib/src/dev_compiler/dart_sdk.sound.js

Unhandled exception:
PathNotFoundException: Cannot open file, path = '/b/s/w/ir/cache/builder/sdk/out/ReleaseX64/dart-sdk/lib/_internal/ddc_platform_sound.dill' (OS Error: No such file or directory, errno = 2)
#0      _File.throwIfError (dart:io/file_impl.dart:618:7)
#1      _File.openSync (dart:io/file_impl.dart:473:5)
#2      _File.readAsBytesSync (dart:io/file_impl.dart:533:18)
#3      loadComponentFromBinary (package:kernel/kernel.dart:27:36)
#4      compileSdkFromDill (package:dev_compiler/src/kernel/command.dart:558:19)
#5      _compile (package:dev_compiler/src/kernel/command.dart:171:12)
#6      compile (package:dev_compiler/src/kernel/command.dart:52:12)
#7      internalMain (package:dev_compiler/ddc.dart:37:24)
#8      main (file:///b/s/w/ir/cache/builder/sdk/pkg/dev_compiler/bin/dartdevc.dart:18:14)
#9      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:293:33)
#10     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:192:26)




stdout: �[2K
[SEVERE] build_web_compilers:entrypoint on web/main.dart: Bad state: Unable to locate required sdk resource build_web_compilers|lib/src/dev_compiler/dart_sdk.sound.js



stdout: �[2K
[INFO] Running build completed, took 15.8s
�[2K
[INFO] Caching finalized dependency graph...

stdout: �[2K
[INFO] Caching finalized dependency graph completed, took 582ms


stdout: �[2K
[SEVERE] Failed after 16.4s
�[2K
[INFO] ------------------------------------------------------------------------


11:28 �[32m+5�[0m�[31m -2�[0m: create integration web-simple �[1m�[31m[E]�[0m�[0m

  TimeoutException after 0:05:00.000000: Test timed out after 5 minutes.

  package:test_api/src/backend/invoker.dart 338:28  Invoker._handleError.<fn>
  dart:async/zone.dart 1390:47                      _rootRun
  dart:async/zone.dart 1300:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 336:10  Invoker._handleError
  package:test_api/src/backend/invoker.dart 291:9   Invoker.heartbeat.<fn>.<fn>
  dart:async/zone.dart 1398:13                      _rootRun
  dart:async/zone.dart 1300:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 290:38  Invoker.heartbeat.<fn>
  dart:async-patch/timer_patch.dart 18:15           Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 398:19         _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 429:5          _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 192:26      _RawReceivePort._handleMessage
  

11:28 �[32m+5�[0m�[31m -2�[0m: create integration console-simple�[0m

console-simple: creating template

11:28 �[32m+5�[0m�[31m -2�[0m: create integration web-simple�[0m

stdout: �[2K
[WARNING] Build script updated. Shutting down the Build Daemon.
�[2K
[WARNING] Please restart WebDev.



11:29 �[32m+5�[0m�[31m -2�[0m: create integration console-simple�[0m

console-simple: analyzing generated project

console-simple: checking formatting

console-simple: running the following commands:

  [run]

[1 / 1] Running "[run]"...

[1 / 1] Done "[run]".

11:30 �[32m+6�[0m�[31m -2�[0m: �[31mSome tests failed.�[0m



Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
For example, 'dart test --chain-stack-traces'.

stderr:
Unhandled exception:
Dummy exception to set exit code.

--- Re-run this test:
python3 tools/test.py -n unittest-asserts-release-linux pkg/dartdev/test/commands/create_integration_test
@nshahan nshahan added the area-dart-cli Use area-dart-cli for issues related to the 'dart' command like tool. label Jan 17, 2023
@bkonyi
Copy link
Contributor

bkonyi commented Jan 17, 2023

We can probably update the timeout for these tests to be something more sane (2 minutes instead of the default 5 maybe?). The only issue with that is that these tests can just be slow in general, and we don't want to have false negatives when the test is just slow.

@bkonyi bkonyi self-assigned this Jan 17, 2023
@nshahan
Copy link
Contributor Author

nshahan commented Jan 17, 2023

One thing I'm wondering is why does it timeout? It seems like the failure should be reported up and cause the entire test to fail faster without changing any timeouts. In this case I'm removing a file from the packaged SDK so there is a missing file and we need to update build_web_compilers.

@nshahan
Copy link
Contributor Author

nshahan commented Jan 17, 2023

FYI I'm going to add a skip for this test temporarily as we update the SDK and then release an updated version of build_web_compilers that works with the SDK changes. Skipping seems to be easier than running it multiple times to approve the failure and land the change.

@nshahan nshahan mentioned this issue Jan 23, 2023
Closed
4 tasks
copybara-service bot pushed a commit that referenced this issue Feb 1, 2023
@nshahan
[dartdev] Enable create_integration_test
bdc1ddd 
Bump build_web_compilers dependency in the web template
pubspec.yaml.

New webdev and build families of packages have been published
that work with the Dart 3 SDK so this test should work again.

Fixes: #51106
Issue: #51037
Change-Id: I854fafb683bff20786b1b31b0368d08b49029e83
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280053
Reviewed-by: Ben Konyi <[email protected]>
Commit-Queue: Nicholas Shahan <[email protected]>
copybara-service bot pushed a commit that referenced this issue Feb 2, 2023
@nshahan
[dartdev] Skip create_integration_test on windows
6df68a7 
The test is still failing on windows only.
Skipping because the test takes too long to fail and deflake.

Issue: #51037
Issue: #51106
Change-Id: I804e4c7a4e378883943aa63816b71e6fe94de647
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/280250
Reviewed-by: Ben Konyi <[email protected]>
Commit-Queue: Nicholas Shahan <[email protected]>
@annagrin annagrin mentioned this issue Feb 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bkonyi bkonyi

Labels
area-dart-cli Use area-dart-cli for issues related to the 'dart' command like tool.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nshahan @bkonyi