Open
Description
With the current effort on testing, we should start looking into our run duration, platform coverage, and hard data on flakyness / disabled tests.
This issue here is to record that information over time.
This is the first sample (of #3535), ran on 10/19/2024
| Target | Prep Time | Non-flaky | Flaky |
|---|---|---|---|
| windows | 238 s | 256 tests, 159 skipped in 81.173s | 166 tests, 66 skipped in 263.612s |
| rootful-1.6 | 277 s | 562 tests, 276 skipped in 120.279s | 2 runs, 737 tests, 81 skipped, 2 failures in 1024.881s |
| rootful-1.7 | 292 s | 562 tests, 276 skipped in 141.374s | 735 tests, 76 skipped in 1134.421s |
| rootful-2 | 273 s | 562 tests, 276 skipped in 124.186s | 735 tests, 80 skipped in 1041.623s |
| rootful-2-arm | 458 s | 567 tests, 275 skipped in 185.060s | 735 tests, 81 skipped in 1308.355s |
| rootless-1.6 | 283 s | 502 tests, 305 skipped in 161.850s | 727 tests, 143 skipped in 893.197s |
| rootless-1.7 | 272 s | 550 tests, 274 skipped in 130.962s | 727 tests, 115 skipped in 1020.648s |
| rootless-1.7-slirp | 288 s | 550 tests, 274 skipped in 151.411s | 727 tests, 115 skipped in 1012.942s |
| rootless-2 | 274 s | 550 tests, 274 skipped in 135.481s | 2 runs, 729 tests, 119 skipped, 2 failures in 937.661s |
| ipv6 | 272 s | 341 tests, 328 skipped in 70.056s | na |
| kube | 153 s | 33.571 s | na |
| docker | 25 s | 467 tests, 302 skipped in 126.707s ipv6: 331 tests, 325 skipped in 15.012s |
CURRENTLY BUST |
Here is the list of the worst offenders (taken from the arm run):
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer 2m38.51s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-stargz-no-byAddr 2m21.76s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants 2m7.84s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file 1m24.52s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory 1m3.08s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory 42.04s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRunWithLogBinary 27.11s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeImages 26.48s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-byAddr 25.61s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-no-byAddr 25.56s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRunPort 24.71s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeRestart 24.51s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeStart 22.37s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRestartIPCContainer 21.37s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants/Login_against_`tls:_true_port:_443_auth:_token` 21.36s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_exists_and_is_a_directory 21.23s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_is_in_a_volume 21.08s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_end_with_`/.` 21.04s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_not_end_with_`/.` 21s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposePauseAndUnpause 20.92s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeDownRemoveUsedNetwork 20.78s
Current take-aways:
- arm is significantly slower - either the machine we are using is not big enough, or it is emulated, or github offering is not really good...
- build time overall is consistent - at 4 minutes though, we should be able to cut that to 1 minute or less (implies rethinking the way we build the image and possibly push the "dependencies" part on a registry instead of rebuilding them every time)
- windows testing is really weak, with 100 safe tests, and 100 flaky tests
- we now have about 300 safe tests (not counting ipv6 and kube), and about 650 "flaky" tests - so, about 1/3 are considered safe - when the
containertests suite will be done, the balance should shift towards 50% - for the worst offenders:
- TestCopy are very slow - current in-progress rewrite will parallelize and reduce this
- Compose tests are definitely the number 1 contributor (including some IPFS tests) - we should look into this and figure out why compose is so slow
- current flakyness status
- 2 failed runs out of the 8 main target platforms (with 4 test failures) - these are both
TestIPFSCompNoBuild- the issue being likely [MAJOR] content digest not found #3513 (it might be a snapshotter issue though, and not a nerdctl one) - 4 "rewritten" tests are explicitly flagged as flaky (3 on windows, 1 on IPFS)
- 9 tests are currently disabled (marked
NerdctlNeedsFixing), although some of them might have been fixed recently - they should be reviewed
Tagging @djdongjin @AkihiroSuda
Will post other samples on this ticket in the future to track progress.
Metadata
Metadata
Assignees
Labels
No labels