Skip to content

Migrate test setup/teardown to fixtures #413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zbeekman opened this issue Jul 11, 2019 · 0 comments · Fixed by #421
Closed

Migrate test setup/teardown to fixtures #413

zbeekman opened this issue Jul 11, 2019 · 0 comments · Fixed by #421

Comments

@zbeekman
Copy link
Contributor

Right now, the tests are somewhat stateful: Tests modify inputs/outputs of other tests and/or don't clean up after themselves properly. This can cause problems when JSON-Fortran is added as an ExternalProject in a CMake "SuperBuild" with testing enabled.

What happens is that subsequent runs of the build will see that JSON-Fortran is up to date, skipping the configure and build steps, but will run the test step. (Or, perhaps it will re-run the configure step I'm not 100% clear on this, but I think my initial understanding is correct.) Because the test inputs/outputs are stateful, the initial run after configuring and building will pass, but subsequent runs will fail.

To resolve this the following steps should be taken:

  1. Analyze inter-test run order dependencies. CTest allows tests to be run in parallel, and some tests may be taking the output of others as inputs. This should be explicitly declared.
  2. CMake/CTest has added support for "test fixtures" which allow setup/teardown tasks for groups of tests to be declared. Use this to copy inputs (and cleanup outputs) between test runs.
zbeekman added a commit to nrc-fuels/json-fortran that referenced this issue Jul 22, 2019
 - Fixes jacobwilliams#413
 - Use fixtures to delete old json outputs before re-running tests, then copy
   pristine inputs back into the build directory
zbeekman added a commit to nrc-fuels/json-fortran that referenced this issue Jul 22, 2019
 - Fixes jacobwilliams#413
 - Also fixes regression testing and output validation
zbeekman added a commit to nrc-fuels/json-fortran that referenced this issue Jul 23, 2019
 - Fixes jacobwilliams#413
 - Also fixes regression testing and output validation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants