Skip to content

Run full tests only on Wednesday scheduled jobs #1833

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 13, 2022
Merged

Conversation

seisman
Copy link
Member

@seisman seisman commented Mar 19, 2022

Description of proposed changes

Address comment #1686 (comment).

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
  • If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

  • /format: automatically format and lint the code
  • /test-gmt-dev: run full tests on the latest GMT development version

@seisman seisman added the maintenance Boring but important stuff for the core devs label Mar 19, 2022
@seisman seisman added this to the 0.6.1 milestone Mar 19, 2022
@seisman seisman added the skip-changelog Skip adding Pull Request to changelog label Mar 19, 2022
@seisman
Copy link
Member Author

seisman commented Mar 19, 2022

It should work, but I can't test it.

@weiji14
Copy link
Member

weiji14 commented Mar 19, 2022

I was thinking of running these full tests on a less regular basis, maybe weekly or monthly instead?

@seisman
Copy link
Member Author

seisman commented Mar 19, 2022

I was thinking of running these full tests on a less regular basis, maybe weekly or monthly instead?

I agree. Changing the following lines to weekly?

# Schedule daily tests

@weiji14
Copy link
Member

weiji14 commented Mar 19, 2022

I was thinking of running these full tests on a less regular basis, maybe weekly or monthly instead?

I agree. Changing the following lines to weekly?

# Schedule daily tests

But this means the daily unit tests will become weekly.

Or is there a way to just run the doctests, and not the unit tests? That way we could put it in the docs workflow as a weekly/monthly scheduled run (in addition to the current docs build).

@seisman
Copy link
Member Author

seisman commented Mar 21, 2022

I was thinking of running these full tests on a less regular basis, maybe weekly or monthly instead?

I agree. Changing the following lines to weekly?

# Schedule daily tests

But this means the daily unit tests will become weekly.

We have PRs merging into the main branch every few days, so the unit tests are also run every few days, making the daily unit tests less necessary. So a weekly job is still acceptable to me.

Or is there a way to just run the doctests, and not the unit tests? That way we could put it in the docs workflow as a weekly/monthly scheduled run (in addition to the current docs build).

Sphinx has its own doctest extension, which can run doctests.

cd doc/
sphinx-build -b doctest . _build

The above command works, but it reports many "errors" that are not reported by pytest.

run: make test PYTEST_EXTRA="-r P"

# Run full tests including doctests
- name: Run full tests
if: github.event_name == 'schedule'
Copy link
Member

@weiji14 weiji14 May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if GitHub Actions uses & or and, but try something like this to run full tests on Wednesday only. Xref https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule

Suggested change
if: github.event_name == 'schedule'
if: github.event_name == 'schedule' && github.event.schedule == '0 0 * * 3'

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if GitHub Actions uses & or and,

It's && (https://docs.github.com/en/actions/learn-github-actions/expressions#operators).

@seisman seisman changed the title Run full tests in scheduled daily jobs Run full tests in on Wednesday scheduled job May 11, 2022
@seisman seisman changed the title Run full tests in on Wednesday scheduled job Run full tests in on Wednesday scheduled jobs May 11, 2022
@@ -134,9 +134,15 @@ jobs:
pip install dist/*

# Run the tests
- name: Test with pytest
- name: Run tests
if: github.event_name != 'schedule'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if statement means the tests won't be run in daily jobs, which is not what we want. However, removing this if statement means on Wednesday job, most tests will be run twice by both make test and make fulltest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about something like

Suggested change
if: github.event_name != 'schedule'
if: github.event.schedule != '0 0 * * 3'

Though not sure if it will error for regular (non-scheduled) tests, e.g. in PRs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No errors in PRs.

@seisman seisman changed the title Run full tests in on Wednesday scheduled jobs Run full tests in Wednesday scheduled jobs May 11, 2022
@weiji14 weiji14 changed the title Run full tests in Wednesday scheduled jobs Run full tests only on Wednesday scheduled jobs May 11, 2022
@weiji14 weiji14 removed the skip-changelog Skip adding Pull Request to changelog label May 11, 2022
Copy link
Member

@weiji14 weiji14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also need to update doc/maintenance.md to mention the new full tests schedule.

Copy link
Member

@weiji14 weiji14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Just one last tiny suggestion, and wait for second approval to merge.

@weiji14 weiji14 added the final review call This PR requires final review and approval from a second reviewer label May 11, 2022
@seisman seisman merged commit d353399 into main May 13, 2022
@seisman seisman deleted the schedule-fulltest branch May 13, 2022 03:05
@seisman seisman removed the final review call This PR requires final review and approval from a second reviewer label May 13, 2022
sixy6e pushed a commit to sixy6e/pygmt that referenced this pull request Dec 21, 2022
weiji14 added a commit that referenced this pull request Dec 30, 2022
Previous expression was matching too strictly on the crontab, so using endsWith instead. xref https://docs.github.com/en/actions/learn-github-actions/expressions#endswith. Patches #1833.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Boring but important stuff for the core devs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants