Skip to content

FIX: Update and cleanup the peer review section of our guide #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Dec 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 0 additions & 30 deletions _config.yml

This file was deleted.

69 changes: 38 additions & 31 deletions _static/pyos.css
Original file line number Diff line number Diff line change
@@ -1,53 +1,60 @@

/* HEADER BLOCK */
div.header__block {
color: #222;
body, p {
font-size: 1.04em;
}

/* not working and not sure why */
.caption-text {

h1, h2, h3 {
font-family:'Franklin Gothic Medium', Arial, sans-serif;
font-weight: bold;
}

h1 {
margin-bottom: 1em;
font-size: 2.5rem;
}

/*
.announcement div {
background-color: #ccc;
} */
h2 {
margin-top: 1.7em;
}

.header-item.announcement, .header-item.announcement .noprint {
background-color:#ccc;
h3, .content-container h3 {
margin-top: 1.6em!important;
}

div .announcement .header-item .noprint, header-item.announcement {
background-color: #ccc!important;
ul li p {
padding-bottom: .8em;
line-height: 1.5em;
}

/* notes */
.epigraph {
font-style: italic;
border-left: 4px solid #ccc!important;
margin-left: 0!important;
font-size: 1.2em!important;
}

div.admonition.note .admonition-title, div.admonition.note .admonition-title::before {
background-color: #C1CFD4;
div.header__block {
color: #222;

}

div.admonition.note {
border-color: #C1CFD4;
background-color: #E3F4FA;
/* not working and not sure why */
.caption-text {

}


div.admonition.important .admonition-title, div.admonition.important .admonition-title::before {
background-color: #107762;
color: #ffffff!important;
/*
.announcement div {
background-color: #ccc;
} */
/*
.header-item.announcement, .header-item.announcement .noprint {
background-color:#ccc;
}

div.admonition.important .admonition-title {
background-color: #107762;
border-color: #1abc9c;
div.admonition.note {
margin-top: 4em;
margin-bottom: 4em;
background-color: #F8F9FB
}

div.admonition.important {
border-color: #0e6654;
background-color: #ecfcf9;
}
2 changes: 1 addition & 1 deletion about-peer-review/code-of-conduct.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ go there now.](https://www.pyopensci.org/governance/code-of-conduct)
## NOTE: we are in the process of moving this file to our governance documentation and making significant changes to our code of conduct.


- We are committed to providing a friendly, safe and welcoming environment for all, regardless of gender, sexual orientation, disability, ethnicity, religion, or similar personal characteristic.
- We are committed to providing a friendly, safe and welcoming environment for all, regardless of gender, sexual orientation, disability, ethnicity, religion, age, or any personal characteristics.
- Please avoid using openly sexual nicknames or other nicknames that might detract from a friendly, safe and welcoming environment for all.
- Please be kind and courteous. There’s no need to be mean or rude.
- Respect that people have differences of opinion and that every design or implementation choice carries a trade-off and numerous costs. There is seldom a right answer.
Expand Down
21 changes: 13 additions & 8 deletions about-peer-review/how-peer-review-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ high quality tools
that supports scientists across domains with a suite of different data
types and structures.

### Who reviews pyOpenSci packages ?
### Who reviews pyOpenSci packages?

Our peer review process is run by volunteer members of the Python scientific
community:
Expand All @@ -30,19 +30,24 @@ Our entire peer review process occurs on GitHub in the

We use GitHub because:

* It is free to create an account
* Anyone can read the review discussion without an account making the process entirely open
* It facilitates collaboration and supports community around a package
* It facilitates open discussion between reviewers and package maintainers and the pyOpenSci volunteers
* Numerous packages store their code bases on GitHub
* It is free to create an account,
* Anyone can read the review discussion without an account, making the process entirely open,
* It facilitates collaboration and supports the community around a package,
* It facilitates an open discussion between reviewers, package maintainers and the pyOpenSci volunteers,
* Numerous packages store their code bases on GitHub.

### We use GitHub issue templates and labels to organize the review steps

* We use GitHub issue templates as submission templates for new reviews and pre-submission review questions.
* We label issues to track every step of the package submission and review progress (e.g. [1/initial-editor-checks, 2/reviewers-needed, 6/pyopensci-approved](https://github.com/pyOpenSci/software-review/labels)
* We label issues to track every step of the package submission and review progress (e.g. [1/initial-editor-checks, 2/reviewers-needed, 6/pyopensci-approved](https://github.com/pyOpenSci/software-review/labels)).

```{note}
Click [here](https://github.com/ropensci/software-review/issues/24) to read the review thread from an rOpenSci review of the `ropenaq` package. Note that the process is an ongoing conversation until the package is accepted. Two external reviews are important milestones in the review process.
[Click here to read the review thread from a December 2022
pyOpenSci pre-submission issue.](https://github.com/pyOpenSci/software-review/issues/65) Note the conversational
tone of the pre-review. In this case the package was improved
before the formal review even began.

In the actual review process, two external reviews are important milestones. The editor will also provide the author with some feedback.
```

For more detailed overview of the peer review process, [check out our process
Expand Down
6 changes: 3 additions & 3 deletions about-peer-review/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Software peer review refers to a peer-review process that focuses on open source
* Code quality,
* Documentation quality,
* Package usability,
* Test coverage that supports both maintenance of code function. Test coverage also makes it easier for contributors to see how their contributions impacts other parts of the code,
* Test coverage that supports the maintenance of code function. Test coverage also makes it easier for contributors to see how their contributions impact other parts of the code,
* Evaluation of infrastructure such as continuous integration to run rest suites and check code linters, that supports automated checks on pull requests. This infrastructure supports software quality and reliability.

## What types of packages does pyOpenSci review?
Expand All @@ -20,7 +20,7 @@ pyOpenSci reviews higher level software packages that support scientific workflo

<img src="../images/python-stack-jupyter-earth.png" alt="Image showing the tiers of software in the python ecosystem starting with Python itself and as you move out packages become more domain specific. In this image packages like xarray and numpy are considered core to scientific python. Packages and distributions like astropy, simpeg and metpy are considered to be domain specific." width="700px">

Diagram showing the tiers of software in the python ecosystem starting with Python itself and as you move out packages become more domain specific. In this image packages like xarray and numpy are considered core to scientific python. Packages and distributions like astropy, simpeg and metpy are considered to be domain specific.. pyOpenSci's review
Diagram showing the tiers of software in the python ecosystem starting with Python itself and as you move out packages become more domain specific. In this image packages like xarray and numpy are considered core to scientific python. Packages and distributions like astropy, simpeg and metpy are considered to be domain specific. pyOpenSci's review
process focuses on domain specific packages rather than core packages as
these packages tend to have more variability in long term maintenance and
package infrastructure and quality compared to established core packages. **Source: ["Jupyter meets earth" project](https://jupytearth.org/jupyter-resources/introduction/ecosystem.html)**
Expand Down Expand Up @@ -54,7 +54,7 @@ faced within the scientific Python community are broader in scale given the
numerous and diverse applications that the Python programming language is used for.

```{note}
[This blog post](https://www.numfocus.org/blog/how-ropensci-uses-code-review-to-promote-reproducible-science/) written by editors from our partner organization, rOpenSci, is a good introduction to pyOpenSci software peer review
[This blog post](https://www.numfocus.org/blog/how-ropensci-uses-code-review-to-promote-reproducible-science/) written by editors from our partner organization, rOpenSci, is a good introduction to pyOpenSci software peer review.
```

### Peer review of open source software helps maintain consistent quality
Expand Down
44 changes: 19 additions & 25 deletions about-peer-review/policies-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,25 @@

## Review Process Guidelines

- Packages are reviewed for quality, fit, documentation, clarity and the review process
is quite similar to a manuscript review (see our [packaging guide](../authoring/overview)
and [reviewing guide](../peer-review-guides/reviewer-guide) for more details). Unlike a
manuscript review, this process will be an ongoing conversation.
- Once all major issues and questions, and those addressable with reasonable effort, are
resolved, the editor assigned to a package will make a decision (accept, hold, or
reject). Rejections are usually done early (before the review process begins, see the
aims and scope section). In rare cases a package may also not be on-boarded after
review & revision. It is ultimately editor’s decision on whether or not to reject the
package based on how the reviews are addressed.
- Communication between authors, reviewers and editors will first and foremost take
place on GitHub, although you can choose to contact the editor by email. When
submitting a package, please make sure your GitHub notification settings make it
unlikely you will miss a comment.
- The author can choose to have their submission put on hold (editor applies the holding
label). The holding status will be revisited every 3 months, and after one year the
issue will be closed.
- If the author hasn’t requested a holding label, but is simply not responding, we
should close the issue within one month after the last contact intent. This intent
will include a comment tagging the author, but also an email using the email address
listed in the DESCRIPTION of the package which is one of the rare cases where the
editor will try to contact the author by email.
- If a submission is closed and the author wishes to re-submit, they’ll have to start a
new submission. If the package is still in scope, the author will have to respond to
the initial reviews before the editor starts looking for new reviewers.
pyOpenSci packages are reviewed for quality, fit, scope,
documentation and usability. The review process
is similar to a manuscript review, however it has a stronger
focus on Python packaging best practices.

Unlike a manuscript review, our peer review process is an ongoing conversation. Once all major issues and questions are addressed, the review editor package will make a decision to accept, hold, or reject the package.

Rejections are usually done early in the process, before the review process begins. In rare cases a package may also not be on-boarded into the pyOpenSci ecosystem after review & revision.

It is ultimately the editor’s decision on whether or not to reject the package based on how the reviews are addressed.

## Review communication approach

Communication between authors, reviewers and editors takes
place on GitHub. You can, however choose to contact the editor by email if needed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just wondering if this wouldn't go against the transparency of the whole process. What do you think @lwasser?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a great question @arianesasso in some cases I think there is a place for email. Many people miss github "pings" for instance i might have hundreds in a day of emails with github's name on them. Ideally i'd see it but sometimes i dont. So i think it's ok to email people too. but i totally hear your point and like that we can start by pinging on github.

oh another point -- in some cases maybe someone wants to review but they'd like some help or support. it might be more comfortable for them to acknowledge that in an email as well vs in a public issue. So i think there is a place for both! but again excellent insight here :)


When submitting a package, please make sure that your GitHub notification settings are setup to notify you when you receive feedback on the review issue.



## Submitting your package for review in other venues

Expand Down
23 changes: 11 additions & 12 deletions about-peer-review/pyopensci-related-joss-ropensci.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,28 +11,27 @@ The JOSS review process is about publication. A review from JOSS will provide
you with a citable, [Crossref digital object identifier (DOI)](https://www.crossref.org/).
pyOpenSci aligns closely with the broad mission of
JOSS to provide maintainers with credit for their open source work. However,
our mission is also more focused. pyOpenSci not open source maintainers getting academic credit for their work. We also support:
our mission is also more focused. pyOpenSci is not just about open source maintainers getting academic credit for their work. We also support:

* the Python tools that drive scientific open reproducible science workflows;
* enforcement and encouragement of Python-specific packaging standards across tools;
* and curated lists of peer reviewed, and maintained Python scientific software.
* the Python tools that drive scientific open reproducible science workflows;
* Enforcement and encouragement of Python-specific packaging standards across tools;
* Curated lists of peer reviewed, and maintained Python scientific software.

## How is review at pyOpenSci different from the JOSS review process?

We are not a typical publisher or journal. Rather we are a community that provides support for both a diverse group of software maintainers and long term maintenance of our packages.

The pyOpenSci review process is different from that of JOSS in a few ways:

1. Our review is specifically design to enforce modern, community-accepted best practices for Python packaging.
1. Our review is specifically designed to enforce modern, community-accepted best practices for Python packaging.
1. We place heavy emphasis on documentation and usability in our reviews and associated standardization of both.
1. We build community around and visibility for its tools.
1. We will promote packages and package maintainers once they are accepted into our ecosystem.
1. We support long term maintenance of packages. If the maintainer needs to step down, we will ensure a new maintainer takes over OR sunset and remove the package from our ecosystem.
1. We support long term maintenance of packages. If the maintainer needs to step down, we will ensure a new maintainer takes over or sunset and remove the package from our ecosystem.
1. We provide a welcoming forum for you to ask questions and get help with maintaining your package as needed.

JOSS reviews are also [more limited in scope](https://joss.readthedocs.io/en/latest/review_criteria.html)
compared to pyOpenSci. Some of the
[JOSS submission criteria](https://joss.readthedocs.io/en/latest/review_criteria.html)
JOSS reviews are also [more limited in scope](https://joss.readthedocs.io/en/latest/submitting.html?highlight=scope#submission-requirements)
compared to pyOpenSci. Some of the JOSS submission criteria
are, in places, less stringent or less specific to the Python programming
language than those of pyOpenSci.

Expand All @@ -41,8 +40,8 @@ language than those of pyOpenSci.
You don't have to chose between pyOpenSci and JOSS; You can submit your package to both.

pyOpenSci and JOSS are complementary, partner organizations. You don't have
to chose one or the other! After a package to pyOpenSci has been reviewed and
accepted by pyOpenSci you can also choose to register it with JOSS.
to choose one or the other! After a package to pyOpenSci has been reviewed and
accepted you can also choose to register it with JOSS.

JOSS has [more limited scope](https://joss.readthedocs.io/en/latest/review_criteria.html)
for packages that it will review. For instance, while pyOpenSci will review
Expand All @@ -56,5 +55,5 @@ You do not need to go through two reviews!
Once accepted by JOSS, you now have both a pyOpenSci acceptance and one by JOSS.

* JOSS will give you a Crossref DOI for citation.
* And you can display that and your pyOpenSci peer reviewed badge at the top of your package's README.md file.
* You can display that and your pyOpenSci peer reviewed badge at the top of your package's README.md file.

3 changes: 3 additions & 0 deletions accepted-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

- [ ] **Archive** (JOSS only, may be post-review): The repository DOI resolves correctly
- [ ] **Version** (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?
36 changes: 36 additions & 0 deletions appendices/editor-in-chief-checks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
```markdown
## Editor in Chief checks

Hi there! Thank you for submitting your package for pyOpenSci
review. Below are the basic checks that your package needs to pass
to begin our review. If some of these are missing, we will ask you
to work on them before the review process begins.

- [ ] **Installation** The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
- [ ] The package imports properly into a standard Python environment `import package-name`.
- [ ] **Fit** The package meets criteria for [fit](https://www.pyopensci.org/peer-review-guide/about-peer-review/aims-and-scope.html#package-scope) and [overlap](https://www.pyopensci.org/peer-review-guide/about-peer-review/aims-and-scope.html#package-overlap).
- [ ] **Documentation** The package has sufficient online documentation (README, sphinx docs) to allow us to evaluate package function and scope *without installing the package*. This includes:
- [ ] Short tutorials or [vignettes](https://kbroman.org/pkg_primer/pages/vignettes.html) that help a user understand how to use the package and what it can do for them (often these have a name like "Getting started")
- [ ] API documentation: this includes clearly written docstrings with variables defined using a standard docstring format. *We suggest using the [Numpy](https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard) docstring format*.
- [ ] **README** The package has a `README.md` file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
- [ ] **Contributing File** The package has a `CONTRIBUTING.md` file that details how to install and contribute to the package.
- [ ] **Issue Submission Documentation** All of the information is filled out in the `YAML` header of the issue (located at the top of the issue template).
- [ ] **Automated tests** Package has a testing suite and is tested via GitHub actions or another Continuous Integration service.
- [ ] **License** The package has an [OSI approved license](https://opensource.org/licenses).
- [ ] **Repository** The repository link resolves correctly.
- [ ] **Package overlap** The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
- [ ] **Archive** (JOSS only, may be post-review): The repository DOI resolves correctly.
- [ ] **Version** (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---
- [ ] [Initial onboarding survey was filled out ](https://forms.gle/F9mou7S3jhe8DMJ16)
We appreciate each maintainer of the package filling out this survey individually. :raised_hands:
Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. :raised_hands:
---

*******

## Editor comments


```
Loading