Add primer tests

[DanielNoord]

TODO List:

• Create branch with changes in PR and branch with merge-base (see mypy)
• Run pydocstringformatter with merge-base branch in write mode
• Checkout to changes branch
• Do a non-write branch to get diff
• Prettify diff
• Probably tons more...

[coveralls]

Pull Request Test Coverage Report for Build 1999456191

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 1998301260:	0.0%
Covered Lines:	405
Relevant Lines:	405

---

💛 - Coveralls

[DanielNoord]

@Pierre-Sassoulas Small achievement! It works. There are changes on main that would lead to changes to pylint, but the new version of the primer only shows the changes that would be introduced by this PR.

I'm going to do some code cleanup and add some annotations/comments but I actually think this is almost mergeable in an alpha form for the project.

It also gave me some good insight into how to develop this in the future for pylint.

[Pierre-Sassoulas]

It also gave me some good insight into how to develop this in the future for pylint.

Yeap, having that in pylint will make the new checkers and every modification SO MUCH BETTER. I'm super hyped in anticipation.

[DanielNoord]

Yeap, having that in pylint will make the new checkers and every modification SO MUCH BETTER. I'm super hyped in anticipation.

I have written most of the code here to be somewhat copyable to pylint as well. One of the big issues is the speed: we need to run the program twice for every package we "prime" one. For pydocstringformatter this isn't an issue. For pylint this is...
I was thinking about maybe creating a "needs-primer" label so we can only schedule primers when necessary.. Hm, first 2.13 though 😄

[Pierre-Sassoulas]

need to run the program twice for every package we "prime" one.

Unless we save the baseline in the repository ? We'd also need to specify the commit for each primer repository. OR maybe it's possible to make a job manual (I did not check the doc of github actions) ?

[DanielNoord]

Unless we save the baseline in the repository ? We'd also need to specify the commit for each primer repository. OR maybe it's possible to make a job manual (I did not check the doc of github actions) ?

Yeah, I was thinking about that as well. But I foresee issues with fixing false-positives and comparing those etc. There is probably a reason why mypy doesn't save the baseline and runs twice for every primer test. They have more than enough knowledgeable contributors to do that optimisation if they saw a possibility/necessity to do so.

Yeah we can. We might also want to start with a smaller subset of repositories to do false positive checking for. I know that home-assistant have actually disabled pylint in some of their CI because it takes too long. So us running over them is perhaps not the best choice.

[github-actions]

According to the primer, this change has no effect on the checked open source code. 🤖🎉

[DanielNoord]

@Pierre-Sassoulas Not sure if you have the bandwidth to check this. I can merge as is, but I left it open to give you the opportunity to review (and potentially spot any errors that could be copied to pylint when I eventually copy this).

[Pierre-Sassoulas]

To be fair I skimmed most of the code but LGTM. It looks great actually.

[Pierre-Sassoulas]

Maybe we could create a test with pytest for that ? It seem automatable even if not very convenient because you need to duplicate the git repository.

[DanielNoord]

Yeah, I think it does not offer much: when would the test pass or fail?
At some point I could explore trying to create a script that does the same and handles cloning the repository, but for now I think this is fine.

[Pierre-Sassoulas]

Ha right it makes no sense as a test. A script to set it up is more like it.