Skip to content

improve files and exclude parameter handling #12543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

joennlae
Copy link

@joennlae joennlae commented Apr 7, 2022

"Problem" description

So this is not a really big deal it just cost me a lot of time and will likely happen to someone else in the future.

I have a bit of a complex file structure due to imported libraries etc. and this led to the error.

File Structure

# File structure
src/python/module_a/__init__.py
src/python/module_a/a.py
src/python/module_a/b.py
src/python/module_b/__init__.py
src/python/module_b/a.py
src/python/main.py
src/python/config.json
src/python/sample.json

mypy.ini

# part of mypy.ini
files = src/python/*,
exclude = (?x)(
    ^config\.json$
    | ^sample\.json$
    | \.json$
  )

It still tried to check the config.json and sample.json.

Error message

src/python/config.json: error: Duplicate module named "__main__" (also at
"src/python/sample.json")
src/python/config.json: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules for more info
src/python/config.json: note: Common resolutions include: a) using `--exclude` to avoid checking one of them, b) adding `__init__.py` somewhere, c) using `--explicit-package-bases` or adjusting MYPYPATH

Solution

After a long search I found out that when using a * in the path it gets expanded here:

for path in paths:
path = expand_path(path.strip())
globbed_files = fileglob.glob(path, recursive=True)
if globbed_files:
expanded_paths.extend(globbed_files)

And then the *.json files fall through the cracks here and do not get checked:

else:
mod = os.path.basename(path) if options.scripts_are_modules else None
sources.append(BuildSource(path, mod, None))

Proposed Solution

I thought I would fix that and wrote a unit test mocking the path expansion. Additionally, I added the filters again at the end when all the files are added. I do believe this should make it more stable in general and hopefully make adding files more predictable in the future.

Fixes #10377 #10820

@joennlae joennlae force-pushed the improve-files-and-exclude-handling branch from 99b62b3 to fc6c38c Compare April 7, 2022 20:29
@joennlae joennlae force-pushed the improve-files-and-exclude-handling branch from fc6c38c to c316ca6 Compare April 7, 2022 20:31
@joennlae
Copy link
Author

joennlae commented Apr 7, 2022

@JukkaL I now use a very conservative approach. I could also change it to automatically remove all files that are not *.py and *.pyi. At the moment I opted to just add more informative output.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 8, 2022

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

@nkakouros
Copy link

I don't think this fixes either of #10377 and #10820. These have to do with the fact that the exclude option is not respected when follow-imports is set to normal and this PR is about excluding non python files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

exclude should imply a per-module follow import skip for matching modules
2 participants