MYPYPATH doesn't respect PEP420 namespace packages

Take this directory tree:

/path/to/src
└───namespace
    └───package
        │   __init__.py
        │
        └───__pycache__
                __init__.cpython-35.pyc

And this MYPYPATH:

> $env:MYPYPATH
/path/to/src

then running mypy -p namespace.package gives:

> mypy -p namespace.package
Can't find package 'namespace.package'

However, mypy works fine when you add an empty __init__.py into namepace (turning it into a regular package).

I know... I'm not sure what to do about this. The algorithm from PEP 420 is quite complex (namespace packages are only considered if no "classic" package or module is found). The workaround of adding a dummy __init__.py is simple enough (you could also have a dummy __init__.pyi so that Python is not affected by this file).

I also expect that usually you'd want to create a separate set of stubs rather than type-check the original source code (though obviously this may vary per package).

I'd like to bump this. In future, there will be python programmers who never heard of pre PEP420 era of __init__.py and that's good. What's not good is mypy forcing them to create dummy files.

It might get the ball rolling if somebody explained why this is such a big problem for mypy.

I just bumped into Jukka Lehtosalo and he said there shouldn't be any major issue blocking this. But given the easy workaround that exists for this, it'll likely require an outsider to write a PR.

What's not good is mypy forcing them to create dummy files.

This issue is more complex than you think.

I don't think it's trivial, I've read your previous comment - so I don't even think I'd be able to write a PR. I just think it's an issue worth looking into eventually

I'm working on a project with a directory structure like the following:

./a
./a/c
./a/c/file1.py
./b
./b/c
./b/c/file2.py

Note that both, a and b, contain a subfolder called c. Adding an __init__ everywhere defines some modules multiple times (the actual project contains dozens folders) and things get hairy. Any progress on this issue would be much appreciated.

Adding an __init__ everywhere defines some modules multiple times [...]

I presume you mean in each c folder but not in a or b right? So c is a namespace package and you have both a and b in PYTHONPATH or sys.path. (And when running mypy, you pass it both a and b, which will have the same effect -- or perhaps you pass it their parent directory or the current directory.)

With that setup, I agree you have a predicament. We will have to support namespace packages in mypy. At the very least it should be possible to do something like setting MYPYPATH=a:b and then run mypy -m c and have it find both parts of c.

Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without __init__.py are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty __init__.py turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add __init__.py (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of __init__.py becomes the special case, used only when you need some package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one.

Or perhaps this reflects your particular culture.

…

On Nov 25, 2017 21:48, "Carl Meyer" ***@***.***> wrote: Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without __init__.py are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty __init__.py turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add __init__.py (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of __init__.py becomes the special case, used only when you need some package-level code or imports. This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1645 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMunwfsZJ03cqKrWM6ONIipLvzuh_ks5s6Pu5gaJpZM4IuKil> .

Or perhaps this reflects your particular culture.

Certainly possible. I'd be curious to hear from another Py3-only culture
which (convincing to new Python 3 developers) arguments are used in
favor of continuing the tradition of empty __init__.py files. I'm not
aware of a strong practical argument that can be made for them, but if
I'm missing a good one I may start using it ;)

Yeah, that's why we eventually implemented mypy --namespace-packages.

And for those that stumble on this issue via Google, that flag was added in 0.640, so make sure you're on that or later, then use the aforementioned --namespace-packages argument (permalink)

FYI. For completeness, I'll mention that I recently realized that my patch only works for the top level namespace package. It doesn't help traversing in to nested sub-namespace packages. That doesn't affect us in practice so I'm not motivated to fix it.

It is the year 2020. Python 3 is the only supported major release of Python. Is using the non-default --namespace-packages flag still the recommended solution?

As a creator of various namespaced packages targeted for Python 3, this requires users of my packages to set this flag to get any type information out of my packages (requiring them to somehow know that my package is a namespace package, carrying all this baggage). Also as a developer of these packages, getting the __init__.pyi alternative to work in bdist_wheel seems downright impossible with Python 3, so I'm having a hard time building something that works on the major supported platform using a major supported feature in what is a pure python package.

I would open a PR to change the default but given the discussion here, the decision appears a lot more nuanced and personal than I currently understand, and I do not wish to upset the maintainers.

I hope that I am not unaware of some context around this issue, but I agree with @nickroeker that it would make sense for the --namespace-packages flag to be the default behaviour for mypy, especially as the current behaviour is really confusing for Py3-only users like myself.

I very recently stumbled into an issue (#8497) that I believe was due to this (adding the --namespace-packages flag solved it), and being a "relatively new" Python user that has never used Python 2, I had a very confusing time understanding why I was expected to add an empty __init__.py in a namespace package for mypy to work.

Is the main obstacle to making this behaviour default the need for retrocompatibility with Python 2 ?

Enabling support for namespace packages by default in Python 3 mode sounds like a good thing. Would somebody create a new issue about it? I haven't looked into this in detail, but we might need to fix some issues with namespace package support first.

I can create an issue this afternoon 👍

@JukkaL

Here's the issue that proposes supporting this by default: #8584

(Thanks @JukkaL -- I am very much in favor of the proposal.)

static: mypy requires __init__.py

static: mypy requires __init__.py (#3027)

I'm confused because I still get the same error when running mypy --namespace-packages . at the root of my repo that doesn't have a py file (mypy 0.782, python 3.7.7):

$ mypy --namespace-packages .
There are no .py[i] files in directory '.'

So what exactly is the work-around if --namespace-packages doesn't work?

I have several python packages I want to check under the root folder as well various folders and subfolders with python files.

I'm probably missing something simple and dumb. Thanks for the help.

@aloosley

@aloosley It's been a while so I'm not honestly sure anymore, but I think for --namespace-packages to work you may still need a py.typed file in the right place. I think the right place for that is in the non-namespace folders. For example, if you have package company.service, the service/ folder would need a py.typed file.

Aside from that, I find I generally need to run mypy from the outside. So instead of being in company/ and running mypy ./, go up a directory and run mypy company/.

@aloosley

@aloosley @nickroeker alternatively you can use the -p arg, this worked for me (note that you can specify either a package or a directory, not both):

mypy --namespace-packages -p mynamespace.mypackage

The behaviour is buggy in mypy v0.790 and earlier, this should be fixed in v0.800

Navigation Menu

MYPYPATH doesn't respect PEP420 namespace packages #1645

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

MYPYPATH doesn't respect PEP420 namespace packages #1645

Description

Activity

gvanrossum commented on Jun 4, 2016

djetelina commented on May 21, 2017

agronholm commented on May 21, 2017

agronholm commented on May 21, 2017

gvanrossum commented on May 21, 2017

djetelina commented on May 21, 2017

richardmss commented on Jul 27, 2017

gvanrossum commented on Jul 27, 2017

carljm commented on Nov 26, 2017

gvanrossum commented on Nov 26, 2017

carljm commented on Nov 26, 2017

18 remaining items

gvanrossum commented on Mar 13, 2019

nicktimko commented on May 29, 2019

warsaw commented on Aug 19, 2019

lynn-ca commented on Mar 10, 2020

ErwanDL commented on Mar 14, 2020

JukkaL commented on Mar 20, 2020

ErwanDL commented on Mar 20, 2020

russelldavis commented on Mar 30, 2020

aloosley commented on Jul 6, 2020

lynn-ca commented on Jul 6, 2020

csymeonides-mf commented on Feb 1, 2021

hauntsaninja commented on Feb 1, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions