Import gRPC stubs from the grpc-stubs project

[shabbyrobe]

Following on from #10020, it seemed from that discussion that there was both desire from the issue tracker on the grpc-stubs project to submit these types to typeshed, and willingness on the part of typeshed to accept them, so here's my best first effort at making that happen. Happy to sign any CLA or copyright assignment you require.

This PR attempts to massage the stubs into a hopefully acceptable shape. The original types are functional, albeit gappy. I'm not sure what typeshed's current stance is on incomplete types but I do see a fair bit of prior art in there so I presume there's at least some leeway.

Full disclosure, there's an element of handoff here in my motivation as I'm not involved in any way with any project that uses gRPC and haven't been for nearly 5 years. I'm in no position to continue as maintainer of those stubs as it is; I hoped that by submitting to typeshed that might open up the maintenance pool somewhat. If that disqualifies this from inclusion that's fair; I can't quite tell if the original submitter of typings to typeshed is expected to continue to be responsible after inclusion, but I will need to archive and deprecate grpc-stubs regardless of the outcome, as I don't have enough mental context for working with gRPC any more and no proximate opportunity to revive it.

I found while preparing this PR that what was in the grpc-sttubs project was substantially misaligned with the typeshed conventions so I've worked through all of the lints and test failures that came up while migrating. I've also ported the tests over from the approach used in grpc-stubs to the approach used in typeshed.

There were a couple of other seemingly related things I wasn't sure about. The implementation package is called grpcio; should I have bundled up the grpc* folders into a grpcio folder or are they ok where they are? Also, when using requires in my METADATA.toml, what is the correct way to refer to the base grpc package? Pretty sure I'm doing this wrong as I see I'm getting a third party subtest failure.

I am happy to spend a bit more time massaging this into a fit state for inclusion; I'm quite sure there's a fair bit that's still wrong about these (especially as they were intended to evolve alongside a project, but it just didn't pan out that way, and I ceased to be involved soon after their brisk creation, c'est la vie). They're a bit gappy in places and there were some outstanding issues on the original repo that I was not steeped enough in gRPC any more to be able to contextualise, let alone fix.

Please let me know whether it's worth proceeding and if there's anything that needs to be done, or if there's no pathway from this point and I should just archive the repo. Thank you.

[srittau]

Thanks for the PR! The stubtest checks currently fail: https://github.com/python/typeshed/actions/runs/7343810747/job/19994850116?pr=11204. This is the relevant output, I think:

× python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-4nj3_q11/grpc_785350fd81[74](https://github.com/python/typeshed/actions/runs/7343810747/job/19994850116?pr=11204#step:5:75)466f9c3ae4e38035d8f1/setup.py", line 33, in <module>
          raise RuntimeError(HINT)
      RuntimeError: Please install the official package with: pip install grpcio
      [end of output]

I have no experience with grpc, so please correct me if my understanding is wrong: While these are the stubs for "gprc", what you actually install is the grpcio package? This is something that our current test setup doesn't handle, although I don't think it would be too hard to implement.

[shabbyrobe]

Thanks for getting back to me so quickly. I can see the failure but not sure what to do about it. I think you're right that it boils down to the grpcio vs grpc thing. You're right that you install grpcio and get the set of packages I've added here. I wasn't sure if that meant I should put all the things I added here in a grpcio folder or if there was another mechanism, but it sounds like it might not actually be supported by the current test framework?

[AlexWaygood]

I wasn't sure if that meant I should put all the things I added here in a grpcio folder or if there was another mechanism, but it sounds like it might not actually be supported by the current test framework?

I think just putting the stubs in a grpcio folder should be fine. It'll mean the stubs will be uploaded to PyPI as types-gprcio rather than as types-grpc, but I don't think there'd be any other issues (though I might be missing something)

[srittau]

I haven't thought too deeply about it, but usually our stubtest checks install the package "x" when building the stubs in the "x" directory (the package will be called "types-x"). We could add a field like packages in the tool.stubtest table of METADATA.toml that will override this default.

On a related note: It would make sense to me to also add (dummy) stubs for grpcio* etc. that depend on the grpc* stubs so that users would install grpcio types-grpcio instead of grpcio and types-grpc. But that should probably done in a separate PR so we can work out the kinks with this one separately.

[shabbyrobe]

Putting it in grpcio worked @AlexWaygood, I am seeing useful output in the result of stubtest now. Looks like there's quite a few things for me to work through there, I'll pick off that list and then see what's left that I might need a hint or two with and report back.

[shabbyrobe]

Interesting, it looks like the typings in the aio package are reasonably complete. I don't remember that always being the case, but I might be misremembering. I've been pretty hands-off with grpc for a good long while now. I'm inclined to remove the aio typings from this PR, rather than fuss over all those stubtest warnings.

Does typeshed play nice when part of a package is typed well and doesn't require stubs but another part of it isn't, and does?

[Avasam]

What is the remaining work to get this PR merged?

I haven't investigated in details. But there's a few easy linting failures to fix. And the added test-cases aren't passing.

Stubtest is also whining about apparent mismatches between the stubs and what's found at runtime. stubtest isn't always 100% correct and sometimes we do have to "cheat" a little bit, but those cases still need to be explicitly excluded and explained.

[Akuli]

those cases still need to be explicitly excluded and explained.

I wouldn't worry too much about stubtest for this PR. We can simply silence its errors with allowlists (see any file named stubtest_allowlist.txt) and fix them later if needed.

The test cases (in @tests/test_cases/) should be fixed, because there's no good way to silence them. This seems like it should be very easy: just delete the line assert_type(grpc.aio.Channel(), grpc.aio.Channel). It is trying to instantiate an abstract class, which doesn't make sense.

[macro1]

i took a look at the failing tests and made a pr to @shabbyrobe's branch:
shabbyrobe#2
tested against fork under my user, and i believe it will result in CI fully passing

[srittau]

Thanks! Not a full review, just a few things I noticed below and a general observation: Except for general cases, we generally use from typing import ... imports instead of namespacing typing imports (typing.TypeVar etc.)

[srittau]

Considering this test is never run, only checked with type checkers, this is clearer:

Suggested change

	client_interceptors = [typing.cast(grpc.aio.ClientInterceptor, "interceptor")]
	client_interceptors: list[grpc.aio.ClientInterceptor] = []

[Avasam]

Other casts can also generally be avoided by making it a function parameter.

For example:

async def metadata(call: grpc.aio.Call) -> None:
    metadata = await call.initial_metadata()

instead of

async def metadata() -> None:
    metadata = await typing.cast(grpc.aio.Call, None).initial_metadata()

Not sure how important that is.

[srittau]

It's also unusual not to import Any directly from typing – at least here in typeshed.

Suggested change

	server_interceptors = [typing.cast(grpc.aio.ServerInterceptor[typing.Any, typing.Any], "interceptor")]
	server_interceptors: list[grpc.aio.ServerInterceptor[Any, Any]] = []

[Avasam]

Turns out there's a few places we do that, but I don't really see a good reason. I opened #13761 to standardise.

[Avasam]

This is now addressed

[Avasam]

This comment leads me to believe that the type here simply isn't complete (would require more work). Rather than being unrepresentable by the type system.

Suggested change

	import typing
	import grpc
	# XXX: don't yet know how to add a stub for google.rpc.status_pb2.Status
	# without affecting other stuff; may need to make a stub-only package for
	# google.rpc as well.
	# Returns a google.rpc.status.Status message corresponding to a given grpc.Call.
	def from_call(call: grpc.Call) -> typing.Any: ...
	# Convert a google.rpc.status.Status message to grpc.Status.
	def to_status(status: typing.Any) -> grpc.Status: ...
	from typing import Incomplete
	import grpc
	# Returns a google.rpc.status.Status message corresponding to a given grpc.Call.
	def from_call(call: grpc.Call) -> Incomplete: ...
	# Convert a google.rpc.status.Status message to grpc.Status.
	def to_status(status: Incomplete) -> grpc.Status: ...

Which can then be rewritten as

Suggested change

	import typing
	import grpc
	# XXX: don't yet know how to add a stub for google.rpc.status_pb2.Status
	# without affecting other stuff; may need to make a stub-only package for
	# google.rpc as well.
	# Returns a google.rpc.status.Status message corresponding to a given grpc.Call.
	def from_call(call: grpc.Call) -> typing.Any: ...
	# Convert a google.rpc.status.Status message to grpc.Status.
	def to_status(status: typing.Any) -> grpc.Status: ...
	import grpc
	# Returns a google.rpc.status.Status message corresponding to a given grpc.Call.
	def from_call(call: grpc.Call): ...
	# Convert a google.rpc.status.Status message to grpc.Status.
	def to_status(status) -> grpc.Status: ...

with an entry in pyrightconfig.stricter.json

And yeah, to complete this, you'd need a stub for google.rpc, wherever that comes from (if it's https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto, we have some tooling to generate stubs from proto files)

[Avasam]

I just realized, but aren't your top-level packages actually separate distributions?
Shouldn't these all be their own stubs rather than all lumped in stubs/grpcio ?

If someone installs grcpio, it doesn't come with, for example, the package grpc_health. You have to specifically install grpcio-health-checking from what I can see.

To avoid delaying this PR further I wouldn't mind splitting it up as a quick fix afterward. Just thought I'd at least mention it while I notice it.

grpc is a bit weird in that the source is a mono-repo and the distributions versions seems to be synchronized.

[macro1]

will we need to add types-grpcio to typeshed-internal/stub_uploader before we can split in that way?

[Avasam]

Stub-uploader automatically accepts dependencies on other typeshed stubs IIRC.

But as I mentioned, I don't mind doing that as a follow-up, and seeing if we hit any roadblocks there. Just so this PR doesn't get dragged on too much longer.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

spark (https://github.com/apache/spark)
- python/pyspark/sql/connect/utils.py:72: error: Cannot find implementation or library stub for module named "grpc_status"  [import-not-found]
- python/pyspark/sql/connect/client/reattach.py:30: error: Cannot find implementation or library stub for module named "grpc_status"  [import-not-found]
- python/pyspark/sql/connect/plan.py:656: error: Unused "type: ignore" comment  [unused-ignore]
- python/pyspark/sql/connect/client/core.py:59: error: Cannot find implementation or library stub for module named "grpc_status"  [import-not-found]

jax (https://github.com/google/jax)
+ jax/experimental/jax2tf/examples/serving/model_server_request.py:18: error: Unused "type: ignore" comment  [unused-ignore]