Optimize import paths for optional dependencies #18561

awaelchli · 2023-09-14T21:51:54Z

What does this PR do?

Re-enables typehints for loggers with type-checking guard
Mocks third-party packages that won't be importable for docs (sphinx implicitly sets TYPE_CHECKING=True and runs the code under the guard)
We can now include the missing XLA and DeepSpeed classes in our API page
No longer needed to guard availability inside the TYPE_CHECKING guard

Before submitting

Was this discussed/agreed via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

📚 Documentation preview 📚: https://pytorch-lightning--18561.org.readthedocs.build/en/18561/

cc @Borda @carmocca @justusschock @awaelchli

github-actions · 2023-09-14T21:52:26Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅
pl-cpu (macOS-12, pytorch, 3.11, 2.0)	success	✅
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0)	success	✅
pl-cpu (windows-2022, pytorch, 3.11, 2.0)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py, tests/tests_pytorch/conftest.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
[pytorch-lightning (GPUs) (testing Lightning	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=174745&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e)	success
[pytorch-lightning (GPUs) (testing PyTorch	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=174745&view=logs&jobId=3f274fac-2e11-54ca-487e-194c91f3ae9f)	success

These checks are required after the changes to src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py, tests/tests_pytorch/conftest.py, src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py.

🟢 fabric: Docs

Check ID	Status
docs-make (fabric, doctest)	success	✅
docs-make (fabric, html)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, docs/source-fabric/api/strategies.rst, docs/source-fabric/conf.py, .github/workflows/docs-build.yml.

🟢 pytorch_lightning: Docs

Check ID	Status
docs-make (pytorch, doctest)	success	✅
docs-make (pytorch, html)	success	✅

These checks are required after the changes to src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py, docs/source-pytorch/conf.py, docs/source-pytorch/extensions/logging.rst, docs/source-pytorch/visualize/supported_exp_managers.rst, .github/workflows/docs-build.yml.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅
fabric-cpu (macOS-12, fabric, 3.11, 2.0)	success	✅
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0)	success	✅
fabric-cpu (windows-2022, fabric, 3.11, 2.0)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/conftest.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
[lightning-fabric (GPUs) (testing Fabric	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=174747&view=logs&jobId=fe777007-6e77-5e50-6e71-fc5977ab193a)	success
[lightning-fabric (GPUs) (testing Lightning	latest)](https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=174747&view=logs&jobId=47e66f3c-897a-5428-da11-bf5c7745762e)	success

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, tests/tests_fabric/conftest.py.

🟢 lightning_app: Docs

Check ID	Status
docs-make (app, doctest)	success	✅
docs-make (app, html)	success	✅

These checks are required after the changes to .github/workflows/docs-build.yml.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.11)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.11)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.11)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.11)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.11)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.11)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.11)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.11)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.11)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.11)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.11)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.11)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.11)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.11)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.11)	success	✅

These checks are required after the changes to src/lightning/fabric/plugins/precision/deepspeed.py, src/lightning/fabric/plugins/precision/transformer_engine.py, src/lightning/fabric/strategies/deepspeed.py, src/lightning/fabric/strategies/xla.py, src/lightning/fabric/strategies/xla_fsdp.py, src/lightning/pytorch/loggers/comet.py, src/lightning/pytorch/loggers/mlflow.py, src/lightning/pytorch/loggers/wandb.py, src/lightning/pytorch/plugins/precision/deepspeed.py, src/lightning/pytorch/strategies/deepspeed.py, src/lightning/pytorch/strategies/xla.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

for more information, see https://pre-commit.ci

Borda

I have mixed feelings about it, some typing got improved but some checks were dropped =/

docs/source-fabric/conf.py

docs/source-pytorch/extensions/logging.rst

src/lightning/fabric/plugins/precision/deepspeed.py

awaelchli · 2023-09-15T09:41:58Z

I have mixed feelings about it, some typing got improved but some checks were dropped =/

I'm not removing any checks. The end result (docs) and typing information remains the same as before this PR. Whether to install the dependencies in the docs build is a different topic and can be discussed separately. This PR does not stay in the way on decisions there.

Borda

except the docs changes it LGTM

docs/source-fabric/conf.py

docs/source-pytorch/extensions/logging.rst

docs/source-pytorch/visualize/supported_exp_managers.rst

src/lightning/pytorch/plugins/precision/deepspeed.py

Co-authored-by: Jirka Borovec <[email protected]>

This reverts commit 838411f.

awaelchli added 4 commits September 14, 2023 21:56

mock third party packages for autodoc typehints

7cbd473

mock

3ac49e6

loggers

e6a6d2c

wip

5cc17a3

awaelchli requested review from carmocca, justusschock, williamFalcon, Borda, edenlightning and lantiga as code owners September 14, 2023 21:51

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Sep 14, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

e44bd8b

for more information, see https://pre-commit.ci

awaelchli added the fun Staff contributions outside working hours - to differentiate from the "community" label label Sep 14, 2023

awaelchli and others added 7 commits September 14, 2023 23:59

revert

14210f9

debug

ea98db7

Merge branch 'master' into docs/mock-third-party

d074423

comet

ea2f3d9

xla

2c0b5f6

[pre-commit.ci] auto fixes from pre-commit.com hooks

7903138

for more information, see https://pre-commit.ci

undo

7ad26a5

github-actions bot removed the fabric lightning.fabric.Fabric label Sep 14, 2023

fabric deepspeed

93319c6

github-actions bot added the fabric lightning.fabric.Fabric label Sep 14, 2023

awaelchli and others added 5 commits September 15, 2023 00:46

strategy

f42dd1b

update

0417f32

handle xla

01e8664

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b2a9c4

for more information, see https://pre-commit.ci

fix

011a77b

xlafsdp

45bb7bc

carmocca approved these changes Sep 14, 2023

View reviewed changes

mock for doctest

182090b

awaelchli requested a review from ethanwharris as a code owner September 14, 2023 23:35

github-actions bot added the ci Continuous Integration label Sep 14, 2023

awaelchli added 7 commits September 15, 2023 01:48

monkeypatch no longer required

29c2475

xla

26cab23

comet_ml is mocked, so we need to skip the doctest

4418b24

config

e6b31ed

update

2fd050e

tf engine

25da60a

fix

f749e88

awaelchli added this to the 2.1 milestone Sep 15, 2023

mock

a948e32

awaelchli changed the title ~~[WIP] Mock optional packages for autodoc typehints~~ Mock optional packages for autodoc typehints Sep 15, 2023

awaelchli added docs Documentation related refactor labels Sep 15, 2023

Borda reviewed Sep 15, 2023

View reviewed changes

docs/source-fabric/conf.py Show resolved Hide resolved

docs/source-pytorch/extensions/logging.rst Show resolved Hide resolved

src/lightning/fabric/plugins/precision/deepspeed.py Show resolved Hide resolved

awaelchli changed the title ~~Mock optional packages for autodoc typehints~~ Optimize import paths for optional dependencies Sep 15, 2023

awaelchli requested a review from Borda September 15, 2023 09:52

Borda approved these changes Sep 15, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Sep 15, 2023

awaelchli and others added 3 commits September 15, 2023 14:00

Update docs/source-fabric/conf.py

ef00d44

Co-authored-by: Jirka Borovec <[email protected]>

try Jirka's suggestion

838411f

Revert "try Jirka's suggestion"

a3ce7fb

This reverts commit 838411f.

awaelchli merged commit c1ee22a into master Sep 15, 2023

awaelchli deleted the docs/mock-third-party branch September 15, 2023 21:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize import paths for optional dependencies #18561

Optimize import paths for optional dependencies #18561

Uh oh!

awaelchli commented Sep 14, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Sep 14, 2023 •

edited

Loading

Uh oh!

Borda left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awaelchli commented Sep 15, 2023 •

edited

Loading

Uh oh!

Borda left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Optimize import paths for optional dependencies #18561

Optimize import paths for optional dependencies #18561

Uh oh!

Conversation

awaelchli commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

PR review

Uh oh!

github-actions bot commented Sep 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚡ Required checks status: All passing 🟢

Groups summary

Uh oh!

Borda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awaelchli commented Sep 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Borda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awaelchli commented Sep 14, 2023 •

edited

Loading

github-actions bot commented Sep 14, 2023 •

edited

Loading

awaelchli commented Sep 15, 2023 •

edited

Loading