-
Notifications
You must be signed in to change notification settings - Fork 3.5k
[wip] Support serialized checkpoint loading #9585
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[wip] Support serialized checkpoint loading #9585
Conversation
for more information, see https://pre-commit.ci
pytorch_lightning/plugins/training_type/training_type_plugin.py
Outdated
Show resolved
Hide resolved
if self._ckpt_path: | ||
self._load_checkpoint_weights() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note to reviewer: this must be moved after the setup_environment
is called to ensure that torch distributed is initialized. it also matches the symmetry for loading checkpoints after pre-dispatch, and makes it easier for us to reason about merging these checkpoint loading paths altogether.
for more information, see https://pre-commit.ci
* Document exceptions in ipu.py * Document exceptions in tpu.py * Document exceptions in gpu.py
…tly into `trainer.py` (Lightning-AI#9495) Co-authored-by: Adrian Wälchli <[email protected]> Co-authored-by: thomas chaton <[email protected]>
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…iferdai/pytorch-lightning into serialized-load-checkpoint
messed up git, moving to #9605 |
What does this PR do?
Fixes #9406
Does your PR introduce any breaking changes? If yes, please list them.
Before submitting
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:
Did you have fun?
Make sure you had fun coding 🙃