Skip to content

Fix spawn plugins not deleting temp checkpoint #10935

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 6, 2021
Merged

Fix spawn plugins not deleting temp checkpoint #10935

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
401847e
delete ddp spawn temp checkpoint
awaelchli Dec 3, 2021
642a21f
fix
awaelchli Dec 3, 2021
5b8bbbb
revert
awaelchli Dec 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
- Fixed an issue with item assignment on the logger on rank > 0 for those who support it ([#10917](https://github.com/PyTorchLightning/pytorch-lightning/pull/10917))


- Fixed an issue with `DDPSpawnPlugin` and related plugins leaving a temporary checkpoint behind ([#10934](https://github.com/PyTorchLightning/pytorch-lightning/pull/10934))


## [1.5.4] - 2021-11-30

Expand Down
1 change: 1 addition & 0 deletions pytorch_lightning/plugins/training_type/ddp_spawn.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,7 @@ def __recover_results_in_main_process(self, spawn_output: "_SpawnOutput", traine
spawn_output.last_path, map_location=(lambda storage, loc: storage)
)
self.lightning_module.load_state_dict(ckpt)
self.checkpoint_io.remove_checkpoint(spawn_output.last_path)

# get the `callback_metrics` and set it to the trainer
if is_overridden("get_from_queue", self.lightning_module):
Expand Down