Skip to content

Follow-up changes to #10575 #10957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Dec 7, 2021
Merged

Follow-up changes to #10575 #10957

merged 10 commits into from
Dec 7, 2021

Conversation

carmocca
Copy link
Contributor

@carmocca carmocca commented Dec 6, 2021 •
edited by github-actions bot
Loading

What does this PR do?

Follow-up to #10575

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • [n/a] Did you make sure to update the documentation with your changes? (if necessary)
  • [n/a] Did you write any new necessary tests? (not for typos and docs)
  • [n/a] Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • [n/a] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @carmocca @awaelchli @Borda @ninginthecloud

@carmocca carmocca added bug Something isn't working hooks Related to the hooks API labels Dec 6, 2021
@carmocca carmocca added this to the 1.6 milestone Dec 6, 2021
@carmocca carmocca self-assigned this Dec 6, 2021
@carmocca carmocca mentioned this pull request Dec 6, 2021
Merged
12 tasks
daniellepintz
daniellepintz reviewed Dec 6, 2021
View reviewed changes
@four4fish four4fish mentioned this pull request Dec 6, 2021
Merged
12 tasks
carmocca
carmocca commented Dec 6, 2021
View reviewed changes
pytorch_lightning/loops/optimization/optimizer_loop.py
@@ -144,12 +144,10 @@ def closure(self, *args: Any, **kwargs: Any) -> ClosureResult:
)

if self._zero_grad_fn is not None:
with self._profiler.profile("zero_grad"):
Copy link
Contributor Author

@carmocca carmocca Dec 6, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we get rid of profile("training_step_and_backward") then we could remove the profiler reference from this class. Just pointing it out for a possible follow-up.

They are removed from here because each function will already profile them

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could remove it. PyTorch Profiler would track those operations already at the PyTorch level.

@carmocca carmocca marked this pull request as ready for review December 6, 2021 17:55
@mergify mergify bot added the has conflicts label Dec 6, 2021
@mergify mergify bot removed the has conflicts label Dec 6, 2021
@mergify mergify bot added the ready PRs ready to be merged label Dec 7, 2021
tchaton
tchaton approved these changes Dec 7, 2021
View reviewed changes
Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

pytorch_lightning/loops/optimization/optimizer_loop.py
@@ -144,12 +144,10 @@ def closure(self, *args: Any, **kwargs: Any) -> ClosureResult:
)

if self._zero_grad_fn is not None:
with self._profiler.profile("zero_grad"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could remove it. PyTorch Profiler would track those operations already at the PyTorch level.

@carmocca carmocca merged commit 99adc45 into master Dec 7, 2021
@carmocca carmocca deleted the refactor/follow-up-call-hook branch December 7, 2021 14:27
daniellepintz
daniellepintz reviewed Dec 7, 2021
View reviewed changes
pytorch_lightning/plugins/training_type/training_type_plugin.py
@@ -164,8 +164,7 @@ def optimizer_step(

def optimizer_zero_grad(self, current_epoch: int, batch_idx: int, optimizer: Optimizer, opt_idx: int) -> None:
"""Zeros all model parameter's gradients."""
model_ref = self.lightning_module
model_ref.optimizer_zero_grad(current_epoch, batch_idx, optimizer, opt_idx)
self.lightning_module.optimizer_zero_grad(current_epoch, batch_idx, optimizer, opt_idx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not _call_LM_hook?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it's called already with _call_ttp_hook.

https://github.com/PyTorchLightning/pytorch-lightning/blob/eec2bae6c984895d1d4df4e75cf628c58128928a/pytorch_lightning/loops/optimization/optimizer_loop.py#L403

No TTP subclasses this hook, so technically the optimizer_loop could call _call_LM_hook directly and this could be removed

daniellepintz
daniellepintz reviewed Dec 9, 2021
View reviewed changes
pytorch_lightning/trainer/trainer.py
@@ -1479,16 +1479,15 @@ def _call_callback_hooks(
hook_name: str,
*args: Any,
**kwargs: Any,
) -> Optional[Any]:
output = None
) -> None:
Copy link
Contributor

@daniellepintz daniellepintz Dec 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we returning None for callback hooks? I thought part of the motivation for #8506 was to allow returning something for callback hooks? i.e. 4) in the description

Copy link
Contributor Author

@carmocca carmocca Dec 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was described as a limitation, but it's not something we want to support with the refactor. However, the refactor allows for adding it easily when we want to.

daniellepintz added a commit to daniellepintz/pytorch-lightning that referenced this pull request Dec 22, 2021
@daniellepintz
follow ups to Lightning-AI#10957
5eab0de
@daniellepintz daniellepintz mentioned this pull request Dec 22, 2021
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniellepintz daniellepintz daniellepintz left review comments

@ananthsub ananthsub ananthsub approved these changes

@awaelchli awaelchli awaelchli approved these changes

@tchaton tchaton tchaton approved these changes

@justusschock justusschock justusschock approved these changes

@Borda Borda Awaiting requested review from Borda Borda is a code owner

@kaushikb11 kaushikb11 Awaiting requested review from kaushikb11

@rohitgr7 rohitgr7 Awaiting requested review from rohitgr7

@SeanNaren SeanNaren Awaiting requested review from SeanNaren

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

Assignees

@carmocca carmocca

Labels
bug Something isn't working hooks Related to the hooks API ready PRs ready to be merged
Projects
None yet
Milestone
pl:1.6
Development

Successfully merging this pull request may close these issues.
6 participants
@carmocca @ananthsub @awaelchli @tchaton @justusschock @daniellepintz