Provide intra-fit() ckpt_path access to enable additional finetuning modalities #10198

speediedan · 2021-10-27T20:22:14Z

What does this PR do?

This PR includes a few minor changes that enable callbacks to restore checkpoints iteratively within the context of Trainer.fit().

An example use case that depends on access to ckpt_path (fit_ckpt_path) within fit() is available in a notebook-based tutorial (lightning-tutorials PR).

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Yes :)
Make sure you had fun coding 🙃

…pytorch-lightning#10198

speediedan · 2021-12-03T04:22:58Z

pytorch_lightning/callbacks/finetuning.py

    @staticmethod
-    def __apply_mapping_to_param_groups(param_groups: List[Dict[str, Any]], mapping: dict) -> List[Dict[str, Any]]:
+    def _apply_mapping_to_param_groups(param_groups: List[Dict[str, Any]], mapping: dict) -> List[Dict[str, Any]]:


Requesting this method be changed to protected instead of private since it has proved useful for FinetuningScheduler and I can conceive of it being useful in other contexts.

speediedan · 2021-12-03T04:38:42Z

pytorch_lightning/trainer/trainer.py

+        self.fit_ckpt_path = self.__set_ckpt_path(
+            ckpt_path, model_provided=model, model_connected=self.lightning_module is not None
+        )


This attribute is needed for FinetuningScheduler and I think it could make sense for other potential extensions as well.

speediedan · 2021-12-03T04:42:42Z

tests/callbacks/test_finetuning_callback.py

-            # add trivial test parameter to convblock to validate parent (non-leaf) module parameter handling
-            self.parent_param = nn.Parameter(torch.zeros((1), dtype=torch.float))
-            self.bn = nn.BatchNorm2d(out_channels)
+class ConvBlockParam(nn.Module):


FinetuningScheduler uses both ConvBlockParam and ConvBlock for testing and I think it's sufficiently useful elsewhere to have the two classes defined outside the complex_model test.

… and added new *.yaml files to manifest

…_theme via PR Lightning-Universe/lightning_sphinx_theme#41 , (pip install -r requirements/doc.txt to test)

…ver installing packages unrequired outside of submodule usage)

…ria, user-specified epoch transitions or a composition of the two (the default mode). Also enables parameters to be fully specified or selected via regex.

…st_finetuningscheduling_epoch_trans_only

…tribute and EarlyStopping modification. FTSEarlyStopping extension of EarlyStopping added. FTS dependencies managed by a callbackresolvermixin

…n to move it to the hub. leaving the PR open for a few minor changes required to support user registered callbacks like the finetuning_scheduler

speediedan · 2022-02-01T20:55:22Z

since the scope of this PR has changed so much, to maximize clarity I've opened a replacement PR

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch 2 times, most recently from a2803c0 to 4c4c747 Compare October 28, 2021 00:45

speediedan changed the title ~~Feature/10197 finetuning scheduler callback [WIP]~~ Feature/10197 finetuning scheduler callback Oct 28, 2021

speediedan marked this pull request as ready for review October 28, 2021 01:36

speediedan requested review from awaelchli, Borda, carmocca, edenlightning, justusschock, kaushikb11, rohitgr7, SeanNaren, tchaton and williamFalcon as code owners October 28, 2021 01:36

speediedan changed the title ~~Feature/10197 finetuning scheduler callback~~ Feature/10197 finetuning scheduler callback [WIP] Oct 28, 2021

speediedan changed the title ~~Feature/10197 finetuning scheduler callback [WIP]~~ Feature/10197 finetuning scheduler callback Oct 28, 2021

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from d7da388 to ef67620 Compare October 28, 2021 19:36

speediedan added a commit to speediedan/lightning_sphinx_theme that referenced this pull request Oct 29, 2021

side-by-side code and improved footnote support used by Lightning-AI/…

ee69351

…pytorch-lightning#10198

speediedan mentioned this pull request Oct 29, 2021

Enable Side-by-Side Code Comparison and Improved Footnote Handling Lightning-Universe/lightning_sphinx_theme#41

Merged

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from ef67620 to 349f73f Compare October 29, 2021 19:33

speediedan added a commit to speediedan/lightning_sphinx_theme that referenced this pull request Oct 31, 2021

side-by-side code and improved footnote support used by Lightning-AI/…

db20814

…pytorch-lightning#10198

speediedan added a commit to speediedan/lightning_sphinx_theme that referenced this pull request Oct 31, 2021

side-by-side code and improved footnote support used by Lightning-AI/…

5780c2b

…pytorch-lightning#10198

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from 349f73f to f302153 Compare October 31, 2021 03:00

speediedan added a commit to speediedan/lightning_sphinx_theme that referenced this pull request Oct 31, 2021

side-by-side code and improved footnote support used by Lightning-AI/…

79da83e

…pytorch-lightning#10198

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from f302153 to 1c77531 Compare October 31, 2021 03:22

mergify bot added the has conflicts label Nov 1, 2021

tchaton added this to the v1.6 milestone Nov 1, 2021

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from 1c77531 to 65fb438 Compare November 1, 2021 20:03

mergify bot removed the has conflicts label Nov 1, 2021

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from 65fb438 to e26b11f Compare November 3, 2021 16:39

speediedan commented Dec 3, 2021

View reviewed changes

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from f3a478e to ee9c4c8 Compare December 7, 2021 20:25

mergify bot added the has conflicts label Dec 9, 2021

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from ee9c4c8 to 5064ddb Compare December 12, 2021 23:38

mergify bot removed the has conflicts label Dec 12, 2021

mergify bot added the has conflicts label Jan 12, 2022

speediedan added 18 commits January 27, 2022 16:20

first public fts commit

d63d225

fix to enable fts when resume_from_checkpoint deprecated

2fe65f6

added fts loss animation to doc

656a8f5

updated changelog with PR#

60be6e9

fixed fts_supporters reference and made schedule tests 3.7 compatible…

5f618cc

… and added new *.yaml files to manifest

changed boring_ft_schedule fixture scope to address env variable leakage

8ab52e3

move documentation image files to fts subfolder

42fc63a

add fts_superglue pl_example graceful exit and horovod test condition

7b9f448

fix deepspeed compat test after strategy flag change

d7796a5

removed main.css changes, enabling use of updated pt_lightning_sphinx…

e1c7b4d

…_theme via PR Lightning-Universe/lightning_sphinx_theme#41 , (pip install -r requirements/doc.txt to test)

adjust for callback_connector protection

89f1101

update lightning_sphinx_theme PR commit sha in requirements/docs.txt

d7d65c1

keep notebooks submodule requirements reference (choosing pip error o…

3a0d20b

…ver installing packages unrequired outside of submodule usage)

Finetuning phase transitions can now be driven by EarlyStopping crite…

c83f3a3

…ria, user-specified epoch transitions or a composition of the two (the default mode). Also enables parameters to be fully specified or selected via regex.

refactor test_finetuning_scheduler_callback.py tests, particularly te…

3ae6c4b

…st_finetuningscheduling_epoch_trans_only

refactor finetuningscheduler to remove core pl trainer convenience at…

47c71a9

…tribute and EarlyStopping modification. FTSEarlyStopping extension of EarlyStopping added. FTS dependencies managed by a callbackresolvermixin

last commit prior to documentation/pl_example porting

3a968b7

remove fts callback, associated tests and documentation in preparatio…

30a985a

…n to move it to the hub. leaving the PR open for a few minor changes required to support user registered callbacks like the finetuning_scheduler

speediedan force-pushed the feature/10197_finetuning_scheduler_callback branch from 5064ddb to 30a985a Compare January 28, 2022 00:27

mergify bot removed the has conflicts label Jan 28, 2022

speediedan changed the title ~~Feature/10197 finetuning scheduler callback~~ Provide intra-fit() ckpt_path access to enable additional finetuning modalities Feb 1, 2022

speediedan closed this Feb 1, 2022

speediedan deleted the feature/10197_finetuning_scheduler_callback branch May 12, 2022 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Provide intra-fit() ckpt_path access to enable additional finetuning modalities #10198

Provide intra-fit() ckpt_path access to enable additional finetuning modalities #10198

Uh oh!

speediedan commented Oct 27, 2021 •

edited

Loading

Uh oh!

speediedan Dec 3, 2021

Uh oh!

speediedan Dec 3, 2021

Uh oh!

speediedan Dec 3, 2021

Uh oh!

speediedan commented Feb 1, 2022

Uh oh!

Uh oh!

Provide intra-fit() ckpt_path access to enable additional finetuning modalities #10198

Provide intra-fit() ckpt_path access to enable additional finetuning modalities #10198

Uh oh!

Conversation

speediedan commented Oct 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

Uh oh!

speediedan Dec 3, 2021

Choose a reason for hiding this comment

Uh oh!

speediedan Dec 3, 2021

Choose a reason for hiding this comment

Uh oh!

speediedan Dec 3, 2021

Choose a reason for hiding this comment

Uh oh!

speediedan commented Feb 1, 2022

Uh oh!

Uh oh!

speediedan commented Oct 27, 2021 •

edited

Loading