Finetuning scheduler

lightning_examples/finetuning-scheduler/.meta.yml

@@ -0,0 +1,21 @@
+title: Finetuning Scheduler
+author: Dan Dale
+created: 2021-11-29
+updated: 2021-11-29
+license: CC BY-SA
+build: 1
+tags:
+  - finetuning
+description: |
+  This notebook introduces the FinetuningScheduler callback and demonstrates the use of FinetuningScheduler to finetune
+  a small foundational model on the [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task
+  of [SuperGLUE](https://super.gluebenchmark.com/) with iterative early-stopping defined according to a user-specified
+  schedule. It uses HuggingFace's ``datasets`` and ``transformers`` libraries to retrieve the relevant benchmark data
+  and foundational model weights.
+requirements:
+  - transformers
+  - datasets
+  - scikit-learn
+  - sentencepiece
+accelerator:
+  - GPU

lightning_examples/finetuning-scheduler/RteBoolqModule_ft_schedule_deberta_base.yaml

@@ -0,0 +1,18 @@
+
+0:
+  params:
+  - model.classifier.bias
+  - model.classifier.weight
+  - model.pooler.dense.bias
+  - model.pooler.dense.weight
+  - model.deberta.encoder.LayerNorm.bias
+  - model.deberta.encoder.LayerNorm.weight
+  - model.deberta.encoder.rel_embeddings.weight
+  - model.deberta.encoder.layer.{0,11}.(output|attention|intermediate).*
+1:
+  params:
+  - model.deberta.embeddings.LayerNorm.bias
+  - model.deberta.embeddings.LayerNorm.weight
+2:
+  params:
+  - model.deberta.embeddings.word_embeddings.weight

lightning_examples/text-transformers/text-transformers.py

@@ -220,7 +220,7 @@ def setup(self, stage=None) -> None:
         if stage != "fit":
             return
         # Get dataloader by calling it - train_dataloader() is called after setup() by default
-        train_loader = self.trainer.datamodule.train_dataloader()
+        train_loader = self.train_dataloader()
 
         # Calculate total steps
         tb_size = self.hparams.train_batch_size * max(1, self.trainer.gpus)