Refactor distributed_sampler update in data_connector.py
for distributed strategy
#12217
Labels
Milestone
data_connector.py
for distributed strategy
#12217
Uh oh!
There was an error while loading. Please reload this page.
Proposed refactor
This issue follows @ananthsub 's #11756 to move strategy-specific dataloader logic to the stategies.
Motivation
_prepare_dataloader()
function indata_connector.py
handles the logic whether the dataloader needs to update the distributed sampler when user is using distributed strategy (see the following code link)https://github.com/PyTorchLightning/pytorch-lightning/blob/cc43d07db1ab77385feff04c01f040c5cad805a9/pytorch_lightning/trainer/connectors/data_connector.py#L360-L372
However, modifying sampler in dataloader can be pushed directly to strategy class. Similar to issue #12216 , we can move the logic to
Strategy
base class. For any strategy classes haveis_distributed=True
flag, they can verify if the dataloader need to update the sampler.Pitch
We can move the logic to
Strategy
base class. For any strategy classes haveis_distributed=True
flag, they can verify if the dataloader need to update the sampler.Additional context
cc: @edward-io @four4fish @ananthsub
If you enjoy Lightning, check out our other projects! ⚡
Metrics: Machine learning metrics for distributed, scalable PyTorch applications.
Lite: enables pure PyTorch users to scale their existing code on any kind of device while retaining full control over their own loops and optimization logic.
Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, fine-tuning, and solving problems with deep learning.
Bolts: Pretrained SOTA Deep Learning models, callbacks, and more for research and production with PyTorch Lightning and PyTorch.
Lightning Transformers: Flexible interface for high-performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
cc @justusschock @awaelchli @rohitgr7 @ninginthecloud @otaj @tchaton @akihironitta @carmocca
The text was updated successfully, but these errors were encountered: