Skip to content

Commit 6f1f81d

Browse files
ejguanfacebook-github-bot
authored andcommitted
Update tutorial about placing sharding_filter (#487)
Summary: See the feedback from a user: #454 (comment) We should explicitly ask users to place `sharding_filter` as early as possible. Pull Request resolved: #487 Reviewed By: wenleix Differential Revision: D36812259 Pulled By: ejguan fbshipit-source-id: 4c983f3216a80be398f85b20871e65b0e41627e0
1 parent cd38927 commit 6f1f81d

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

docs/source/tutorial.rst

+5
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,11 @@ When we re-run, we will get:
176176
...
177177
n_sample = 6
178178
179+
Note:
180+
181+
- Place ``ShardingFilter`` (``datapipe.sharding_filter``) as early as possible in the pipeline, especially before expensive
182+
operations such as decoding, in order to avoid repeating these expensive operations across worker/distributed processes.
183+
179184

180185
You can find more DataPipe implementation examples for various research domains `on this page <torchexamples.html>`_.
181186

0 commit comments

Comments
 (0)