[docs] Regional compilation docs #11556
Conversation
| the repeated blocks of the provided `nn.Module`. | ||
|
|
||
| ```py | ||
| # Make sure you're on the latest `accelerate`: `pip install -U accelerate`. |
There was a problem hiding this comment.
Merge after accelerate new version is released this week.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| Enabling regional compilation might require simple yet intrusive changes to the | ||
| modeling code. However, 🤗 Accelerate provides a utility [`compile_regions()`](https://huggingface.co/docs/accelerate/main/en/usage_guides/compilation#how-to-use-regional-compilation) which automatically _only_ compiles | ||
| the repeated blocks of the provided `nn.Module`. |
There was a problem hiding this comment.
no we actually compile the rest of the model as well 😅 I found out in my post that some people thought only the encoder/decoder block will be compiled in regional, which is not true.
I changed the docs to be more explicit huggingface/accelerate#3572 (comment)
There was a problem hiding this comment.
👁️ But https://docs.pytorch.org/tutorials/recipes/regional_compilation.html suggests a completely different recipe no? No full compilation but only regional and I always thought that is what should be done.
What am I missing?
There was a problem hiding this comment.
regional compilation is simply: cut into regions and then compile those regions. I didn't compare the two approaches but I believe in the context of the pytorch tutorial they were simply trying to reduce cold start, not trying to keep inference optimized as well (they didn't benchamrk inference).
There was a problem hiding this comment.
So
- inference latency of compiling full model >= inference latency of regionally compiling repeated blocks + compiling additional blocks in a model
- cold start time of compiling full model >> cold start time of regionally compiling repeated blocks + compiling additional blocks in a model
Is my understanding right or is it still fragmented?
Do you think providing an option to NOT compile the rest of the blocks could still make sense?
There was a problem hiding this comment.
yes that is how it works !
Do you think providing an option to NOT compile the rest of the blocks could still make sense?
doesn't make sense for me personally, since you will miss on the tuning of the task-specific head. Do you have any specific cases where we don't want to compile the rest of the model ?
Co-authored-by: Ilyas Moutawwakil <[email protected]>
What does this PR do?
huggingface/accelerate#3529