Performance Tuning Guide is very out of date

[msaroufim]

🚀 Descirbe the improvement or the new tutorial

The first thing you see when you Google PyTorch performance is this. The recipe is well written but it's very much out of data today
https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html

Some concrete things we should fix

1. For fusions we should talk about torch.compile instead of jit.script
2. We should mention overhead reduction with cudagraphs
3. We should talk about the *-fast series as places people can learn more
4. For CPU specific optimization the most important one is launcher core pinning so we should either make that a default or explain the point more
5. Instead of the CPU section we can instead go more into the inductor CPU backend
6. AMP section is fine but maybe expand to quantization
7. DDP section needs to be moved somewhere else with some FSDP performance guide
8. GPU sync section is good
9. Mention tensor cores and how to enable them and why they're not enabled by default

cc @sekyondaMeta @svekars @kit1980 @drisspg who first made me aware of this with an internal note that was important enough to make public

Existing tutorials on this topic

No response

Additional context

No response

[svekars]

Related: #2695

[orion160]

It's pretty interesting. I'll go over a PR on this days updating it!

[msaroufim]

feel free to ping me here or on https://discord.gg/FBMQJQJn whenever you're ready for a review

[orion160]

/assigntome

[orion160]

@msaroufim What are the *-fast series?

[orion160]

[X] Ops fusion with torch.compile -> done by @desertfire
[] CUDA graphs
[] *-fast -> ???
[] core pinning -> Already in the doc, put a minor section explaining it
[] inductor CPU backend -> ???
[] Tensor core explanation

[msaroufim]

Hi @orion160 what I was referring to was https://pytorch.org/blog/accelerating-generative-ai-2/ and https://pytorch.org/blog/accelerating-generative-ai

[orion160]

Mmmmm I read it, I could mention it but it feels a bit out of place... The tuning perf leverages high level tweaks and these blog entries leverages optimizations in an adhoc model centric way

[orion160]

It could be like an epilogue mentioning these case studies.

@msaroufim Do you think with that changes the issue can be complete?