write _shard_map; refactor flash attention to support 5d inputs. #8730
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qihqi
force-pushed
the
hanq_shard_map
branch
from
d5c7ae7 to
af2bbf6
Compare
pgmoka
reviewed
Feb 24, 2025
qihqi
force-pushed
the
hanq_shard_map
branch
2 times, most recently
from
0a27ca4 to
8a5a29f
Compare
qihqi
changed the title
qihqi
force-pushed
the
hanq_shard_map
branch
2 times, most recently
from
d885e7b to
8983df7
Compare
tengyifei
requested changes
Feb 28, 2025
pgmoka
reviewed
Mar 1, 2025
Comment on lines
+91
to
+94
| q = torch.randn(4, 2, 2, 128, 4).to("xla") | ||
| k = torch.randn(4, 2, 2, 128, 4).to("xla") | ||
| v = torch.randn(4, 2, 2, 128, 4).to("xla") |
There was a problem hiding this comment.
In test_flash_attention_spmd_data_parallel, , (8, 2, 128, 8) is used. Would the equivalent here be something like (8, 2, 2, 128, 8)?
There was a problem hiding this comment.
(8, 2, 2, 128, 8) would also work.
In general, the axis to shard has to be multiple of number of devices in this axis.
In this case the mesh is (n_devices // 2, 2, 1,1,1) so it's 4,2, 1,1,1; therefore having 4,2 as leading dim is sufficient.
For TPUs with 4 devices it would be 2,2,1,1,1 which 4,2 is still multiple-of. Therefore this configuration is general.
tengyifei
approved these changes
Mar 1, 2025
|
Still need to fix TPU test which looks relevant |
pgmoka
pushed a commit
that referenced
this pull request
Mar 5, 2025
pgmoka
pushed a commit
that referenced
this pull request
Mar 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jax's shard_map works by enabling manual sharding on inputs, and disabling it on outputs.
here we introduce _shard_map to simulate this behavior. This is sufficient to support use cases of calling pallas.
It is not sufficient to support other use cases of shard_map, such as desire to use manual collectives. Because current collective implementation, such as
xm.all_gatherchecks fail with manual sharding.