Update transformer_tutorial.py | Resolving issue #1778 #2402
Conversation
Add description for positional encoding calculation for Transformers
YoushaaMurhij
changed the title
✅ Deploy Preview for pytorch-tutorials-preview ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
github-actions
bot
added
question
intro
docathon-h1-2023
| # input length (in this case, ``10000``). Dividing this term by ``d_model`` scales | ||
| # the values to be within a reasonable range for the exponential function. | ||
| # The negative sign in front of the logarithm ensures that the values decrease exponentially. | ||
| # The reason for writing ``math.log(10000.0)`` instead of ``4`` in the code is to make it clear |
There was a problem hiding this comment.
I don't understand this comment. math.log(10000.0) is 9.2, not 4.
There was a problem hiding this comment.
Sorry, I removed the redundant description.
| # for positional encoding. The purpose of this calculation is to create | ||
| # a range of values that decrease exponentially. | ||
| # This allows the model to learn to attend to positions based on their relative distances. | ||
| # The ``math.log(10000.0)`` term in the exponent represents the maximum effective |
There was a problem hiding this comment.
I'm not sure this is correct, the maximum input length is max_len, not 10000. Am I missing something?
There was a problem hiding this comment.
The purpose of this value is to make the frequencies of the sine and cosine functions very large. This is important because it helps to ensure that the positional encodings are unique for each position in the sequence. Right?
There was a problem hiding this comment.
I think, I need to update this, too.
There was a problem hiding this comment.
Also, can you please make the description less lengthy and in a simpler language. Thank you!
|
@YoushaaMurhij please submit a PR updating the description as suggested. Tag @NicolasHug and @kit1980 to review your update. |
svekars
removed
question
intro
docathon-h1-2023
Add description for positional encoding calculation for Transformers
Fixes #1778
Description
Add explanation for calculating positional encoding for Transformers by taking the log first and then applying the exponential function.
Checklist
cc @suraj813