[Help Wanted] Why take the log function and then apply exp?

[Superhzf]

In line of code, you calculate positional encoding for Transformers by taking the log first and then apply the exponential function.

Would you please elaborate on why you do this instead of directly doing the calculation?

I'm aware that log transformation can make multiplication become addition, but it seems that this is not the case here.

cc @suraj813

[svekars]

@albanD can you respond to this?

[albanD]

Which log/exp are you talking about exactly?
If you mean this line

tutorials/beginner_source/transformer_tutorial.py

Line 115 in 33492c7

div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))

then I think this is just to make it easier when you multiply with the arange that has to be in the log space.

[YoushaaMurhij]

\assigntome

[HuangZurong]

@Superhzf explained in the blog https://medium.com/@hunter-j-phillips/positional-encoding-7a93db4109e6