RoPE in float32 precision

[imoneoi]

Fixes #863

[esmeetu]

@imoneoi Many thanks for this PR.

@WoosukKwon Hope you can consider this PR, this will truely support codellama with float16 dtype, and truely support Codellama family.
And also solve many issues related with codellama gibberish output.

[esmeetu]

The changes indeed will affect other model's' performance. I think it's better to change kernel dynamically by 'base' value. If the model base is larger than 10000, we should use this PR strategy, otherwise as usual.

[WoosukKwon]

Hi @imoneoi, I've modified and closed the PR by mistake. I'm really sorry. Because the PR is closed, I don't have a permission to revert back your repo. Your original PR can be found here.

I created a new PR #1004, which uses FP32 for initializing RoPE but does not use FP32 in the kernel. I find this more aligned with the HF's implementation. If the precision issue persists after the PR, we will revisit your PR and modify the kernel. Thanks again for your issue and PR. For sure, you will be recognized as an author of #1004.

[imoneoi]

@WoosukKwon It doesn't matter. I re-opened the FP32 RoPE kernel PR in #1061. I'm doing an evaluation of code-llama, you might consider this if precision issues persist.