attn_bias not aligned & some questions regarding float16

[MM-IR]

Hi,

1. When playing with MPT-7b models, I frequently meet the issues of "attn_bias not aligned", with tensor_parallel_size - 2, how do alleviate this issue?

2. Besides, I just find that your default model loading scripts load float16 versions, for fair evaluation, is it necessary to switch to float32?

Thanks very much in advance!

[WoosukKwon]

Hi @MM-IR, sorry for the very late response. I believe the bug you reported was fixed by #834.

For the second question, I think it's pretty common to use FP16 for evaluating models, because its impact on model accuracy is negligible. For example, HF open LLM leaderboard does not support FP32, but only supports FP16, BF16, and 4/8-bit quantized formats.