SYCL: Add fp16 type support to unary op kernels

[qnixsynapse]

There are probably better ways to do this.

Need to disable fp16 support on devices which does not support fp16 in hardware.

Either we do this by checking if the build is compiled with GGML_SYCL_F16 compile flag and disable it in device_supports_op function or we add info about current hardware features and check using a function.

Need proper testing..

[NeoZhangJianyu]

I find there is no UT cases for FP16 be opened.
Could you change the UT cases and enable FP16 for the OPs?
And test them and make sure they are passed.

[qnixsynapse]

I find there is no UT cases for FP16 be opened. Could you change the UT cases and enable FP16 for the OPs? And test them and make sure they are passed.

I think there are UT cases that are present in test backend ops which was disabled at that time by me (in #12201 )

[NeoZhangJianyu]

OK, I suggest enabling them and use them test this PR.

[qnixsynapse]

OK, I suggest enabling them and use them test this PR.

Already enabled and tested.

[qnixsynapse]

It seems that in actual inference of a fp16 model(gemma 2 2B F16 in this case), the intermediate hidden embeddings are converted to fp32:

call ggml_sycl_add done
call ggml_sycl_rms_norm
call ggml_sycl_rms_norm done
call ggml_sycl_mul
call ggml_sycl_mul done
[SYCL] ggml_sycl_cpy: Tensor supplied: f32 to f16
[SYCL] ggml_sycl_cpy: Tensor supplied: f32 to f16
call ggml_sycl_tanh: DST Tensor type: f32 <-----------------------
call ggml_sycl_tanh done
ggml_sycl_op_soft_max: F32 mask
[SYCL] call ggml_sycl_dup
[SYCL] ggml_sycl_cpy: Tensor supplied: f32 to f32
[SYCL] call ggml_sycl_dup done
call ggml_sycl_rms_norm
call ggml_sycl_rms_norm done
call ggml_sycl_mul
call ggml_sycl_mul done
call ggml_sycl_add
call ggml_sycl_add done
call ggml_sycl_rms_norm
call ggml_sycl_rms_norm done
call ggml_sycl_mul
call ggml_sycl_mul done
call ggml_sycl_gelu: DST Tensor type: f32 <------------------------
call ggml_sycl_gelu done
call ggml_sycl_mul
call ggml_sycl_mul done
call ggml_sycl_rms_norm
call ggml_sycl_rms_norm done
call ggml_sycl_mul
call ggml_sycl_mul done
call ggml_sycl_add
call ggml_sycl_add done

So, there is no way to test the numerical stability of the fp16 operations with the exception of test-backend-ops:

  GELU(type=f16,ne_a=[128,2,2,2],v=0): call ggml_sycl_gelu: DST Tensor type: f16
call ggml_sycl_gelu done
OK
  GELU(type=f16,ne_a=[5,7,11,13],v=0): call ggml_sycl_gelu: DST Tensor type: f16
call ggml_sycl_gelu done
OK
  GELU(type=f32,ne_a=[128,2,2,2],v=0): call ggml_sycl_gelu: DST Tensor type: f32
call ggml_sycl_gelu done
OK
  GELU(type=f32,ne_a=[5,7,11,13],v=0): call ggml_sycl_gelu: DST Tensor type: f32
call ggml_sycl_gelu done
OK
TANH(type=f16,ne_a=[128,2,2,2],v=0): call ggml_sycl_tanh: DST Tensor type: f16
call ggml_sycl_tanh done
OK
  TANH(type=f16,ne_a=[5,7,11,13],v=0): call ggml_sycl_tanh: DST Tensor type: f16
call ggml_sycl_tanh done
OK
  TANH(type=f32,ne_a=[128,2,2,2],v=0): call ggml_sycl_tanh: DST Tensor type: f32
call ggml_sycl_tanh done
OK
  TANH(type=f32,ne_a=[5,7,11,13],v=0): call ggml_sycl_tanh: DST Tensor type: f32
call ggml_sycl_tanh done
OK

I am marking this PR "ready for review" for now to get some comments from others.

[NeoZhangJianyu]

It's OK!

Thank you!

[qnixsynapse]

It's OK!

Thank you!

Thank you! Lets wait for other's comment before we merge it.

[Rbiessy]

Either we do this by checking if the build is compiled with GGML_SYCL_F16 compile flag and disable it in device_supports_op function or we add info about current hardware features and check using a function.

Did we reach a conclusion on this question? I think we should not instantiate fp16 kernels if the user does not provide GGML_SYCL_F16.