How to implement accelerator support as a backend?

[taronaeo]

Hi everyone,

I'm trying to implement a zDNN (z-Deep Neural Network) backend for the IBM NNPA Accelerator in the Telum processors but I am unsure of how to start. This accelerator is a synchronous compute device that requires it's own ztensor data structure to run computation from/to.

I've tried taking references from the CPU and CUDA implementations but I am confused by which methods/functions are required to be implemented for an accelerator device.

It would be really helpful if someone could provide a "shopping cart" list of items to do, or maybe point me to a documentation that I may have missed. Thank you!

[zhouwg]

hello, glad to see you here. I think the following links can provide a useful reference for your needs:

https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-blas/ggml-blas.cpp

https://github.com/ggml-org/ggml/blob/master/examples/simple/simple-backend.cpp

https://github.com/ggml-org/ggml/tree/master/examples/simple

accordingly, there is a ggml-qnn backend might be another useful reference for your zDNN backend: when I started this ggml-qnn backend I had nothing knowledge about qnn and limited knowledge about ggml internal, so I'm sure you will be success with your zDNN backend.

at the same time, you can utilize a command line program to understand how a specified backend works(offload a specified ggml op to hardware accelerator or NPU through the highly-well designed ggml backend subsystem) and verify your zDNN backend accordingly:

https://github.com/kantv-ai/llama.cpp/blob/kantvai-ggmlqnn-npurpc/tests/ggml-qnn-ut.cpp

one more important thing: slaren is the author of the highly-well designed ggml backend subsystem, maybe you can ask him any related technical questions.

[taronaeo]

Hey @zhouwg, thanks for providing that useful list of references that I could use! It has been helpful referencing ggml-blas.cpp so far but I have a follow-up implementation question because zDNN is a little different where it requires it's down zTensor data format.

Is there a specific function that I should be calling/putting the tensor initialisation implementation such that ggml_tensor's data can be mapped to ztensor? There are a whole bunch of steps required to prepare the ztensor for computation and I can't seem to find a suitable function to have all these done.

The preparation steps include:

1. Keeping track of all pre-transform descriptors, descriptors, the actual zTensor itself and allocating the tensor storage i.e.,

zdnn_tensor_desc pre_tfm_desc, tfm_desc;
zdnn_tensor z_input1, z_input2, z_result;

void * z_input_storage1 = malloc(...);
void * z_input_storage2 = malloc(...);
void * z_result_stoage = malloc(...);

2. Load tensor data i.e.,

// set input tensor data from 0 to 127 sequentially and repeat
for (uint64_t i = 0; i < num_elements; i++) {
    ((float *)z_input_storage1)[i] = (float)(i & 0x7F);
    ((float *)z_input_storage2)[i] = (float)(i & 0x7F);
}

3. Initialize tensor descriptor with pre-transformed (original) shape information and transform i.e.,

zdnn_init_pre_transformed_desc(ZDNN_NHWC, type, &pre_tfm_desc, dim_n, dim_h, dim_w, dim_c);

zdnn_generate_transformed_desc(&pre_tfm_desc, &tfm_desc);

4. Initialize ztensors with the tensor descriptors i.e.,

zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_input1);
zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_input2);
zdnn_init_ztensor_with_malloc(&pre_tfm_desc, &tfm_desc, &z_result);

5. Convert tensor to supported format for zDNN operation i.e.,

zdnn_transform_ztensor(&z_input1, z_input_storage1);
zdnn_transform_ztensor(&z_input2, z_input_storage2);

6. Finally do the zDNN operation i.e.,

zdnn_add(&z_input_storage1, &z_input_storage2, &z_result);

I've investigated the .init_tensor mapping from ggml_backend_buffer_i (CUDA example) and it appears that it is just padding the tensor and zero-ing out the empty spaces to prevent NaN. But it doesn't seem to be the correct place to initialize the actual tensor.

Any pointers would be very much appreciated! :)

[zhouwg]

hello @taronaeo,

glad to see your technical discussion.

when I see this:

Is there a specific function that I should be calling/putting the tensor initialisation implementation such that ggml_tensor's data can be mapped to ztensor? There are a whole bunch of steps required to prepare the ztensor for computation and I can't seem to find a suitable function to have all these done.

I think following code snippet in function ggml_qnn_add() would be helpful for your zDNN backend:

QNN_VER_PTR(*tensor_0)->clientBuf = {src0->data,
                                             qnn_get_ggml_tensor_data_size(src0)};
        QNN_VER_PTR(*tensor_1)->clientBuf = {src1->data,
                                             qnn_get_ggml_tensor_data_size(src1)};
        QNN_VER_PTR(*tensor_2)->clientBuf = {dst->data,
                                             qnn_get_ggml_tensor_data_size(dst)};

        QNN_VER_PTR(*tensor_0)->dimensions = dimensions_input_0;
        QNN_VER_PTR(*tensor_0)->rank       = qnn_get_ggml_tensor_rank(src0);
        QNN_VER_PTR(*tensor_0)->dataType   = src0_qnn_type;
        QNN_VER_PTR(*tensor_1)->dimensions = dimensions_input_1;
        QNN_VER_PTR(*tensor_1)->rank       = qnn_get_ggml_tensor_rank(src1);
        QNN_VER_PTR(*tensor_1)->dataType   = src1_qnn_type;
        QNN_VER_PTR(*tensor_2)->dimensions = dimensions_output;
        QNN_VER_PTR(*tensor_2)->rank       = qnn_get_ggml_tensor_rank(dst);
        QNN_VER_PTR(*tensor_2)->dataType   = dst_qnn_type;

or you might be get some useful hints from function ggml_qnn_general_node in project ggml-qnn.

from my personal point of view, Qualcomm QNN SDK is a highly-well designed AI SDK and it matches well with highly-well designed and compact ggml machine learning library. I guess IBM's zDNN SDK also provide similar mechanism.

in the fact, we are trying to implement a concise(without much complex encapsulation) QNN backend for llama.cpp through the C API provided by Qualcomm QNN SDK, so domain developers can clearly know how data path works in this specified backend, how ggml tensors mapping to the QNN tensors and how QNN SDK handles the hardware acceleration.

I've investigated the .init_tensor mapping from ggml_backend_buffer_i (CUDA example) and it appears that it is just padding the tensor and zero-ing out the empty spaces to prevent NaN. But it doesn't seem to be the correct place to initialize the actual tensor.

.init_tensor is not actually used in a specified backend(I was also a bit confused in the early stage of ggml-qnn), we can also see this in Intel's sycl backend. generally speaking, we'll create/initialize a hardware-relative tensor in a specified hardware accelerated op function, pls refer to code snippet in function ggml_qnn_general_node in project ggml-qnn, this approach can also be found in Huawei's CANN backend. one more important thing, as a backend developer, we should provide a well-designed data structure to hold/save these hardware-relative resources and ensure no memory leak in a specified cleanup function.

summary as following for a specified ggml backend:

• follow the principle of highly-well designed ggml backend subsystem(this is template which can be found at specified backend from Intel/Huawei/Qualcomm or our ggml-qnn backend)
• mapping ggml tensors to hardware-relative tensors(key-point)
• provide a simple and functional data structure to manage hardware-relative resources(key-point)
• implement hardware accelerated op functions one by one(key-point)

hope this a bit useful for you and pls correct me if I made incorrect expression.

[taronaeo]

Hey @zhouwg! Sorry for my absence, been working on other tasks. It appears that this is exactly what I need to implement backend support for the zAIU/NNPA backend. I will be marking your reply as the answer, thank you very much!

[zhouwg]

you are welcome. your excellent work on IBM NNPA backend is impressive and we hope it can bee seen in the future.