Discussion: batching benchmark and improvement

[vansangpfiev]

Motivation

• AFAIK, we have already supported batching but do not have a benchmark for it yet. We should review the implementation to see if can do any improvements.

Discussion

Resources

[nguyenhoangthuan99]

Llama.cpp implementation:

• Server will gather requests (1 prompt/ 1 request) and build batch. Llama.cpp engine will process a whole batch at 1 step.
• llama.cpp engine doesn't support parallel prompts process. refer to this: llama : add batched inference endpoint to server ggml-org/llama.cpp#3478 (comment)

Cortex inplementation:

• Each request(1 prompt/ 1 request) sent to server will be prepared and add to task queue. There is a background process gathers prompts in the task queue, build batch and process batch then push result to output queue.
  => Current implementation of cortex llama.cpp can support batching but need to adjust some params to sync with latest llama.cpp implementation and add doc to Readme.md. Also a benchmark script test run batch to verify implementation is needed.

[nguyenhoangthuan99]

#168

Result when run script in 3090 Linux:
{'message': 'Model already loaded'}

Finished in 27.825968503952026 s
Total token: 6108
Throughput when run parallel: 219.50718441776795 tokens/s
############################
Finished in 38.07835125923157 s
Total token: 4966
Throughput when run in sequence: 130.4153104264477 tokens/s
###########################
--- 70.19260907173157 seconds ---