Help with docker command to run llama3.1 8b Q6 quantised model with llama cpp to utilise GPU to the full potential

[vishnuthegeek]

I have a gpu machine Standard_NC24ads_A100_v4 with 80 GB of GPU Accelerator-Memory need help with running a llama3.1 8b Q6 quantised model which could utilise full potential of GPU machine and extract the best performance out of it. Could someone help me with the command below, and suggest changes to best utilise my GPU pls?

Current docker start command:
sudo docker run --gpus all -d --name project_name --network llama-net -p 8080:8080 -v /llama/models/llama3:/models --restart unless-stopped local/llama.cpp:server-cuda -m /models/Meta-Llama-3.1-8B-Instruct-Q6_K_L.gguf --port 8080 --host 0.0.0.0 --n-gpu-layers 70 -c 16240 --parallel 5 -n 4096 -t 80 --batch-size 4096 --ubatch-size 1024

[vishnuthegeek]

Any help would be much appreciated.