4 x A100 performance issue

[danilabagroff]

Running llama-bench in the Docker container for the following models:

1. Qwen2.5-72B-Instruct-BF16
2. Llama-3.3-70B-Instruct-BF16
3. Llama-3.1-8B-Instruct-BF16

The results I've obtained are significantly different from my expectations. If so, where would you recommend I start troubleshooting?

Thank you!

(main) [email protected]:/usr/local/llama.cpp/bin$ ./llama-bench -m /workspace/Qwen2.5-72B-Instruct-BF16.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 1: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 2: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 3: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| qwen2 70B BF16                 | 135.43 GiB |    72.71 B | CUDA       |  99 |         pp512 |        550.86 ± 0.50 |
| qwen2 70B BF16                 | 135.43 GiB |    72.71 B | CUDA       |  99 |         tg128 |         10.79 ± 0.01 |

build: b95c8af (1)
(main) [email protected]:/usr/local/llama.cpp/bin$ ./llama-bench -m /workspace/Llama-3.3-70B-Instruct-BF16.gguf 
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 1: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 2: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 3: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| llama 70B BF16                 | 131.42 GiB |    70.55 B | CUDA       |  99 |         pp512 |        560.66 ± 0.40 |
| llama 70B BF16                 | 131.42 GiB |    70.55 B | CUDA       |  99 |         tg128 |         11.15 ± 0.01 |

build: b95c8af (1)
(main) [email protected]:/usr/local/llama.cpp/bin$ ./llama-bench -m /workspace/Llama-3.1-8B-Instruct-BF16.gguf 
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 1: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 2: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
  Device 3: NVIDIA A100-SXM4-80GB, compute capability 8.0, VMM: yes
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| llama 8B BF16                  |  14.96 GiB |     8.03 B | CUDA       |  99 |         pp512 |      4052.20 ± 89.93 |
| llama 8B BF16                  |  14.96 GiB |     8.03 B | CUDA       |  99 |         tg128 |         79.04 ± 0.13 |

build: b95c8af (1)