Skip to content

Commit b9ea7db

Browse files
kunal-vaishnavikunal-vaishnavi
authored
Update E2E numbers (#396)
### Description This PR updates the end-to-end benchmarking numbers for LLaMA-2. ### Motivation and Context The numbers were gathered with the end-to-end benchmarking script in [this PR](microsoft/onnxruntime#19985).
1 parent 9709f61 commit b9ea7db

File tree

3 files changed

+145
-126
lines changed

3 files changed

+145
-126
lines changed

‎python/models/llama2/LLaMA-2 E2E Notebook.ipynb ‎python/models/llama/LLaMA-2 E2E Notebook.ipynb

File renamed without changes.

‎python/models/llama/README.md

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# LLaMA-2
2+
3+
This folder contains a Jupyter notebook that demonstrates how to export, optimize, and run the LLaMA-2 model with ONNX Runtime. For more details, please see the notebook and the [ORT README](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/llama/README.md).
4+
5+
## LLaMA-2 7B FP16 CUDA (1 A100 80GB)
6+
7+
| Engine | Batch Size | Prompt Length | Prompt Processing Latency (ms) | Prompt Processing Throughput (tps) | Average Latency of First 128 Tokens Generated (ms) | Average Throughput of First 128 Tokens Generated (tps) | Average Latency of First 256 Tokens Generated (ms) | Average Throughput of First 256 Tokens Generated (tps) | Wall-Clock Latency (s) | Wall-Clock Throughput (tps) |
8+
|-----------------|----|------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
9+
| onnxruntime | 1 | 16 | 11.967659 | 1336.936489 | 10.52479073 | 95.01376562 | 10.54278947 | 94.85155731 | 3.08197999 | 88.25495327 |
10+
| onnxruntime | 1 | 64 | 12.41350174 | 5155.676564 | 10.51662862 | 95.08750721 | 10.55776421 | 94.71702343 | 3.122560978 | 104.3349877 |
11+
| onnxruntime | 1 | 256 | 22.4044323 | 11426.3105 | 10.7493531 | 93.02885402 | 10.78576129 | 92.71482774 | 3.139767647 | 163.0693916 |
12+
| onnxruntime | 1 | 1024 | 75.05702972 | 13642.95928 | 11.31167263 | 88.40425575 | 11.34056505 | 88.17902774 | 3.332163334 | 384.1348313 |
13+
| onnxruntime | 1 | 2048 | 135.2889538 | 15137.96909 | 12.08372787 | 82.75591863 | 12.11640146 | 82.53275559 | 3.582954168 | 643.044787 |
14+
| onnxruntime | 1 | 3840 | 251.5854836 | 15263.20178 | 13.44519481 | 74.37601419 | 13.48242071 | 74.17065688 | 4.047522068 | 1011.977188 |
15+
| onnxruntime | 4 | 16 | 12.75753975 | 5016.641238 | 10.92023589 | 366.2924539 | 10.99625602 | 363.7601736 | 3.188626289 | 341.2127673 |
16+
| onnxruntime | 4 | 64 | 22.7124691 | 11271.3417 | 11.15260646 | 358.6605531 | 11.19375136 | 357.3422236 | 3.256895304 | 393.0123264 |
17+
| onnxruntime | 4 | 256 | 73.77910614 | 13879.26818 | 11.26689278 | 355.0224609 | 11.35130133 | 352.3825051 | 3.345386028 | 612.186451 |
18+
| onnxruntime | 4 | 1024 | 250.616312 | 16343.7087 | 12.52830587 | 319.2770068 | 12.6034962 | 317.3722542 | 3.847688437 | 1330.669072 |
19+
| onnxruntime | 4 | 2048 | 506.0505867 | 16188.10494 | 14.06471804 | 284.3995869 | 14.14138451 | 282.8577355 | 4.497682095 | 2049.055448 |
20+
| onnxruntime | 4 | 3840 | 978.5776138 | 15696.25115 | 16.76318049 | 238.6182026 | 16.83990005 | 237.531101 | 5.664571524 | 2892.363514 |
21+
| onnxruntime | 16 | 16 | 21.32916451 | 12002.34542 | 11.54885069 | 1385.419245 | 11.97430678 | 1336.194261 | 3.479871035 | 1250.621059 |
22+
| onnxruntime | 16 | 64 | 73.28677177 | 13972.50793 | 11.71741821 | 1365.488516 | 12.04443816 | 1328.413977 | 3.52155304 | 1453.903986 |
23+
| onnxruntime | 16 | 256 | 248.3313084 | 16494.09423 | 12.81819306 | 1248.225855 | 13.13442457 | 1218.172894 | 3.978744745 | 2058.940828 |
24+
| onnxruntime | 16 | 1024 | 975.6298065 | 16793.25487 | 16.74189232 | 955.6864715 | 17.06122886 | 937.7988026 | 5.703416586 | 3590.830109 |
25+
| onnxruntime | 16 | 2048 | 1993.696928 | 16435.79801 | 22.16357179 | 721.9053026 | 22.49017637 | 711.4217219 | 8.114635229 | 4542.902911 |
26+
| onnxruntime | 16 | 3840 | 3924.712181 | 15654.65113 | 31.63040616 | 505.8423822 | 31.95275087 | 500.7393594 | 12.46947217 | 5255.715648 |
27+
| pytorch-eager | 1 | 16 | 32.97473 | 485.2201 | 31.95276 | 31.2962 | 31.8423 | 31.40477 | 8.28506 | 32.83018 |
28+
| pytorch-eager | 1 | 64 | 32.63447 | 1961.117 | 31.33203 | 31.91622 | 31.36941 | 31.87819 | 8.164876 | 39.19227 |
29+
| pytorch-eager | 1 | 256 | 34.46941 | 7426.875 | 31.69294 | 31.55277 | 31.53167 | 31.71414 | 8.207787 | 62.37979 |
30+
| pytorch-eager | 1 | 1024 | 103.928 | 9852.975 | 31.84283 | 31.40424 | 31.80877 | 31.43787 | 8.408238 | 152.2317 |
31+
| pytorch-eager | 1 | 2048 | 244.3801 | 8380.386 | 32.11394 | 31.13912 | 32.11288 | 31.14015 | 8.720115 | 264.2167 |
32+
| pytorch-eager | 1 | 3840 | 611.0726 | 6284.032 | 32.04668 | 31.20448 | 32.02001 | 31.23048 | 9.293344 | 440.7455 |
33+
| pytorch-eager | 4 | 16 | 32.7481 | 1954.312 | 31.60442 | 126.5646 | 31.45407 | 127.1696 | 8.18083 | 132.9938 |
34+
| pytorch-eager | 4 | 64 | 33.18802 | 7713.626 | 31.20292 | 128.1931 | 31.26663 | 127.9319 | 8.132635 | 157.3906 |
35+
| pytorch-eager | 4 | 256 | 89.22571 | 11476.51 | 31.29607 | 127.8116 | 31.29617 | 127.8111 | 8.248695 | 248.2817 |
36+
| pytorch-eager | 4 | 1024 | 392.79 | 10427.96 | 31.26839 | 127.9247 | 31.22812 | 128.0897 | 8.707226 | 588.0174 |
37+
| pytorch-eager | 4 | 2048 | 955.0025 | 8577.988 | 31.27921 | 127.8805 | 31.28768 | 127.8458 | 9.992102 | 922.3284 |
38+
| pytorch-eager | 4 | 3840 | 2467.054 | 6226.05 | 31.35273 | 127.5806 | 31.33206 | 127.6647 | 15.97773 | 1025.427 |
39+
| pytorch-eager | 16 | 16 | 33.24208 | 7701.083 | 31.49257 | 508.0563 | 32.28204 | 495.6316 | 8.396241 | 518.3272 |
40+
| pytorch-eager | 16 | 64 | 86.10473 | 11892.49 | 31.38509 | 509.7962 | 32.1652 | 497.432 | 8.467185 | 604.6874 |
41+
| pytorch-eager | 16 | 256 | 332.7774 | 12308.53 | 32.50902 | 492.171 | 32.52583 | 491.9167 | 8.955728 | 914.7219 |
42+
| pytorch-eager | 16 | 1024 | 1543.535 | 10614.59 | 33.13551 | 482.8656 | 33.08991 | 483.5311 | 16.21622 | 1262.933 |
43+
| pytorch-eager | 16 | 2048 | 3856.058 | 8497.797 | 33.05266 | 484.0761 | 33.0516 | 484.0915 | 28.2955 | 1302.822 |
44+
| pytorch-eager | 16 | 3840 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
45+
| pytorch-compile | 1 | 16 | 12.97314 | 1233.317 | 15.06754 | 66.36782 | 14.94154 | 66.9275 | 3.949777 | 68.86465 |
46+
| pytorch-compile | 1 | 64 | 13.30206 | 4811.285 | 14.92108 | 67.01927 | 14.79991 | 67.56797 | 3.913011 | 81.77846 |
47+
| pytorch-compile | 1 | 256 | 21.05542 | 12158.39 | 14.74479 | 67.82056 | 14.73028 | 67.88739 | 3.938656 | 129.9936 |
48+
| pytorch-compile | 1 | 1024 | 75.77764 | 13513.22 | 14.73202 | 67.87934 | 14.64023 | 68.30496 | 4.274032 | 299.483 |
49+
| pytorch-compile | 1 | 2048 | 159.0262 | 12878.38 | 14.77194 | 67.69592 | 14.69836 | 68.0348 | 5.412601 | 425.6734 |
50+
| pytorch-compile | 1 | 3840 | 339.8554 | 11298.92 | 14.06384 | 71.10431 | 14.04383 | 71.20565 | 7.099414 | 576.949 |
51+
| pytorch-compile | 4 | 16 | 14.82386 | 4317.365 | 14.85815 | 269.2126 | 14.85617 | 269.2483 | 3.927973 | 276.9877 |
52+
| pytorch-compile | 4 | 64 | 20.7674 | 12327.01 | 14.78437 | 270.556 | 14.80803 | 270.1237 | 3.955819 | 323.574 |
53+
| pytorch-compile | 4 | 256 | 70.34887 | 14556.03 | 14.9404 | 267.7305 | 14.9841 | 266.9496 | 4.313815 | 474.7537 |
54+
| pytorch-compile | 4 | 1024 | 290.767 | 14086.88 | 15.47449 | 258.49 | 15.64255 | 255.7128 | 6.704303 | 763.6886 |
55+
| pytorch-compile | 4 | 2048 | 644.6995 | 12706.69 | 17.22421 | 232.2313 | 17.09852 | 233.9384 | 10.17314 | 905.9152 |
56+
| pytorch-compile | 4 | 3840 | 1488.37 | 10320.01 | 17.54926 | 227.9298 | 16.19208 | 247.0344 | 16.26342 | 1007.414 |
57+
| pytorch-compile | 16 | 16 | 20.62188 | 12414 | 16.14968 | 990.7319 | 17.28572 | 925.6194 | 5.190115 | 838.5171 |
58+
| pytorch-compile | 16 | 64 | 68.86672 | 14869.3 | 15.93814 | 1003.881 | 17.00272 | 941.0257 | 5.524729 | 926.7423 |
59+
| pytorch-compile | 16 | 256 | 262.5498 | 15600.85 | 16.28529 | 982.4817 | 18.87143 | 847.8423 | 7.905223 | 1036.277 |
60+
| pytorch-compile | 16 | 1024 | 1134.517 | 14441.39 | 19.28937 | 829.4722 | 20.54817 | 778.6581 | 16.55617 | 1237.001 |
61+
| pytorch-compile | 16 | 2048 | 3682.501 | 8898.3 | 32.4632 | 492.8657 | 32.31265 | 495.1621 | 28.07167 | 1313.21 |
62+
| pytorch-compile | 16 | 3840 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
63+
64+
## LLaMA-2 13B FP16 CUDA (1 A100 80GB)
65+
66+
| Engine | Batch Size | Prompt Length | Prompt Processing Latency (ms) | Prompt Processing Throughput (tps) | Average Latency of First 128 Tokens Generated (ms) | Average Throughput of First 128 Tokens Generated (tps) | Average Latency of First 256 Tokens Generated (ms) | Average Throughput of First 256 Tokens Generated (tps) | Wall-Clock Latency (s) | Wall-Clock Throughput (tps) |
67+
|-----------------|----|------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
68+
69+
| onnxruntime | 1 | 16 | 11.13351822 | 1437.101883 | 10.54286957 | 94.85083673 | 10.57387795 | 94.57268227 | 3.068660975 | 88.6380093 |
70+
| onnxruntime | 1 | 64 | 11.76165581 | 5441.41072 | 10.52967831 | 94.9696629 | 10.58431901 | 94.47938965 | 3.021231651 | 105.9170686 |
71+
| onnxruntime | 1 | 256 | 21.47537231 | 11920.63151 | 10.69651358 | 93.48840561 | 10.74470952 | 93.06905859 | 3.068811178 | 166.8398511 |
72+
| onnxruntime | 1 | 1024 | 75.85664749 | 13499.14653 | 11.28780469 | 88.59118555 | 11.37996651 | 87.8737208 | 3.290901899 | 388.9511262 |
73+
| onnxruntime | 1 | 2048 | 136.8558168 | 14964.65439 | 12.11677119 | 82.53023716 | 12.13857438 | 82.38199712 | 3.555114031 | 648.0804779 |
74+
| onnxruntime | 1 | 3840 | 255.1124668 | 15052.18482 | 13.43484595 | 74.433306 | 13.47557642 | 74.2083284 | 4.00520277 | 1022.669821 |
75+
| onnxruntime | 4 | 16 | 11.84427261 | 5403.4555 | 10.9059643 | 366.7717856 | 10.99776383 | 363.7103015 | 3.129503489 | 347.658983 |
76+
| onnxruntime | 4 | 64 | 21.20970249 | 12069.94771 | 10.98915562 | 363.9952093 | 11.09936275 | 360.3810499 | 3.171034098 | 403.6538115 |
77+
| onnxruntime | 4 | 256 | 74.3429184 | 13774.00864 | 11.28847338 | 354.3437509 | 11.38650626 | 351.293005 | 3.290605307 | 622.3778938 |
78+
| onnxruntime | 4 | 1024 | 254.2414045 | 16110.67248 | 12.47876324 | 320.5445863 | 12.57592719 | 318.0679992 | 3.774018049 | 1356.64428 |
79+
| onnxruntime | 4 | 2048 | 513.8937044 | 15941.03981 | 14.03383911 | 285.0253569 | 14.12440091 | 283.197852 | 4.431999207 | 2079.422755 |
80+
| onnxruntime | 4 | 3840 | 981.845336 | 15644.01178 | 16.76735282 | 238.5588258 | 16.86015446 | 237.2457506 | 5.601641655 | 2924.856856 |
81+
| onnxruntime | 16 | 16 | 35.34348 | 7243.203 | 21.84567 | 732.4107 | 21.85107 | 732.2296 | 6.171947 | 705.126 |
82+
| onnxruntime | 16 | 64 | 125.9579 | 8129.699 | 21.25177 | 752.8786 | 21.75558 | 735.4436 | 6.062363 | 844.5551 |
83+
| onnxruntime | 16 | 256 | 463.7862 | 8831.656 | 22.7791 | 702.3983 | 23.27515 | 687.4284 | 6.786547 | 1207.094 |
84+
| onnxruntime | 16 | 1024 | 1873.496 | 8745.146 | 28.65446 | 558.3773 | 29.13779 | 549.115 | 9.696249 | 2112.157 |
85+
| onnxruntime | 16 | 2048 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
86+
| onnxruntime | 16 | 3840 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
87+
| pytorch-eager | 1 | 16 | 42.12493 | 379.8226 | 41.37673 | 24.16818 | 41.3158 | 24.20382 | 10.72739 | 25.35564 |
88+
| pytorch-eager | 1 | 64 | 42.26999 | 1514.076 | 41.31047 | 24.20694 | 41.32404 | 24.19899 | 10.72874 | 29.82644 |
89+
| pytorch-eager | 1 | 256 | 45.08846 | 5677.728 | 41.18236 | 24.28224 | 41.16458 | 24.29273 | 10.70244 | 47.83956 |
90+
| pytorch-eager | 1 | 1024 | 181.966 | 5627.425 | 41.06609 | 24.35099 | 41.0729 | 24.34695 | 10.88831 | 117.5572 |
91+
| pytorch-eager | 1 | 2048 | 403.207 | 5079.277 | 42.42671 | 23.57006 | 42.3149 | 23.63234 | 11.11297 | 207.3253 |
92+
| pytorch-eager | 1 | 3840 | 1018.319 | 3770.922 | 41.1257 | 24.3157 | 41.12887 | 24.31382 | 12.18967 | 336.0221 |
93+
| pytorch-eager | 4 | 16 | 43.02286 | 1487.581 | 41.28961 | 96.87667 | 41.34238 | 96.75302 | 10.7351 | 101.3498 |
94+
| pytorch-eager | 4 | 64 | 44.4156 | 5763.741 | 40.88859 | 97.82681 | 41.0022 | 97.55573 | 10.65282 | 120.156 |
95+
| pytorch-eager | 4 | 256 | 160.3949 | 6384.245 | 41.0125 | 97.53125 | 41.05703 | 97.42545 | 10.85462 | 188.6755 |
96+
| pytorch-eager | 4 | 1024 | 659.7745 | 6208.182 | 40.50975 | 98.74166 | 40.46299 | 98.85578 | 11.51654 | 444.5778 |
97+
| pytorch-eager | 4 | 2048 | 1623.698 | 5045.274 | 40.59845 | 98.52594 | 40.50154 | 98.76167 | 15.95867 | 577.4918 |
98+
| pytorch-eager | 4 | 3840 | 4142.559 | 3707.853 | 41.05242 | 97.43639 | 40.99807 | 97.56556 | 25.11443 | 652.374 |
99+
| pytorch-eager | 16 | 16 | 44.22766 | 5788.233 | 43.33905 | 369.1821 | 46.63927 | 343.0585 | 12.10059 | 359.6518 |
100+
| pytorch-eager | 16 | 64 | 154.077 | 6646.029 | 41.8791 | 382.0522 | 46.081 | 347.2147 | 12.17601 | 420.499 |
101+
| pytorch-eager | 16 | 256 | 567.7781 | 7214.086 | 41.41301 | 386.352 | 41.57611 | 384.8364 | 12.75126 | 642.4463 |
102+
| pytorch-eager | 16 | 1024 | 2726.571 | 6009.013 | 41.55068 | 385.0719 | 42.79878 | 373.8425 | 25.90561 | 790.5624 |
103+
| pytorch-eager | 16 | 2048 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
104+
| pytorch-eager | 16 | 3840 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
105+
| pytorch-compile | 1 | 16 | 19.38103 | 825.5494 | 19.05318 | 52.48469 | 52.48469 | 52.94427 | 5.558527 | 48.93383 |
106+
| pytorch-compile | 1 | 64 | 19.37849 | 3302.632 | 18.47518 | 54.12666 | 18.42021 | 54.28819 | 5.583981 | 57.30679 |
107+
| pytorch-compile | 1 | 256 | 34.72291 | 7372.653 | 18.41529 | 54.3027 | 18.41931 | 54.29085 | 5.862381 | 87.33652 |
108+
| pytorch-compile | 1 | 1024 | 138.4733 | 7394.93 | 19.16589 | 52.17604 | 19.16391 | 52.1814 | 6.957563 | 183.9725 |
109+
| pytorch-compile | 1 | 2048 | 271.4814 | 7543.794 | 17.87733 | 55.93676 | 17.86563 | 55.9734 | 8.67444 | 265.6079 |
110+
| pytorch-compile | 1 | 3840 | 597.6365 | 6425.311 | 18.66753 | 53.56894 | 18.60976 | 53.73523 | 11.2927 | 362.7123 |
111+
| pytorch-compile | 4 | 16 | 19.46838 | 3287.381 | 19.08949 | 209.5394 | 19.07973 | 209.6465 | 6.095531 | 178.4914 |
112+
| pytorch-compile | 4 | 64 | 34.53485 | 7412.803 | 19.00581 | 210.462 | 19.21915 | 208.1257 | 6.400232 | 199.9928 |
113+
| pytorch-compile | 4 | 256 | 130.9406 | 7820.343 | 24.85916 | 160.9065 | 22.26585 | 179.6473 | 7.743704 | 264.4729 |
114+
| pytorch-compile | 4 | 1024 | 505.2351 | 8107.116 | 20.01902 | 199.81 | 20.86037 | 191.7511 | 11.46451 | 446.5955 |
115+
| pytorch-compile | 4 | 2048 | 1153.294 | 7103.134 | 19.50131 | 205.1144 | 20.31255 | 196.9226 | 16.80264 | 548.4854 |
116+
| pytorch-compile | 4 | 3840 | 2578.104 | 5957.867 | 19.23593 | 207.9442 | 19.10223 | 209.3997 | 24.32255 | 673.6136 |
117+
| pytorch-compile | 16 | 16 | 34.4058 | 7440.605 | 20.89283 | 765.8128 | 24.65018 | 649.0825 | 8.886711 | 489.72 |
118+
| pytorch-compile | 16 | 64 | 128.428 | 7973.342 | 24.17005 | 661.9762 | 27.00489 | 592.4853 | 10.14654 | 504.6057 |
119+
| pytorch-compile | 16 | 256 | 461.1762 | 8881.638 | 24.57411 | 651.0918 | 26.20077 | 610.6691 | 13.1846 | 621.331 |
120+
| pytorch-compile | 16 | 1024 | 2606.956 | 6284.725 | 39.67544 | 403.2722 | 39.51063 | 404.9543 | 25.67172 | 797.7651 |
121+
| pytorch-compile | 16 | 2048 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
122+
| pytorch-compile | 16 | 3840 | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
123+
124+
## LLaMA-2 70B FP16 CUDA (4 A100 80GB)
125+
126+
| Engine | Batch Size | Prompt Length | Prompt Processing Latency (ms) | Prompt Processing Throughput (tps) | Average Latency of First 128 Tokens Generated (ms) | Average Throughput of First 128 Tokens Generated (tps) | Average Latency of First 256 Tokens Generated (ms) | Average Throughput of First 256 Tokens Generated (tps) | Wall-Clock Latency (s) | Wall-Clock Throughput (tps) |
127+
|-------------|----|------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
128+
| onnxruntime | 1 | 16 | 33.90884399 | 471.8533018 | 32.09379315 | 31.15867281 | 32.19444305 | 31.06126105 | 9.030313015 | 30.12077207 |
129+
| onnxruntime | 1 | 64 | 41.69011116 | 1535.136228 | 33.19698013 | 30.12322193 | 33.02684613 | 30.27839825 | 9.254086733 | 34.57931714 |
130+
| onnxruntime | 1 | 256 | 68.52030754 | 3736.118666 | 32.60990791 | 30.66552665 | 32.65001439 | 30.62785787 | 9.191497326 | 55.70365544 |
131+
| onnxruntime | 1 | 1024 | 207.4344158 | 4936.50003 | 34.06454809 | 29.35603306 | 34.11752358 | 29.31045091 | 9.773607016 | 130.9649547 |
132+
| onnxruntime | 1 | 2048 | 384.3646049 | 5328.274179 | 36.26316041 | 27.57619548 | 36.19885352 | 27.62518431 | 10.52682996 | 218.8693091 |
133+
| onnxruntime | 1 | 3840 | 688.8821125 | 5574.248381 | 39.21672702 | 25.49932327 | 39.3556226 | 25.40932995 | 11.68651938 | 350.4893001 |
134+
| onnxruntime | 4 | 16 | 41.2812233 | 1550.341654 | 33.15291367 | 120.6530454 | 33.26787427 | 120.2361163 | 9.408552885 | 115.6394627 |
135+
| onnxruntime | 4 | 64 | 67.01588631 | 3819.989768 | 33.51482376 | 119.3501726 | 33.34233537 | 119.9676014 | 9.426578999 | 135.7862699 |
136+
| onnxruntime | 4 | 256 | 207.9341412 | 4924.636206 | 33.40258636 | 119.751206 | 33.4388027 | 119.6215079 | 9.585339069 | 213.6596301 |
137+
| onnxruntime | 4 | 1024 | 723.0362892 | 5664.9992 | 34.91823189 | 114.5533374 | 35.27628351 | 113.3906297 | 10.53820944 | 485.8510385 |
138+
| onnxruntime | 4 | 2048 | 1404.65641 | 5832.031193 | 37.19914332 | 107.529358 | 37.12219838 | 107.7522392 | 11.71326399 | 786.8003324 |
139+
| onnxruntime | 4 | 3840 | 2701.535702 | 5685.655011 | 41.06120393 | 97.41555574 | 41.02838039 | 97.49349015 | 14.05006671 | 1166.11546 |
140+
| onnxruntime | 16 | 16 | 67.89302826 | 3770.637524 | 33.8511169 | 472.6579643 | 34.15830154 | 468.4073645 | 9.626672506 | 452.0772881 |
141+
| onnxruntime | 16 | 64 | 204.0295601 | 5018.880595 | 34.05243531 | 469.8636046 | 34.30210985 | 466.4436115 | 9.769803524 | 524.0637631 |
142+
| onnxruntime | 16 | 256 | 715.4211998 | 5725.298609 | 35.19898839 | 454.558518 | 35.41505337 | 451.7852856 | 10.63756585 | 770.1009906 |
143+
| onnxruntime | 16 | 1024 | 2801.265717 | 5848.784677 | 38.95713016 | 410.7078713 | 39.20867946 | 408.0729119 | 13.68341517 | 1496.702376 |
144+
| onnxruntime | 16 | 2048 | 5650.010824 | 5799.634907 | 46.37654498 | 345.0019834 | 46.1302707 | 346.843835 | 18.42047048 | 2001.251816 |
145+
| onnxruntime | 16 | 3840 | 10810.8182 | 5683.196118 | 54.980563 | 291.0119345 | 54.17970195 | 295.3135477 | 25.82262492 | 2537.92944 |

0 commit comments

Comments
 (0)