Skip to content

Commit f3a390c

Browse files
miledxzmiledxz
committed
Longevity and NFR test results for NGF version 1.5.0 (nginx#2771)
Longevity and NFR test results for NGF v1.5.0
1 parent ec3981a commit f3a390c

File tree

85 files changed

+1856
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

85 files changed

+1856
-0
lines changed

‎tests/results/dp-perf/1.5.0/1.5.0-oss.md

+92
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: false
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: 8624530af3c518afd8f7013566a102e8b3497b76
10+
- Date: 2024-11-11T18:50:09Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 12
16+
- k8s version: v1.30.5-gke.1443001
17+
- vCPUs per node: 16
18+
- RAM per node: 65853972Ki
19+
- Max pods per node: 110
20+
- Zone: us-west2-a
21+
- Instance Type: n2d-standard-16
22+
23+
## Summary:
24+
25+
- Performance seems to have improved. There is improved latency and response times across all routing methods.
26+
27+
28+
## Test1: Running latte path based routing
29+
30+
```text
31+
Requests [total, rate, throughput] 30000, 1000.03, 999.28
32+
Duration [total, attack, wait] 30s, 29.999s, 532.506µs
33+
Latencies [min, mean, 50, 90, 95, 99, max] 368.077µs, 659.422µs, 631.038µs, 721.486µs, 756.087µs, 878.907µs, 12.742ms
34+
Bytes In [total, mean] 4800660, 160.02
35+
Bytes Out [total, mean] 0, 0.00
36+
Success [ratio] 99.93%
37+
Status Codes [code:count] 200:29978 503:22
38+
Error Set:
39+
503 Service Temporarily Unavailable
40+
```
41+
42+
## Test2: Running coffee header based routing
43+
44+
```text
45+
Requests [total, rate, throughput] 30000, 1000.03, 1000.01
46+
Duration [total, attack, wait] 30s, 29.999s, 611.932µs
47+
Latencies [min, mean, 50, 90, 95, 99, max] 514.848µs, 666.682µs, 653.935µs, 741.683µs, 777.382µs, 867.041µs, 11.422ms
48+
Bytes In [total, mean] 4830000, 161.00
49+
Bytes Out [total, mean] 0, 0.00
50+
Success [ratio] 100.00%
51+
Status Codes [code:count] 200:30000
52+
Error Set:
53+
```
54+
55+
## Test3: Running coffee query based routing
56+
57+
```text
58+
Requests [total, rate, throughput] 30000, 1000.03, 1000.01
59+
Duration [total, attack, wait] 30s, 29.999s, 618.046µs
60+
Latencies [min, mean, 50, 90, 95, 99, max] 511.713µs, 672.907µs, 658.846µs, 751.753µs, 786.911µs, 881.607µs, 10.507ms
61+
Bytes In [total, mean] 5070000, 169.00
62+
Bytes Out [total, mean] 0, 0.00
63+
Success [ratio] 100.00%
64+
Status Codes [code:count] 200:30000
65+
Error Set:
66+
```
67+
68+
## Test4: Running tea GET method based routing
69+
70+
```text
71+
Requests [total, rate, throughput] 30000, 1000.01, 999.99
72+
Duration [total, attack, wait] 30s, 30s, 597.097µs
73+
Latencies [min, mean, 50, 90, 95, 99, max] 506.955µs, 651.103µs, 638.079µs, 720.439µs, 752.758µs, 828.588µs, 11.282ms
74+
Bytes In [total, mean] 4740000, 158.00
75+
Bytes Out [total, mean] 0, 0.00
76+
Success [ratio] 100.00%
77+
Status Codes [code:count] 200:30000
78+
Error Set:
79+
```
80+
81+
## Test5: Running tea POST method based routing
82+
83+
```text
84+
Requests [total, rate, throughput] 30000, 1000.02, 1000.00
85+
Duration [total, attack, wait] 30s, 29.999s, 596.477µs
86+
Latencies [min, mean, 50, 90, 95, 99, max] 503.899µs, 650.611µs, 639.013µs, 718.258µs, 748.085µs, 827.88µs, 9.075ms
87+
Bytes In [total, mean] 4740000, 158.00
88+
Bytes Out [total, mean] 0, 0.00
89+
Success [ratio] 100.00%
90+
Status Codes [code:count] 200:30000
91+
Error Set:
92+
```

‎tests/results/dp-perf/1.5.0/1.5.0-plus.md

+90
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: true
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: a0126a6435dd4bd69c1a7f48ee15eecb76c68400
10+
- Date: 2024-11-12T20:33:03Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 12
16+
- k8s version: v1.30.5-gke.1443001
17+
- vCPUs per node: 16
18+
- RAM per node: 65853972Ki
19+
- Max pods per node: 110
20+
- Zone: us-west2-a
21+
- Instance Type: n2d-standard-16
22+
23+
## Summary:
24+
25+
- Performance seems consistent with previous test run.
26+
27+
## Test1: Running latte path based routing
28+
29+
```text
30+
Requests [total, rate, throughput] 30000, 1000.02, 1000.00
31+
Duration [total, attack, wait] 30s, 29.999s, 676.331µs
32+
Latencies [min, mean, 50, 90, 95, 99, max] 491.485µs, 689.253µs, 676.054µs, 771.129µs, 806.996µs, 909.616µs, 10.138ms
33+
Bytes In [total, mean] 4800000, 160.00
34+
Bytes Out [total, mean] 0, 0.00
35+
Success [ratio] 100.00%
36+
Status Codes [code:count] 200:30000
37+
Error Set:
38+
```
39+
40+
## Test2: Running coffee header based routing
41+
42+
```text
43+
Requests [total, rate, throughput] 30000, 1000.01, 999.99
44+
Duration [total, attack, wait] 30s, 30s, 686.479µs
45+
Latencies [min, mean, 50, 90, 95, 99, max] 533.29µs, 716.92µs, 703.946µs, 799.238µs, 835.966µs, 942.918µs, 11.356ms
46+
Bytes In [total, mean] 4830000, 161.00
47+
Bytes Out [total, mean] 0, 0.00
48+
Success [ratio] 100.00%
49+
Status Codes [code:count] 200:30000
50+
Error Set:
51+
```
52+
53+
## Test3: Running coffee query based routing
54+
55+
```text
56+
Requests [total, rate, throughput] 30000, 1000.01, 999.98
57+
Duration [total, attack, wait] 30s, 30s, 682.739µs
58+
Latencies [min, mean, 50, 90, 95, 99, max] 549.612µs, 724.458µs, 711.218µs, 810.286µs, 846.648µs, 953.929µs, 9.249ms
59+
Bytes In [total, mean] 5070000, 169.00
60+
Bytes Out [total, mean] 0, 0.00
61+
Success [ratio] 100.00%
62+
Status Codes [code:count] 200:30000
63+
Error Set:
64+
```
65+
66+
## Test4: Running tea GET method based routing
67+
68+
```text
69+
Requests [total, rate, throughput] 30000, 1000.01, 999.98
70+
Duration [total, attack, wait] 30.001s, 30s, 683.465µs
71+
Latencies [min, mean, 50, 90, 95, 99, max] 528.936µs, 716.691µs, 698.583µs, 797.784µs, 834.023µs, 930.167µs, 16.219ms
72+
Bytes In [total, mean] 4740000, 158.00
73+
Bytes Out [total, mean] 0, 0.00
74+
Success [ratio] 100.00%
75+
Status Codes [code:count] 200:30000
76+
Error Set:
77+
```
78+
79+
## Test5: Running tea POST method based routing
80+
81+
```text
82+
Requests [total, rate, throughput] 30000, 1000.01, 999.99
83+
Duration [total, attack, wait] 30s, 30s, 719.615µs
84+
Latencies [min, mean, 50, 90, 95, 99, max] 545.338µs, 715.216µs, 702.127µs, 799.224µs, 835.977µs, 940.498µs, 11.445ms
85+
Bytes In [total, mean] 4740000, 158.00
86+
Bytes Out [total, mean] 0, 0.00
87+
Success [ratio] 100.00%
88+
Status Codes [code:count] 200:30000
89+
Error Set:
90+
```

‎tests/results/longevity/1.5.0/oss-cpu.png

79.9 KB
Loading

‎tests/results/longevity/1.5.0/oss-memory.png

118 KB
Loading

‎tests/results/longevity/1.5.0/oss-ngf-memory.png

71.3 KB
Loading

‎tests/results/longevity/1.5.0/oss-reload-time.png

127 KB
Loading

‎tests/results/longevity/1.5.0/oss-reloads.png

104 KB
Loading

‎tests/results/longevity/1.5.0/oss-stub-status.png

140 KB
Loading

‎tests/results/longevity/1.5.0/oss.md

+99
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: false
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: 36f245bcba55935064324ff5803d66110117f7da
10+
- Date: 2024-11-08T19:20:48Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 2
16+
- k8s version: v1.30.5-gke.1443001
17+
- vCPUs per node: 2
18+
- RAM per node: 4018120Ki
19+
- Max pods per node: 110
20+
- Zone: us-west2-a
21+
- Instance Type: e2-medium
22+
23+
## Traffic
24+
25+
HTTP:
26+
27+
```text
28+
Running 5760m test @ http://cafe.example.com/coffee
29+
2 threads and 100 connections
30+
Thread Stats Avg Stdev Max +/- Stdev
31+
Latency 236.88ms 177.22ms 2.00s 72.93%
32+
Req/Sec 232.09 156.40 1.90k 66.16%
33+
156451087 requests in 5760.00m, 53.52GB read
34+
Socket errors: connect 0, read 350645, write 0, timeout 75472
35+
Requests/sec: 452.69
36+
Transfer/sec: 162.39KB
37+
KB
38+
```
39+
40+
HTTPS:
41+
42+
```text
43+
Running 5760m test @ https://cafe.example.com/tea
44+
2 threads and 100 connections
45+
Thread Stats Avg Stdev Max +/- Stdev
46+
Latency 223.09ms 138.95ms 2.00s 63.95%
47+
Req/Sec 230.23 155.14 1.80k 66.18%
48+
155166081 requests in 5760.00m, 52.20GB read
49+
Socket errors: connect 0, read 345712, write 0, timeout 176
50+
Requests/sec: 448.98
51+
Transfer/sec: 158.37KB
52+
53+
```
54+
55+
56+
### Logs
57+
58+
No error logs in nginx-gateway
59+
60+
Error logs in nginx
61+
62+
We could not get non-2xx errors from the cluster but should likely be similar to last release's issues.
63+
64+
### Key Metrics
65+
66+
#### Containers memory
67+
68+
![oss-memory.png](oss-memory.png)
69+
70+
#### NGF Container Memory
71+
72+
![oss-ngf-memory.png](oss-ngf-memory.png)
73+
74+
### Containers CPU
75+
76+
![oss-cpu.png](oss-cpu.png)
77+
78+
### NGINX metrics
79+
80+
![oss-stub-status.png](oss-stub-status.png)
81+
82+
### Reloads
83+
84+
Rate of reloads - successful and errors:
85+
86+
![oss-reloads.png](oss-reloads.png)
87+
88+
Reload spikes correspond to 1 hour periods of backend re-rollouts.
89+
90+
No reloads finished with an error.
91+
92+
Reload time distribution - counts:
93+
94+
![oss-reload-time.png](oss-reload-time.png)
95+
96+
97+
## Comparison with previous runs
98+
99+
Graphs look similar to 1.4.0 results.

‎tests/results/longevity/1.5.0/plus-cpu.png

84 KB
Loading

‎tests/results/longevity/1.5.0/plus-memory.png

106 KB
Loading

‎tests/results/longevity/1.5.0/plus-ngf-memory.png

81.9 KB
Loading

‎tests/results/longevity/1.5.0/plus-reloads.png

23.8 KB
Loading

‎tests/results/longevity/1.5.0/plus-status.png

160 KB
Loading

‎tests/results/longevity/1.5.0/plus.md

+91
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Results
2+
3+
## Test environment
4+
5+
NGINX Plus: true
6+
7+
NGINX Gateway Fabric:
8+
9+
- Commit: 36f245bcba55935064324ff5803d66110117f7da
10+
- Date: 2024-11-08T19:20:48Z
11+
- Dirty: false
12+
13+
GKE Cluster:
14+
15+
- Node count: 2
16+
- k8s version: v1.30.5-gke.1443001
17+
- vCPUs per node: 2
18+
- RAM per node: 4018120Ki
19+
- Max pods per node: 110
20+
- Zone: us-west2-a
21+
- Instance Type: e2-medium
22+
23+
## Traffic
24+
25+
HTTP:
26+
27+
```text
28+
Running 5760m test @ http://cafe.example.com/coffee
29+
2 threads and 100 connections
30+
Thread Stats Avg Stdev Max +/- Stdev
31+
Latency 228.08ms 136.20ms 1.92s 63.92%
32+
Req/Sec 232.02 153.44 1.71k 66.90%
33+
156457702 requests in 5760.00m, 53.53GB read
34+
Non-2xx or 3xx responses: 5
35+
Requests/sec: 452.71
36+
Transfer/sec: 162.41KB
37+
```
38+
39+
HTTPS:
40+
41+
```text
42+
Running 5760m test @ https://cafe.example.com/tea
43+
2 threads and 100 connections
44+
Thread Stats Avg Stdev Max +/- Stdev
45+
Latency 229.75ms 136.23ms 1.92s 63.81%
46+
Req/Sec 229.91 151.31 1.63k 66.59%
47+
155060805 requests in 5760.00m, 52.19GB read
48+
Non-2xx or 3xx responses: 3
49+
Requests/sec: 448.67
50+
Transfer/sec: 158.33KB
51+
```
52+
53+
### Logs
54+
55+
No error logs in nginx-gateway
56+
57+
Error logs in nginx
58+
59+
We could not get non-2xx errors from the cluster but should likely be similar to last release's issues.
60+
61+
62+
### Key Metrics
63+
64+
#### Containers memory
65+
66+
![plus-memory.png](plus-memory.png)
67+
68+
#### NGF Container Memory
69+
70+
![plus-ngf-memory.png](plus-ngf-memory.png)
71+
72+
### Containers CPU
73+
74+
![plus-cpu.png](plus-cpu.png)
75+
76+
### NGINX Plus metrics
77+
78+
![plus-status.png](plus-status.png)
79+
80+
### Reloads
81+
82+
Rate of reloads - successful and errors:
83+
84+
![plus-reloads.png](plus-reloads.png)
85+
86+
Note: compared to NGINX, we don't have as many reloads here, because NGF uses NGINX Plus API to reconfigure NGINX
87+
for endpoints changes.
88+
89+
## Comparison with previous runs
90+
91+
Graphs look similar to 1.4.0 results.

0 commit comments

Comments
 (0)