Skip to content

Commit 8f32a76

Browse files
author
Raul Marrero
committed
Add HealthChecks support for vs/vsr
1 parent eac5b2d commit 8f32a76

File tree

9 files changed

+901
-56
lines changed

9 files changed

+901
-56
lines changed

‎docs/virtualserver-and-virtualserverroute.md

+76-18
Original file line numberDiff line numberDiff line change
@@ -7,24 +7,26 @@ This document is the reference documentation for the resources. To see additiona
77
**Feature Status**: The VirtualServer and VirtualServerRoute resources are available as a preview feature: it is suitable for experimenting and testing; however, it must be used with caution in production environments. Additionally, while the feature is in preview, we might introduce some backward-incompatible changes to the resources specification in the next releases.
88

99
## Contents
10-
- [VirtualServer and VirtualServerRoute Resources](#virtualserver-and-virtualserverroute-resources)
11-
- [Contents](#contents)
12-
- [Prerequisites](#prerequisites)
13-
- [VirtualServer Specification](#virtualserver-specification)
14-
- [VirtualServer.TLS](#virtualservertls)
15-
- [VirtualServer.Route](#virtualserverroute)
16-
- [VirtualServerRoute Specification](#virtualserverroute-specification)
17-
- [VirtualServerRoute.Subroute](#virtualserverroutesubroute)
18-
- [Common Parts of the VirtualServer and VirtualServerRoute](#common-parts-of-the-virtualserver-and-virtualserverroute)
19-
- [Upstream](#upstream)
20-
- [Upstream.TLS](#upstreamtls)
21-
- [Split](#split)
22-
- [Rules](#rules)
23-
- [Condition](#condition)
24-
- [Match](#match)
25-
- [Using VirtualServer and VirtualServerRoute](#using-virtualserver-and-virtualserverroute)
26-
- [Validation](#validation)
27-
- [Customization via ConfigMap](#customization-via-configmap)
10+
- [VirtualServer and VirtualServerRoute Resources](#VirtualServer-and-VirtualServerRoute-Resources)
11+
- [Contents](#Contents)
12+
- [Prerequisites](#Prerequisites)
13+
- [VirtualServer Specification](#VirtualServer-Specification)
14+
- [VirtualServer.TLS](#VirtualServerTLS)
15+
- [VirtualServer.Route](#VirtualServerRoute)
16+
- [VirtualServerRoute Specification](#VirtualServerRoute-Specification)
17+
- [VirtualServerRoute.Subroute](#VirtualServerRouteSubroute)
18+
- [Common Parts of the VirtualServer and VirtualServerRoute](#Common-Parts-of-the-VirtualServer-and-VirtualServerRoute)
19+
- [Upstream](#Upstream)
20+
- [Upstream.TLS](#UpstreamTLS)
21+
- [Upstream.Healthcheck](#UpstreamHealthcheck)
22+
- [Header](#Header)
23+
- [Split](#Split)
24+
- [Rules](#Rules)
25+
- [Condition](#Condition)
26+
- [Match](#Match)
27+
- [Using VirtualServer and VirtualServerRoute](#Using-VirtualServer-and-VirtualServerRoute)
28+
- [Validation](#Validation)
29+
- [Customization via ConfigMap](#Customization-via-ConfigMap)
2830

2931
## Prerequisites
3032

@@ -209,12 +211,68 @@ tls:
209211
| `next-upstream-timeout` | The time during which a request can be passed to the next upstream server. See the [proxy_next_upstream_timeout](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout) directive. The `0` value turns off the time limit. The default is `0`. | `string` | No |
210212
| `next-upstream-tries` | The number of possible tries for passing a request to the next upstream server. See the [proxy_next_upstream_tries](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries) directive. The `0` value turns off this limit. The default is `0`. | `int` | No |
211213
| `tls` | The TLS configuration for the Upstream. | [`tls`](#UpstreamTLS) | No |
214+
| `healthCheck` | The health check configuration for the Upstream. See the [health_check](http://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html#health_check) directive. Note: this feature is supported only in NGINX Plus. | [`healthcheck`](#UpstreamHealthcheck) | No |
212215

213216
### Upstream.TLS
214217
| Field | Description | Type | Required |
215218
| ----- | ----------- | ---- | -------- |
216219
| `enable` | Enables HTTPS for requests to upstream servers. The default is `False`, meaning that HTTP will be used. | `boolean` | No |
217220

221+
### Upstream.Healthcheck
222+
223+
The Healthcheck defines an [active health check](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/). In the example below we enable a health check for an upstream and configure all the available parameters:
224+
225+
```yaml
226+
name: tea
227+
service: tea-svc
228+
port: 80
229+
healthCheck:
230+
enable: true
231+
path: /healthz
232+
interval: 20s
233+
jitter: 3s
234+
fails: 5
235+
passes: 5
236+
port: 8080
237+
tls:
238+
enable: true
239+
connect-timeout: 10s
240+
read-timeout: 10s
241+
send-timeout: 10s
242+
headers:
243+
- name: Host
244+
value: my.service
245+
statusMatch: "! 500"
246+
```
247+
248+
| Field | Description | Type | Required |
249+
| ----- | ----------- | ---- | -------- |
250+
| `enable` | Enables a health check for an upstream server. The default is `false`. | `boolean` | No |
251+
| `path` | The path used for health check requests. The default is `/`. | `string` | No |
252+
| `interval` | The interval between two consecutive health checks. The default is `5s`. | `string` | No |
253+
| `jitter` | The time within which each health check will be randomly delayed. By default, there is no delay. | `string` | No |
254+
| `fails` | The number of consecutive failed health checks of a particular upstream server after which this server will be considered unhealthy. The default is `1`. | `integer` | No |
255+
| `passes` | The number of consecutive passed health checks of a particular upstream server after which the server will be considered healthy. The default is `1`. | `integer` | No |
256+
| `port` | The port used for health check requests. By default, the port of the upstream is used. Note: in contrast with the port of the upstream, this port is not a service port, but a port of a pod. | `integer` | No |
257+
| `tls` | The TLS configuration used for health check requests. By default, the `tls` field of the upstream is used. | [`upstream.tls`](#UpstreamTLS) | No |
258+
| `connect-timeout` | The timeout for establishing a connection with an upstream server. By default, the `connect-timeout` of the upstream is used. | `string` | No |
259+
| `read-timeout` | The timeout for reading a response from an upstream server. By default, the `read-timeout` of the upstream is used. | `string` | No |
260+
| `send-timeout` | The timeout for transmitting a request to an upstream server. By default, the `send-timeout` of the upstream is used. | `string` | No |
261+
| `headers` | The request headers used for health check requests. NGINX Plus always sets the `Host`, `User-Agent` and `Connection` headers for health check requests. | [`[]header`](#Header) | No |
262+
| `statusMatch` | The expected response status codes of a health check. By default, the response should have status code 2xx or 3xx. Examples: `“200”`, `“! 500”`, `"301-303 307"`. See the documentation of the [match](https://nginx.org/en/docs/http/ngx_http_upstream_hc_module.html?#match) directive. | `string` | No |
263+
264+
### Header
265+
The header defines an HTTP Header:
266+
```yaml
267+
name: Host
268+
value: example.com
269+
```
270+
271+
| Field | Description | Type | Required |
272+
| ----- | ----------- | ---- | -------- |
273+
| `name` | The name of the header. | `string` | Yes |
274+
| `value` | The value of the header. | `string` | No |
275+
218276
### Split
219277

220278
The split defines a weight for an upstream as part of the splits configuration.

‎internal/configs/version2/config.go

+29-4
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@ package version2
22

33
// VirtualServerConfig holds NGINX configuration for a VirtualServer.
44
type VirtualServerConfig struct {
5-
Server Server
6-
Upstreams []Upstream
7-
SplitClients []SplitClient
8-
Maps []Map
5+
Server Server
6+
Upstreams []Upstream
7+
SplitClients []SplitClient
8+
Maps []Map
9+
StatusMatches []StatusMatch
910
}
1011

1112
// Upstream defines an upstream.
@@ -37,6 +38,7 @@ type Server struct {
3738
Snippets []string
3839
InternalRedirectLocations []InternalRedirectLocation
3940
Locations []Location
41+
HealthChecks []HealthCheck
4042
}
4143

4244
// SSL defines SSL configuration for a server.
@@ -74,6 +76,23 @@ type SplitClient struct {
7476
Distributions []Distribution
7577
}
7678

79+
// HealthCheck defines a HealthCheck for an upstream in a Server.
80+
type HealthCheck struct {
81+
Name string
82+
URI string
83+
Interval string
84+
Jitter string
85+
Fails int
86+
Passes int
87+
Port int
88+
ProxyPass string
89+
ProxyConnectTimeout string
90+
ProxyReadTimeout string
91+
ProxySendTimeout string
92+
Headers map[string]string
93+
Match string
94+
}
95+
7796
// Distribution maps weight to a value in a SplitClient.
7897
type Distribution struct {
7998
Weight string
@@ -98,3 +117,9 @@ type Parameter struct {
98117
Value string
99118
Result string
100119
}
120+
121+
// StatusMatch defines a Match block for status codes.
122+
type StatusMatch struct {
123+
Name string
124+
Code string
125+
}

‎internal/configs/version2/nginx-plus.virtualserver.tmpl

+20
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,12 @@ map {{ $m.Source }} {{ $m.Variable }} {
3030
}
3131
{{ end }}
3232

33+
{{ range $m := .StatusMatches }}
34+
match {{ $m.Name }} {
35+
status {{ $m.Code }};
36+
}
37+
{{ end }}
38+
3339
{{ $s := .Server }}
3440
server {
3541
listen 80{{ if $s.ProxyProtocol }} proxy_protocol{{ end }};
@@ -82,6 +88,20 @@ server {
8288
}
8389
{{ end }}
8490

91+
{{ range $hc := $s.HealthChecks }}
92+
location @hc-{{ $hc.Name }} {
93+
{{ range $n, $v := $hc.Headers }}
94+
proxy_set_header {{ $n }} "{{ $v }}";
95+
{{ end }}
96+
proxy_connect_timeout {{ $hc.ProxyConnectTimeout }};
97+
proxy_read_timeout {{ $hc.ProxyReadTimeout }};
98+
proxy_send_timeout {{ $hc.ProxySendTimeout }};
99+
proxy_pass {{ $hc.ProxyPass }};
100+
health_check uri={{ $hc.URI }} port={{ $hc.Port }} interval={{ $hc.Interval }} jitter={{ $hc.Jitter }}
101+
fails={{ $hc.Fails }} passes={{ $hc.Passes }}{{ if $hc.Match }} match={{ $hc.Match }}{{ end }};
102+
}
103+
{{ end }}
104+
85105
{{ range $l := $s.Locations }}
86106
location {{ $l.Path }} {
87107
{{ range $snippet := $l.Snippets }}

‎internal/configs/virtualserver.go

+110-6
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,92 @@ func (namer *variableNamer) GetNameForVariableForRulesRouteMainMap(rulesIndex in
8181
return fmt.Sprintf("$vs_%s_rules_%d", namer.safeNsName, rulesIndex)
8282
}
8383

84+
func newHealthCheckWithDefaults(upstream conf_v1alpha1.Upstream, upstreamName string, cfgParams *ConfigParams) *version2.HealthCheck {
85+
return &version2.HealthCheck{
86+
Name: upstreamName,
87+
URI: "/",
88+
Interval: "5s",
89+
Jitter: "0s",
90+
Fails: 1,
91+
Passes: 1,
92+
Port: int(upstream.Port),
93+
ProxyPass: fmt.Sprintf("%v://%v", generateProxyPassProtocol(upstream.TLS.Enable), upstreamName),
94+
ProxyConnectTimeout: generateString(upstream.ProxyConnectTimeout, cfgParams.ProxyConnectTimeout),
95+
ProxyReadTimeout: generateString(upstream.ProxyReadTimeout, cfgParams.ProxyReadTimeout),
96+
ProxySendTimeout: generateString(upstream.ProxySendTimeout, cfgParams.ProxySendTimeout),
97+
Headers: make(map[string]string),
98+
}
99+
}
100+
101+
func generateHealthCheck(upstream conf_v1alpha1.Upstream, upstreamName string, cfgParams *ConfigParams) *version2.HealthCheck {
102+
if upstream.HealthCheck == nil || !upstream.HealthCheck.Enable {
103+
return nil
104+
}
105+
106+
hc := newHealthCheckWithDefaults(upstream, upstreamName, cfgParams)
107+
108+
if upstream.HealthCheck.Path != "" {
109+
hc.URI = upstream.HealthCheck.Path
110+
}
111+
112+
if upstream.HealthCheck.Interval != "" {
113+
hc.Interval = upstream.HealthCheck.Interval
114+
}
115+
116+
if upstream.HealthCheck.Jitter != "" {
117+
hc.Jitter = upstream.HealthCheck.Jitter
118+
}
119+
120+
if upstream.HealthCheck.Fails > 0 {
121+
hc.Fails = upstream.HealthCheck.Fails
122+
}
123+
124+
if upstream.HealthCheck.Passes > 0 {
125+
hc.Passes = upstream.HealthCheck.Passes
126+
}
127+
128+
if upstream.HealthCheck.Port > 0 {
129+
hc.Port = upstream.HealthCheck.Port
130+
}
131+
132+
if upstream.HealthCheck.ConnectTimeout != "" {
133+
hc.ProxyConnectTimeout = upstream.HealthCheck.ConnectTimeout
134+
}
135+
136+
if upstream.HealthCheck.ReadTimeout != "" {
137+
hc.ProxyReadTimeout = upstream.HealthCheck.ReadTimeout
138+
}
139+
140+
if upstream.HealthCheck.SendTimeout != "" {
141+
hc.ProxySendTimeout = upstream.HealthCheck.SendTimeout
142+
}
143+
144+
for _, h := range upstream.HealthCheck.Headers {
145+
hc.Headers[h.Name] = h.Value
146+
}
147+
148+
if upstream.HealthCheck.TLS != nil {
149+
hc.ProxyPass = fmt.Sprintf("%v://%v", generateProxyPassProtocol(upstream.HealthCheck.TLS.Enable), upstreamName)
150+
}
151+
152+
if upstream.HealthCheck.StatusMatch != "" {
153+
hc.Match = generateStatusMatchName(upstreamName)
154+
}
155+
156+
return hc
157+
}
158+
159+
func generateStatusMatchName(upstreamName string) string {
160+
return fmt.Sprintf("%s_match", upstreamName)
161+
}
162+
163+
func generateUpstreamStatusMatch(upstreamName string, status string) version2.StatusMatch {
164+
return version2.StatusMatch{
165+
Name: generateStatusMatchName(upstreamName),
166+
Code: status,
167+
}
168+
}
169+
84170
func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileName string, baseCfgParams *ConfigParams, isPlus bool) version2.VirtualServerConfig {
85171
ssl := generateSSLConfig(virtualServerEx.VirtualServer.Spec.TLS, tlsPemFileName, baseCfgParams)
86172

@@ -91,6 +177,8 @@ func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileNam
91177
virtualServerUpstreamNamer := newUpstreamNamerForVirtualServer(virtualServerEx.VirtualServer)
92178

93179
var upstreams []version2.Upstream
180+
var statusMatches []version2.StatusMatch
181+
var healthChecks []version2.HealthCheck
94182

95183
// generate upstreams for VirtualServer
96184
for _, u := range virtualServerEx.VirtualServer.Spec.Upstreams {
@@ -99,6 +187,13 @@ func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileNam
99187
ups := generateUpstream(upstreamName, u, virtualServerEx.Endpoints[endpointsKey], isPlus, baseCfgParams)
100188
upstreams = append(upstreams, ups)
101189
crUpstreams[upstreamName] = u
190+
191+
if hc := generateHealthCheck(u, upstreamName, baseCfgParams); hc != nil {
192+
healthChecks = append(healthChecks, *hc)
193+
if u.HealthCheck.StatusMatch != "" {
194+
statusMatches = append(statusMatches, generateUpstreamStatusMatch(upstreamName, u.HealthCheck.StatusMatch))
195+
}
196+
}
102197
}
103198
// generate upstreams for each VirtualServerRoute
104199
for _, vsr := range virtualServerEx.VirtualServerRoutes {
@@ -109,6 +204,13 @@ func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileNam
109204
ups := generateUpstream(upstreamName, u, virtualServerEx.Endpoints[endpointsKey], isPlus, baseCfgParams)
110205
upstreams = append(upstreams, ups)
111206
crUpstreams[upstreamName] = u
207+
208+
if hc := generateHealthCheck(u, upstreamName, baseCfgParams); hc != nil {
209+
healthChecks = append(healthChecks, *hc)
210+
if u.HealthCheck.StatusMatch != "" {
211+
statusMatches = append(statusMatches, generateUpstreamStatusMatch(upstreamName, u.HealthCheck.StatusMatch))
212+
}
213+
}
112214
}
113215
}
114216

@@ -179,9 +281,10 @@ func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileNam
179281
}
180282

181283
return version2.VirtualServerConfig{
182-
Upstreams: upstreams,
183-
SplitClients: splitClients,
184-
Maps: maps,
284+
Upstreams: upstreams,
285+
SplitClients: splitClients,
286+
Maps: maps,
287+
StatusMatches: statusMatches,
185288
Server: version2.Server{
186289
ServerName: virtualServerEx.VirtualServer.Spec.Host,
187290
ProxyProtocol: baseCfgParams.ProxyProtocol,
@@ -194,6 +297,7 @@ func generateVirtualServerConfig(virtualServerEx *VirtualServerEx, tlsPemFileNam
194297
Snippets: baseCfgParams.ServerSnippets,
195298
InternalRedirectLocations: internalRedirectLocations,
196299
Locations: locations,
300+
HealthChecks: healthChecks,
197301
},
198302
}
199303
}
@@ -252,8 +356,8 @@ func upstreamHasKeepalive(upstream conf_v1alpha1.Upstream, cfgParams *ConfigPara
252356
return cfgParams.Keepalive != 0
253357
}
254358

255-
func generateProxyPassProtocol(upstream conf_v1alpha1.Upstream) string {
256-
if upstream.TLS.Enable {
359+
func generateProxyPassProtocol(enableTLS bool) string {
360+
if enableTLS {
257361
return "https"
258362
}
259363
return "http"
@@ -278,7 +382,7 @@ func generateLocation(path string, upstreamName string, upstream conf_v1alpha1.U
278382
ProxyBuffering: cfgParams.ProxyBuffering,
279383
ProxyBuffers: cfgParams.ProxyBuffers,
280384
ProxyBufferSize: cfgParams.ProxyBufferSize,
281-
ProxyPass: fmt.Sprintf("%v://%v", generateProxyPassProtocol(upstream), upstreamName),
385+
ProxyPass: fmt.Sprintf("%v://%v", generateProxyPassProtocol(upstream.TLS.Enable), upstreamName),
282386
ProxyNextUpstream: generateString(upstream.ProxyNextUpstream, "error timeout"),
283387
ProxyNextUpstreamTimeout: generateString(upstream.ProxyNextUpstreamTimeout, "0s"),
284388
ProxyNextUpstreamTries: upstream.ProxyNextUpstreamTries,

0 commit comments

Comments
 (0)