Skip to content

[enterprise-3.9] Clearly Specify Router Metrics file format returned #11288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 1, 2018
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 66 additions & 73 deletions install_config/router/default_haproxy_router.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,6 @@ The following example creates a router deployment with `0` replicas and
customizes the router service HTTP and HTTPS ports, then scales it
appropriately (to `1` replica).

====
----
$ oc adm router --replicas=0 --ports='10080:10080,10443:10443' //<1>
$ oc set env dc/router ROUTER_SERVICE_HTTP_PORT=10080 \
Expand All @@ -374,7 +373,6 @@ $ oc scale dc/router --replicas=1
----
<1> Ensures exposed ports are appropriately set for routers that use the
container networking mode `--host-network=false`.
====

[IMPORTANT]
====
Expand All @@ -387,12 +385,10 @@ that you use via `firewall-cmd`).
The following is an example using `iptables` to open the custom router service
ports.

====
----
$ iptables -A INPUT -p tcp --dport 10080 -j ACCEPT
$ iptables -A INPUT -p tcp --dport 10443 -j ACCEPT
----
====

[[working-with-multiple-routers]]
== Working With Multiple Routers
Expand Down Expand Up @@ -435,7 +431,6 @@ $ oc edit dc <deploymentConfigName>
Add the `template.spec.nodeSelector` field with a key and value
corresponding to the label:
+
====
----
...
template:
Expand All @@ -450,7 +445,6 @@ corresponding to the label:
----
<1> The key and value are `router` and `first`, respectively,
corresponding to the `router=first` label.
====

[[using-router-shards]]
== Using Router Shards
Expand Down Expand Up @@ -494,7 +488,6 @@ Separate DNS entries must resolve *.foo.com to the node hosting Router A and *.e
* `*.foo.com A IN 192.168.0.5`
* `*.example.com A IN 192.168.1.9`


*Router Sharding Examples*

This section describes router sharding using namespace and route labels.
Expand Down Expand Up @@ -657,7 +650,6 @@ Here is a convenience script *_mkshard_* that
illustrates how `oc adm router`, `oc set env`, and `oc scale`
can be used together to make a router shard.

====
[source,bash]
----
#!/bin/bash
Expand All @@ -677,7 +669,6 @@ oc scale $dc --replicas=3 //<5>
The selection expression is the value of
the `ROUTE_LABELS` environment variable.
<5> Scale it up.
====

Running *_mkshard_* several times creates several routers:

Expand Down Expand Up @@ -716,7 +707,6 @@ demonstrating how to change the selection expression.
Here is a convenience script *_modshard_* that modifies
an existing router to use a new selection expression:

====
[source,bash]
----
#!/bin/bash
Expand All @@ -739,7 +729,6 @@ oc scale $dc --replicas=3 //<5>
non-`ID` arguments to `modshard` must include the
environment variable name as well as its value.
<5> Scale it back up.
====

[NOTE]
====
Expand Down Expand Up @@ -879,12 +868,10 @@ this default routing subdomain.
The following example shows how you can set the configured suffix
to *v3.openshift.test*:

====
----
routingConfig:
subdomain: v3.openshift.test
----
====

[NOTE]
====
Expand All @@ -896,11 +883,9 @@ xref:../../architecture/networking/routes.adoc#route-hostnames[generated host
name] for the example of a route named *no-route-hostname* without a
host name added to a namespace *mynamespace* would be:

====
----
no-route-hostname-mynamespace.v3.openshift.test
----
====

[[forcing-route-hostnames-to-a-custom-routing-subdomain]]
== Forcing Route Host Names to a Custom Routing Subdomain
Expand All @@ -913,11 +898,9 @@ a route and generate one based on the template provided to the
The following example runs a router, which overrides the route host names using
a custom subdomain template `${name}-${namespace}.apps.example.com`.

====
----
$ oc adm router --force-subdomain='${name}-${namespace}.apps.example.com'
----
====

[[using-wildcard-certificates]]
== Using Wildcard Certificates
Expand All @@ -927,15 +910,13 @@ default certificate instead. In most cases, this certificate should be provided
by a trusted certificate authority, but for convenience you can use the
{product-title} CA to create the certificate. For example:

====
----
$ CA=/etc/origin/master
$ oc adm ca create-server-cert --signer-cert=$CA/ca.crt \
--signer-key=$CA/ca.key --signer-serial=$CA/ca.serial.txt \
--hostnames='*.cloudapps.example.com' \
--cert=cloudapps.crt --key=cloudapps.key
----
====

[NOTE]
====
Expand All @@ -950,19 +931,15 @@ by default *_/etc/ansible/hosts_*.
The router expects the certificate and key to be in PEM format in a single
file:

====
----
$ cat cloudapps.crt cloudapps.key $CA/ca.crt > cloudapps.router.pem
----
====

From there you can use the `--default-cert` flag:

====
----
$ oc adm router --default-cert=cloudapps.router.pem --service-account=router
----
====

[NOTE]
====
Expand Down Expand Up @@ -1113,7 +1090,6 @@ DNS lookup:
# curl -k --resolve www.example.test:443:$routerip https://www.example.test/
----


[[using-wildcard-routes]]
== Using Wildcard Routes (for a Subdomain)

Expand Down Expand Up @@ -1455,18 +1431,14 @@ stack, use the `--host-network=false` option when creating the router. For
example:

ifdef::openshift-enterprise[]
====
----
$ oc adm router --service-account=router --host-network=false
----
====
endif::[]
ifdef::openshift-origin[]
====
----
$ oc adm router --service-account=router --host-network=false
----
====
endif::[]

Internally, this means the router container must publish the 80 and 443
Expand All @@ -1491,7 +1463,6 @@ access any pod in the cluster. If isolation is needed in this case, then do not
add routes across the namespaces.
====


[[exposing-the-router-metrics]]
== Exposing Router Metrics

Expand All @@ -1500,63 +1471,49 @@ xref:../../architecture/networking/assembly_available_router_plugins.adoc#haprox
are, by default, exposed or published in
link:https://prometheus.io/docs/concepts/data_model/[Prometheus format]
for consumption by external metrics collection and aggregation systems (e.g. Prometheus, statsd).
Metrics are also available dirctly from the
link:https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#9[HAProxy router] in its own CSV format.
Metrics are also available directly from the
link:https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#9[HAProxy router] in its own HTML format for viewing in a browser
or CSV download.
These metrics include the HAProxy native metrics and some controller metrics.

When you create a router using the following command, {product-title} makes metrics available in Prometheus format on the stats port, by default 1936.

When you create a router, as below,
====
----
$ oc adm router --service-account=router
----
====
metrics are automatically available in Prometheus format on the stats-port, default 1936. To suppress metrics collection,
====

* To extract the raw statistics in Prometheus format run the following command:
+
----
$ oc adm router --service-account=router --stats-port=0
curl <user>:<password>@<router_IP>:<STATS_PORT>
----
====

To switch to the HAProxy CSV format metrics, edit the xref:../../architecture/networking/routes.adoc#env-variables[environment variables]
for the router dc and delete the following lines:

====
+
For example:
+
----
- name: ROUTER_LISTEN_ADDR
value: 0.0.0.0:1936
- name: ROUTER_METRICS_TYPE
value: haproxy
$ curl admin:[email protected]:1936/metrics
----
====
Where 1936 is the STATS_PORT value.

[NOTE]
====
The `--expose-metrics` and `--metrics-image` options are deprecated. The haproxy-exporter
side car is now integrated into the router controller so you can delete the sidecar container from existing
router deployment configs. You can continue to use the side car in existing routers. New routers use the integrated metrics.
====


You can extract the raw statistics in Prometheus format by using the following.

Information needed to access the metrics is found the router service annotations:

====
+
You can get the information you need to access the metrics from the router service annotations:
+
----
$ oc edit router service <router-service-name>

apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/port: "1936"
prometheus.io/scrape: "true"
prometheus.openshift.io/password: IImoDqON02
prometheus.openshift.io/username: admin
----
====

The metrics port is set from the STATS_PORT, default 1936. You may need to confiugre your firewall to permit access.
Use the above username and password to access the metrics. The path is "/metrics".

+
The `prometheus.io/port` is the stats port, by default 1936. You might need to configure your firewall to permit access.
Use the previous user name and password to access the metrics. The path is *_/metrics_*.
+
----
$ curl <user>:<password>@<router_IP>:<STATS_PORT>/metrics
$ curl <user>:<password>@<router_IP>:<STATS_PORT>
for example:
$ curl admin:[email protected]:1936/metrics
...
Expand Down Expand Up @@ -1586,6 +1543,45 @@ haproxy_server_bytes_in_total{namespace="default",pod="hello-rc-vkjqx",route="he
...
----

* To get metrics in a browser:
+
. Delete the following xref:../../architecture/networking/routes.adoc#env-variables[environment variables]
from the router deployment configuration file:
+
----
$ oc edit service router

- name: ROUTER_LISTEN_ADDR
value: 0.0.0.0:1936
- name: ROUTER_METRICS_TYPE
value: haproxy
----
+
. Launch the stats window using the following URL in a browser, where the `STATS_PORT` value is `1936` by default:
+
----
http://admin:<Password>@<router_IP>:<STATS_PORT>
----
+
You can get the stats in CSV format by adding `;csv` to the URL:
+
For example:
+
----
http://admin:<Password>@<router_IP>:1936;csv
----
+
To get the router IP, admin name, and password:
+
----
oc describe pod <router_pod>
----

* To suppress metrics collection:
+
----
$ oc adm router --service-account=router --stats-port=0
----

[[preventing-connection-failures-during-restarts]]
== Preventing Connection Failures During Restarts
Expand All @@ -1596,6 +1592,7 @@ dropped. The issue is being addressed. In the meantime, it is possible to work
around the problem by installing `iptables` rules to prevent connections during
the reload window. However, doing so means that the router needs to run with
elevated privilege so that it can manipulate `iptables` on the host. It also

means that connections that happen during the reload are temporarily ignored and
must retransmit their connection start, lengthening the time it takes to
connect, but preventing connection failure.
Expand Down Expand Up @@ -1628,11 +1625,9 @@ $ oc patch dc router -p '{"spec":{"template":{"spec":{"containers":[{"name":"rou

Set the option on the router deployment configuration:

====
----
$ oc set env dc/router -c router DROP_SYN_DURING_RESTART=true
----
====

If you used a non-default name for the router, you must change *_dc/router_*
accordingly.
Expand Down Expand Up @@ -1695,7 +1690,6 @@ Add *timeout http-request* to the default HAProxy router image to
protect the deployment against distributed denial-of-service (DDoS) attacks (for
example, slowloris):

====
----
# and the haproxy stats socket is available at /var/run/haproxy.stats
global
Expand All @@ -1712,7 +1706,6 @@ defaults
<1> *timeout http-request* is set up to 5 seconds. HAProxy gives a client 5 seconds
*to send its whole HTTP request. Otherwise, HAProxy shuts the connection with
*an error.
====

Also, when the environment variable `*ROUTER_SLOWLORIS_TIMEOUT*` is set, it
limits the amount of time a client has to send the whole HTTP request.
Expand Down