Elasticsearch 7.13 Breaking Metrics #419

mcmarkj · 2021-05-27T11:12:33Z

Elasticsearch 7.13 appears to have removed several key metrics around cluster disk storage etc.

This means metrics like elasticsearch_filesystem_data_size_bytes are no longer present.

The text was updated successfully, but these errors were encountered:

epicvinny · 2021-05-27T21:25:36Z

Same here, error log:

caller=nodes.go:1851 msg="failed to fetch and decode node stats" err="json: cannot unmarshal array into Go struct field NodeStatsNodeResponse.http of type int"

jivco · 2021-05-29T12:11:25Z

I think the problem is that they have added info on each HTTP client to HTTP stats and this breaks the nodes_response collector.

I think this is a fast solution but I don't know how to export the new metrics. I'm using this branch for building because the original one is broken.

diff.txt

adammike · 2021-06-02T02:31:25Z

Is anyone actively looking at this issue?

hansh0801 · 2021-06-02T02:44:02Z

I also have this problem. Is anyone actively looking at this issue?

egallis31 · 2021-06-02T13:41:50Z

I think the problem is that they have added info on each HTTP client to HTTP stats and this breaks the nodes_response collector.

I think this is a fast solution but I don't know how to export the new metrics. I'm using this branch for building because the original one is broken.

diff.txt

Getting some issues building using that branch - anything missing from the make file?

>> building docker image
Sending build context to Docker daemon  11.51MB
Step 1/9 : FROM quay.io/prometheus/golang-builder as builder
 ---> a6c160c7ad69
Step 2/9 : ADD .   /go/src/github.com/justwatchcom/elasticsearch_exporter
 ---> Using cache
 ---> 34bc37901c23
Step 3/9 : WORKDIR /go/src/github.com/justwatchcom/elasticsearch_exporter
 ---> Using cache
 ---> 2e4434305a5e
Step 4/9 : RUN make
 ---> Running in 2ecf04b34fff
>> formatting code
go: downloading github.com/prometheus/promu v0.12.0
go: downloading github.com/google/go-github/v25 v25.1.3
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/prometheus/common v0.25.0
go: downloading go.uber.org/atomic v1.7.0
go: downloading golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/Masterminds/semver v1.5.0
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15
go: downloading github.com/prometheus/client_golang v1.10.0
go: downloading golang.org/x/net v0.0.0-20210525063256-abc453219eb5
go: downloading google.golang.org/appengine v1.6.7
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading github.com/prometheus/procfs v0.6.0
go: downloading golang.org/x/sys v0.0.0-20210601080250-7ecdf8ef093b
go: downloading google.golang.org/protobuf v1.26.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/google/go-querystring v1.1.0
>> building binaries
 >   elasticsearch_exporter
# github.com/justwatchcom/elasticsearch_exporter
./main.go:163:27: undefined: prometheus.Handler
!! command failed: build -o /go/src/github.com/justwatchcom/elasticsearch_exporter/elasticsearch_exporter -ldflags -s -X github.com/justwatchcom/elasticsearch_exporter/vendor/github.com/prometheus/common/version.Version=1.1.0 -X github.com/justwatchcom/elasticsearch_exporter/vendor/github.com/prometheus/common/version.Revision=c4c7d2bf2ed55725515dd27df4fd41b6c0b5c33c -X github.com/justwatchcom/elasticsearch_exporter/vendor/github.com/prometheus/common/version.Branch=master -X github.com/justwatchcom/elasticsearch_exporter/vendor/github.com/prometheus/common/version.BuildUser=root@2ecf04b34fff -X github.com/justwatchcom/elasticsearch_exporter/vendor/github.com/prometheus/common/version.BuildDate=20210602-13:38:11  -extldflags '-static' -a -tags netgo github.com/justwatchcom/elasticsearch_exporter: exit status 2
make: *** [Makefile:45: build] Error 1
The command '/bin/sh -c make' returned a non-zero code: 2
make: *** [Makefile:63: docker] Error 2

jivco · 2021-06-02T14:12:08Z

My bild is working ok :) Are you using fix-build-go-mod branch from https://github.com/repl-mike-roest/elasticsearch_exporter for building?

jivco@jivco:/tmp/elasticsearch_exporter$ docker build -t jivco-test-image:v1 .
Sending build context to Docker daemon  1.581MB
Step 1/9 : FROM quay.io/prometheus/golang-builder as builder
 ---> a6c160c7ad69
Step 2/9 : ADD .   /go/src/github.com/justwatchcom/elasticsearch_exporter
 ---> aa516d7645e8
Step 3/9 : WORKDIR /go/src/github.com/justwatchcom/elasticsearch_exporter
 ---> Running in 5f6b8a6e32b5
Removing intermediate container 5f6b8a6e32b5
 ---> abcf99aeb651
Step 4/9 : RUN make
 ---> Running in a44568db3170
go: downloading github.com/go-kit/kit v0.10.0
go: downloading github.com/blang/semver v3.5.2-0.20180723201105-3c1074078d32+incompatible
go: downloading github.com/prometheus/client_golang v1.10.0
go: downloading github.com/prometheus/common v0.20.0
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/imdario/mergo v0.3.7-0.20181107191138-ca3dcc1022ba
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20210208195552-ff826a37aa15
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.1
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/prometheus/procfs v0.6.0
go: downloading github.com/go-logfmt/logfmt v0.5.0
go: downloading google.golang.org/protobuf v1.26.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/sys v0.0.0-20210415045647-66c3f260301c
>> formatting code
collector/nodes_response.go
go: downloading github.com/prometheus/promu v0.12.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/google/go-github/v25 v25.1.3
go: downloading go.uber.org/atomic v1.7.0
go: downloading golang.org/x/oauth2 v0.0.0-20210413134643-5e61552d6c78
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/Masterminds/semver v1.5.0
go: downloading github.com/prometheus/common v0.25.0
go: downloading golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c
go: downloading github.com/google/go-github v17.0.0+incompatible
go: downloading github.com/cespare/xxhash v1.1.0
go: downloading golang.org/x/net v0.0.0-20210415231046-e915ea6b2b7d
go: downloading google.golang.org/appengine v1.6.7
go: downloading golang.org/x/sys v0.0.0-20210601080250-7ecdf8ef093b
go: downloading golang.org/x/net v0.0.0-20210525063256-abc453219eb5
go: downloading github.com/google/go-querystring v1.1.0
go get: upgraded github.com/prometheus/common v0.20.0 => v0.25.0
go get: upgraded golang.org/x/net v0.0.0-20210415231046-e915ea6b2b7d => v0.0.0-20210525063256-abc453219eb5
go get: upgraded golang.org/x/oauth2 v0.0.0-20210413134643-5e61552d6c78 => v0.0.0-20210514164344-f6687ab2804c
go get: upgraded golang.org/x/sys v0.0.0-20210415045647-66c3f260301c => v0.0.0-20210601080250-7ecdf8ef093b
>> building binaries
 >   elasticsearch_exporter
>> running tests
?   	github.com/justwatchcom/elasticsearch_exporter	[no test files]
ok  	github.com/justwatchcom/elasticsearch_exporter/collector	0.033s
ok  	github.com/justwatchcom/elasticsearch_exporter/pkg/clusterinfo	0.028s
Removing intermediate container a44568db3170
 ---> 33d4fdd4a3f7
Step 5/9 : FROM        quay.io/prometheus/busybox:latest
 ---> 765c5b099deb
Step 6/9 : MAINTAINER  The Prometheus Authors <[email protected]>
 ---> Using cache
 ---> e58a1fa6b1d4
Step 7/9 : COPY --from=builder /go/src/github.com/justwatchcom/elasticsearch_exporter/elasticsearch_exporter  /bin/elasticsearch_exporter
 ---> f2d0044bbb84
Step 8/9 : EXPOSE      9114
 ---> Running in 164d8708eaae
Removing intermediate container 164d8708eaae
 ---> 4d01b438734f
Step 9/9 : ENTRYPOINT  [ "/bin/elasticsearch_exporter" ]
 ---> Running in e0c9b4101bdb
Removing intermediate container e0c9b4101bdb
 ---> 7f55b14f692d
Successfully built 7f55b14f692d
Successfully tagged jivco-test-image:v1

sonujatav35 · 2021-06-03T02:25:35Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

hansh0801 · 2021-06-03T06:52:50Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

i fix with this following process.
clone this repo, and fix code with @jivco did. and build with env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build command.
then go binary file comes out, dockerize it.

sonujatav35 · 2021-06-03T13:36:26Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

i fix with this following process.
clone this repo, and fix code with @jivco did. and build with env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build command.
then go binary file comes out, dockerize it.

With the help of above comment I created that package but when I moved it on linux environment server. I am getting the error like bash can not execute the query.
Do I need to add some other step so that package can be executed over linux environment.

hansh0801 · 2021-06-03T13:52:39Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

i fix with this following process.
clone this repo, and fix code with @jivco did. and build with env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build command.
then go binary file comes out, dockerize it.

With the help of above comment I created that package but when I moved it on linux environment server. I am getting the error like bash can not execute the query.
Do I need to add some other step so that package can be executed over linux environment.

maybe you could check other linux go build env command. i build with mac env with this command, and run at amazon linux2 OS.

jivco · 2021-06-03T14:34:01Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

i fix with this following process.
clone this repo, and fix code with @jivco did. and build with env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build command.
then go binary file comes out, dockerize it.

With the help of above comment I created that package but when I moved it on linux environment server. I am getting the error like bash can not execute the query.
Do I need to add some other step so that package can be executed over linux environment.

Can you paste the exact error? Chan you check if the file is executable?

As per prometheus-community#419, this functionality is failing for elasticsearch version 7.13+ due to a breaking change in the response from elasticsearch. This adds some test fixtures for a range of elasticsearch versions including 7.13 to help ensure that future changes fix the incompatibility with 7.13 and maintain compatibility with versions < 7.13. Signed-off-by: Joe Adams <[email protected]>

egallis31 · 2021-06-03T16:24:13Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

Unsure if this will help but exact commands:

git clone -b fix-build-go-mod https://github.com/repl-mike-roest/elasticsearch_exporter.git
docker build -t elasticsearch_exporter .

sonujatav35 · 2021-06-04T01:38:45Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

i fix with this following process.
clone this repo, and fix code with @jivco did. and build with env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build command.
then go binary file comes out, dockerize it.

With the help of above comment I created that package but when I moved it on linux environment server. I am getting the error like bash can not execute the query.
Do I need to add some other step so that package can be executed over linux environment.

Can you paste the exact error? Chan you check if the file is executable?

I got this error -bash: ./elasticsearch_exporter: cannot execute binary file
I build it on mac os and followed these command env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go mod init go mod vendor go build
this command is working where i build on my local machine ./elasticsearch_exporter --help, but when i moved over on linux machine getting this error -bash: ./elasticsearch_exporter: cannot execute binary file

sonujatav35 · 2021-06-04T01:49:52Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

Unsure if this will help but exact commands:
git clone -b fix-build-go-mod https://github.com/repl-mike-roest/elasticsearch_exporter.git
docker build -t elasticsearch_exporter .

for this also on local macOS machine is showing the option ./elasticsearch_exporter --help, but on linux machine -bash: ./elasticsearch_exporter: cannot execute binary file

jivco · 2021-06-04T12:39:23Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

Unsure if this will help but exact commands:
git clone -b fix-build-go-mod https://github.com/repl-mike-roest/elasticsearch_exporter.git
docker build -t elasticsearch_exporter .
for this also on local macOS machine is showing the option ./elasticsearch_exporter --help, but on linux machine -bash: ./elasticsearch_exporter: cannot execute binary file

Can you execute ldd elasticsearch_exporter on your macOS and the linux machine and paste here the output?

sonujatav35 · 2021-06-08T12:16:32Z

I am also getting the same issue.
How can I download elasticsearch exporter package for linux environment from this branch https://github.com/repl-mike-roest/elasticsearch_exporter.

Unsure if this will help but exact commands:
git clone -b fix-build-go-mod https://github.com/repl-mike-roest/elasticsearch_exporter.git
docker build -t elasticsearch_exporter .
for this also on local macOS machine is showing the option ./elasticsearch_exporter --help, but on linux machine -bash: ./elasticsearch_exporter: cannot execute binary file
Can you execute ldd elasticsearch_exporter on your macOS and the linux machine and paste here the output?

MacOS output:
elasticsearch_exporter:
/usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)

Linux:
[hostname]$ ldd elasticsearch_exporter
not a dynamic executable
[hostname]$

gkulkarni32 · 2021-06-10T05:02:52Z

Hello Team,
I'm also facing a similar issue where some of the critical metrics are not being emitted. for example: elasticsearch.jvm.memory.*
Can you please provide your inputs?

Thanks
Geeta

Blobonat · 2021-06-19T20:22:41Z

Using the elasticsearch-exporter:master version solved the problem for me.

gkulkarni32 · 2021-06-21T07:30:25Z

Using elasticsearch-exporter:master version worked for me too. Can somebody comment on when the actual version is getting released, we use that (example: 1.2.0) in our images.

sysadmind · 2021-06-29T14:12:58Z

v1.2.1 has been released and should resolve these issues. I'm going to close this issue, but if there are more examples of breaking metrics, please let us know.

As per prometheus-community#419, this functionality is failing for elasticsearch version 7.13+ due to a breaking change in the response from elasticsearch. This adds some test fixtures for a range of elasticsearch versions including 7.13 to help ensure that future changes fix the incompatibility with 7.13 and maintain compatibility with versions < 7.13. Signed-off-by: Joe Adams <[email protected]> Signed-off-by: Toomas Ormisson <[email protected]>

As per prometheus-community#419, this functionality is failing for elasticsearch version 7.13+ due to a breaking change in the response from elasticsearch. This adds some test fixtures for a range of elasticsearch versions including 7.13 to help ensure that future changes fix the incompatibility with 7.13 and maintain compatibility with versions < 7.13. Signed-off-by: Joe Adams <[email protected]>

sysadmind mentioned this issue Jun 3, 2021

test: refactor TestNodesStats #427

Merged

adammike mentioned this issue Jun 9, 2021

Release 1.2.0 #431

Merged

gkulkarni32 mentioned this issue Jun 14, 2021

bitnami/elasticsearch chart not emitting metrics with latest 7.13 release bitnami/charts#6653

Closed

0x46616c6b mentioned this issue Jun 15, 2021

fix _/node/stats related metrics #439

Merged

sysadmind closed this as completed Jun 29, 2021

Elasticsearch 7.13 Breaking Metrics #419

Elasticsearch 7.13 Breaking Metrics #419

Comments

mcmarkj commented May 27, 2021

epicvinny commented May 27, 2021

Uh oh!

jivco commented May 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adammike commented Jun 2, 2021

Uh oh!

hansh0801 commented Jun 2, 2021

Uh oh!

egallis31 commented Jun 2, 2021

Uh oh!

jivco commented Jun 2, 2021

Uh oh!

sonujatav35 commented Jun 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hansh0801 commented Jun 3, 2021

Uh oh!

sonujatav35 commented Jun 3, 2021

Uh oh!

hansh0801 commented Jun 3, 2021

Uh oh!

jivco commented Jun 3, 2021

Uh oh!

egallis31 commented Jun 3, 2021

Uh oh!

sonujatav35 commented Jun 4, 2021

Uh oh!

sonujatav35 commented Jun 4, 2021

Uh oh!

jivco commented Jun 4, 2021

Uh oh!

sonujatav35 commented Jun 8, 2021

Uh oh!

gkulkarni32 commented Jun 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Blobonat commented Jun 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gkulkarni32 commented Jun 21, 2021

Uh oh!

sysadmind commented Jun 29, 2021

Uh oh!

jivco commented May 29, 2021 •

edited

Loading

sonujatav35 commented Jun 3, 2021 •

edited

Loading

gkulkarni32 commented Jun 10, 2021 •

edited

Loading

Blobonat commented Jun 19, 2021 •

edited

Loading