Skip to content

[CI] ForecastIT#testSingleSeries fails reproducibly on master #36258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cbuescher opened this issue Dec 5, 2018 · 6 comments · Fixed by #37110
Closed

[CI] ForecastIT#testSingleSeries fails reproducibly on master #36258

cbuescher opened this issue Dec 5, 2018 · 6 comments · Fixed by #37110
Assignees
@droberts195
Labels
:ml Machine learning >test-failure Triaged test failures from CI

Comments

@cbuescher
Copy link
Member

First seen in this PR build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-2/1289/consoleFull

This reproduces for me on master:

./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner -Dtests.seed=3B7BFF05D7416702 -Dtests.class=org.elasticsearch.xpack.ml.integration.ForecastIT -Dtests.method="testSingleSeries" -Dtests.security.manager=true -Dtests.locale=fr -Dtests.timezone=Cuba -Dcompiler.java=11 -Druntime.java=8

Failure:

> Throwable #1: java.lang.AssertionError:
   > Expected: <1544007600000L>
   >      but: was <1544009400000L>
   >    at __randomizedtesting.SeedInfo.seed([3B7BFF05D7416702:5C63C1FD2F7DCD17]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.ml.integration.ForecastIT.testSingleSeries(ForecastIT.java:103)
   >    at java.lang.Thread.run(Thread.java:748)
@cbuescher cbuescher added >test-failure Triaged test failures from CI :ml Machine learning labels Dec 5, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

@cbuescher
Copy link
Member Author

I haven't found many recent failures like this in the CI mails so far, so I guess its only this specific seeds problem and I'm keeping the test unmuted for now.

@droberts195
Copy link
Contributor

I think this is probably a side effect of elastic/ml-cpp#327 and will cause a lot of failures. I will mute the test.

droberts195 added a commit that referenced this issue Dec 5, 2018
@droberts195
Mute ForecastIT.testSingleSeries
a3c1c69 
Due to #36258
droberts195 added a commit that referenced this issue Dec 5, 2018
@droberts195
Mute ForecastIT.testSingleSeries
62e5a0c 
Due to #36258
droberts195 added a commit that referenced this issue Dec 5, 2018
@droberts195
Mute ForecastIT.testSingleSeries
20fed29 
Due to #36258
@droberts195
Copy link
Contributor

Muting commits are:

@hendrikmuhs
Copy link

elastic/ml-cpp#327 changed the bucket timestamps the forecast is created, before it was at the beginning of a bucket, now it is in the middle.

This perfectly explains the error: 1544009400000 is 30 minutes after 1544007600000 and the bucket span of the test is 1 hour.

The test needs to be adjusted.

@hendrikmuhs hendrikmuhs self-assigned this Dec 5, 2018
@romseygeek romseygeek mentioned this issue Dec 5, 2018
Merged
@hendrikmuhs hendrikmuhs mentioned this issue Dec 6, 2018
Closed
@jdconrad
Copy link
Contributor

jdconrad commented Dec 6, 2018 •
edited
Loading

A few more:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/305/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=debian/101/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/101/console

So 6.x also.

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Jan 3, 2019
@droberts195
[TEST] Revert "Mute ForecastIT.testSingleSeries"
27d73f2 
The problem that caused the test to be muted was fixed in
elastic/ml-cpp#332

Closes elastic#36258
@droberts195 droberts195 mentioned this issue Jan 3, 2019
Merged
droberts195 added a commit that referenced this issue Jan 3, 2019
@droberts195
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
13649aa 
The problem that caused the test to be muted was fixed in
elastic/ml-cpp#332

Closes #36258
droberts195 added a commit that referenced this issue Jan 3, 2019
@droberts195
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
36ab06f 
The problem that caused the test to be muted was fixed in
elastic/ml-cpp#332

Closes #36258
droberts195 added a commit that referenced this issue Jan 3, 2019
@droberts195
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
b89786a 
The problem that caused the test to be muted was fixed in
elastic/ml-cpp#332

Closes #36258
droberts195 added a commit that referenced this issue Jan 3, 2019
@droberts195
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
1c41637 
The problem that caused the test to be muted was fixed in
elastic/ml-cpp#332

Closes #36258
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@droberts195 droberts195

Labels
:ml Machine learning >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
5 participants
@jdconrad @hendrikmuhs @droberts195 @cbuescher @elasticmachine