Skip to content

[CI] XDocsClientYamlTestSuiteIT test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72} failing #85918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mark-vieira opened this issue Apr 14, 2022 · 10 comments
Closed

[CI] XDocsClientYamlTestSuiteIT test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72} failing #85918

mark-vieira opened this issue Apr 14, 2022 · 10 comments
Assignees
@justincr-elastic
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) Team:Security Meta label for security team >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

This reproduces on Java 19. It seems the query params in the redirect url are ordered differently. This might be some implementation detail in the update JDK. Not sure if we can make this test a bit more robust.

Build scan:
https://gradle-enterprise.elastic.co/s/j7rlne5j7yuri/tests/:x-pack:docs:integTest/org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT/test%20%7Byaml=en%2Frest-api%2Fsecurity%2Foidc-prepare-authentication-api%2Fline_72%7D

Reproduction line:
./gradlew ':x-pack:docs:integTest' --tests "org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72}" -Dtests.seed=B47F42789F9BE697 -Dtests.locale=es-EC -Dtests.timezone=America/Lower_Princes -Druntime.java=19

Applicable branches:
master

Reproduces locally?:
Yes

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Den/rest-api/security/oidc-prepare-authentication-api/line_72%7D

Failure excerpt:

java.lang.AssertionError: Failure at [en/rest-api/security/oidc-prepare-authentication-api:19]: Expected a map containing
redirect: expected "http://127.0.0.1:8080/c2id-login?scope=openid&response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp" but was "http://127.0.0.1:8080/c2id-login?response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp&scope=openid"
   realm: "oidc1"
   state: "XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY"
   nonce: "sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4"

  at __randomizedtesting.SeedInfo.seed([B47F42789F9BE697:3C2B7DA231678B6F]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:493)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:828)

  Caused by: java.lang.AssertionError: Expected a map containing
  redirect: expected "http://127.0.0.1:8080/c2id-login?scope=openid&response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp" but was "http://127.0.0.1:8080/c2id-login?response_type=id_token&redirect_uri=https%3A%2F%2Fmy.fantastic.rp%2Fcb&state=XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY&nonce=sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4&client_id=elasticsearch-rp&scope=openid"
     realm: "oidc1"
     state: "XriW3taOwMYq3QQb-HKEvDBpwAuG4DhSQNZQvJQiZRY"
     nonce: "sDBBaqcSwXbjxeGk09ZE0PBUy3Y-TNajF6HS-yp2b-4"

    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:82)
    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:65)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:94)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:482)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:462)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:828)
@mark-vieira mark-vieira added :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI labels Apr 14, 2022
@elasticmachine elasticmachine added the Team:Security Meta label for security team label Apr 14, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@grcevski
Copy link
Contributor

Happened again here:

https://gradle-enterprise.elastic.co/s/smiwt6ofog6bk/console-log?task=:x-pack:docs:integTest

This was referenced Apr 20, 2022
Closed
Merged
elasticsearchmachine pushed a commit that referenced this issue Apr 20, 2022
Ignore redirect URI in response in oidc-prepare-authentication-api.as…
573b14d 
…ciidoc (#86035)

See ~#86034~ / #85918
@mark-vieira
Copy link
Contributor Author

@Luegg We've added Java 19 testing to the 7.17 branch as well. Do you mind backporting #86035 to 7.17 to address these failures?

@dimitris-athanasiou
Copy link
Contributor

And another one here: https://gradle-enterprise.elastic.co/s/b32fonqbgov6w

@fcofdez
Copy link
Contributor

fcofdez commented Jul 27, 2022

There are a couple of related failures:

@justincr-elastic justincr-elastic self-assigned this Jul 27, 2022
@justincr-elastic
Copy link
Contributor

justincr-elastic commented Jul 27, 2022 •
edited
Loading

What version of Java 19 is being used?

This is from the Gradle scan for the CI test failure.

=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 4.15.0-1098-gcp (amd64)
  Runtime JDK Version   : 19-ea (Oracle)
  Runtime java.home     : /var/lib/jenkins/.java/openjdk19
  Gradle JDK Version    : 17.0.2 (Oracle)
  Gradle java.home      : /var/lib/jenkins/.java/openjdk-17.0.2+8-linux
  Random Testing Seed   : B47F42789F9BE697
  In FIPS 140 mode      : false
=======================================

I ran the reproduce command from that CI failure, but I used OpenJDK EA 19 b32 and the test passed.

./gradlew ':x-pack:docs:yamlRestTest' --tests "org.elasticsearch.smoketest.XDocsClientYamlTestSuiteIT.test {yaml=en/rest-api/security/oidc-prepare-authentication-api/line_72}" -Dtests.seed=B47F42789F9BE697 -Dtests.locale=es-EC -Dtests.timezone=America/Lower_Princes -Druntime.java=19

This is from the Gradle scan for my local build which passed.

======================================= |  
| Elasticsearch Build Hamster says Hello! |  
| Gradle Version        : 7.5 |  
| OS Info               : Linux 5.15.0-41-generic (amd64) |  
| Runtime JDK Version   : 19-ea (Oracle, 19-ea+32-2220) |  
| Runtime java.home     : /home/q/.jdks/openjdk-19b32 |  
| Gradle JDK Version    : 18.0.2 (Oracle) |  
| Gradle java.home      : /home/q/.jdks/openjdk-18.0.2 |  
| Random Testing Seed   : B47F42789F9BE697 |  
| In FIPS 140 mode      : false |  
| =======================================

BUILD SUCCESSFUL in 51s

@grcevski
Copy link
Contributor

Based on the raw logs from the run, it appears to be Oracle Corporation 19-ea [OpenJDK 64-Bit Server VM 19-ea+32-2220]. Looks the same... maybe intermittent issue.

@justincr-elastic
Copy link
Contributor

I don't see b32 mentioned in any of the raw logs for the CI failure links provided here:

I only see b32 mentioned in the raw log for my local build:

I am wondering if CI tests are using jdk-19-b0, hence no build number? If so, I wonder if updating to jdk-19-b32 could resolve these issues. OpenJDK 19 EA release download page only seems to provide the latest b32 build, so I would need to try and find an earlier copy of the b0 build.

@mark-vieira
Copy link
Contributor Author

@justincr-elastic as mentioned in this comment this issue was addressed in main but not backported to 7.17. The failures happening now are only on that branch. For example: https://gradle-enterprise.elastic.co/s/2vmaxj63zozeu

justincr-elastic added a commit to justincr-elastic/elasticsearch that referenced this issue Jul 27, 2022
@justincr-elastic
Backport OIDC test fix from 8.x elastic#86035 to 7.17
09fcbed 
Closes elastic#85918
@justincr-elastic justincr-elastic mentioned this issue Jul 27, 2022
Merged
justincr-elastic added a commit that referenced this issue Aug 3, 2022
@justincr-elastic
Backport OIDC test fix from 8.x #86035 to 7.17 (#88868)
246378b 
Closes #85918
@justincr-elastic
Copy link
Contributor

#88868 backported a fix from 8.x to 7.x to fix this CI failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@justincr-elastic justincr-elastic

Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) Team:Security Meta label for security team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@fcofdez @mark-vieira @grcevski @dimitris-athanasiou @elasticmachine @justincr-elastic