[CI] SSLTrustRestrictionsTests #29989

elasticmachine · 2017-12-12T03:30:19Z

Original comment by @dnhatn:

CI: LINK REDACTED
Log: LINK REDACTED

I could not reproduce this.

  2> REPRODUCE WITH: gradle :x-pack-elasticsearch:plugin:test -Dtests.seed=DD960C449909111C -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests -Dtests.method="testCertificateWithTrustedNameIsAccepted" -Dtests.security.manager=true -Dtests.locale=sk -Dtests.timezone=America/Matamoros
  1>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
  1>  at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416) ~[?:?]
  1>  at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1301) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  1>  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1214) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  1>  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  1>  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
  1>  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
  1>  ... 15 more
  1> [2017-12-11T14:14:17,705][INFO ][o.e.x.s.SSLTrustRestrictionsTests] [SSLTrustRestrictionsTests#testCertificateWithTrustedNameIsAccepted]: cleaning up after test
  1> [2017-12-11T14:14:17,771][DEBUG][o.e.x.s.RestrictedTrustManager] Configured with trust restrictions: [{trustedNames=[*.trusted]}]
  1> [2017-12-11T14:14:17,773][DEBUG][o.e.x.s.RestrictedTrustManager] Configured with trust restrictions: [{trustedNames=[*.trusted]}]
  1> [2017-12-11T14:14:17,776][DEBUG][o.e.x.s.RestrictedTrustManager] Configured with trust restrictions: [{trustedNames=[*.trusted]}]
  1> [2017-12-11T14:14:17,777][INFO ][o.e.x.s.SSLConfigurationReloader] [node_s0] reloaded [REDACTED/trust_restrictions.yml] and updated ssl contexts using this file
  1> [2017-12-11T14:14:17,784][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,789][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,798][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,807][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,821][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,836][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,840][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,848][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,852][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,858][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,862][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,866][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,882][DEBUG][o.e.x.s.RestrictedTrustManager] Name [node.trusted] matches trusted pattern [*.trusted]
  1> [2017-12-11T14:14:17,882][DEBUG][o.e.x.s.RestrictedTrustManager] Trusting certificate [CN=trusted] [49c2b0a209e72d22c6197ac942930144c3a396ec] with common-names [[node.trusted]]
  1> [2017-12-11T14:14:17,885][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Starting template upgrade to version 6.1.0, 1 templates will be updated and 0 will be removed
  1> [2017-12-11T14:14:17,893][INFO ][o.e.c.m.TemplateUpgradeService] [node_s0] Finished upgrading templates to version 6.1.0
  1> [2017-12-11T14:14:17,893][INFO ][o.e.x.s.SSLTrustRestrictionsTests] [SSLTrustRestrictionsTests#testCertificateWithTrustedNameIsAccepted]: cleaned up after test
  1> [2017-12-11T14:14:17,894][INFO ][o.e.x.s.SSLTrustRestrictionsTests] [testCertificateWithTrustedNameIsAccepted]: after test
FAILURE 2.32s J0 | SSLTrustRestrictionsTests.testCertificateWithTrustedNameIsAccepted <<< FAILURES!
  2> NOTE: leaving temporary files on disk at: REDACTED
  2> Dec 11, 2017 8:14:18 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
  2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=sk, timezone=America/Matamoros
  2> NOTE: Linux 4.4.62-18.6-default amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=4,threads=1,free=163792792,total=521142272
   > Throwable LINK REDACTED: java.lang.AssertionError: handshake should have been successful, but failed with javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
   >  at __randomizedtesting.SeedInfo.seed([DD960C449909111C:89D6F9E09E1F0075]:0)
  2> NOTE: All tests run in this JVM: [AuthenticationServiceTests, SecurityIndexSearcherWrapperIntegrationTests, ValidateJobConfigActionRequestTests, SlackActionTests, AnalyzeTests, TokenSSLBootsrapCheckTests, LicenseServiceClusterTests, HttpExportBulkResponseListenerTests, WatchRequestValidationTests, SecurityCachePermissionTests, HasPrivilegesRequestBuilderTests, ExportersTests, HistoryTemplateSearchInputMappingsTests, ActiveDirectoryGroupsResolverTests, GetFiltersActionRequestTests, CertificateToolTests, RuleActionTests, XPackUserTests, MultipleAdRealmTests, MapPathTests, XContentRecordReaderTests, XPackExtensionSecurityTests, SslHostnameVerificationTests, RestGetTokenActionTests, JiraActionTests, NodeFailureListenerTests, HttpEmailAttachementParserTests, ProcessCtrlTests, MachineLearningTests, SizeLimitInputStreamTests, GetCategoriesRequestTests, DateFormatDateTransformerTests, HttpClientTests, GraphTests, IndicesStatsMonitoringDocTests, NativeUserRoleMapperTests, UpdateModelSnapshotActionResponseTests, ReadActionsTests, ProfileTests, InputRegistryTests, ExecutableChainInputTests, TransportGetUsersActionTests, LicenseOperationModeTests, UpgradeToTrialTests, YearlyScheduleTests, PutJobActionResponseTests, IpFilterRemoteAddressFilterTests, ForecastTests, PersistentTasksNodeServiceStatusTests, PeriodThrottlerTests, NormalizerResultTests, FileUserRolesStoreTests, AuditorTests, PkiOptionalClientAuthTests, PersistentTasksClusterServiceTests, DataCountsReporterTests, FieldDataCacheWithFieldSubsetReaderTests, WatcherIndexTemplateRegistryTests, BootStrapTests, IndexRecoveryMonitoringDocTests, ModelPlotTests, BucketInfluencerNormalizableTests, HttpHostBuilderTests, IntervalsTests, ForecastJobActionResponseTests, HistoryTemplateEmailMappingsTests, HourlyScheduleTests, SSLTrustRestrictionsTests]
   >  at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.testCertificateWithTrustedNameIsAccepted(SSLTrustRestrictionsTests.java:158)
   >  at java.lang.Thread.run(Thread.java:748)

The text was updated successfully, but these errors were encountered:

ywelsch · 2018-06-01T14:41:02Z

Failed again here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/1801/console

jkakavas · 2018-06-01T14:56:36Z

Still doesn't reproduce locally, the errors in the failed build were :

14:34:56 ERROR   2.30s J0 | SSLTrustRestrictionsTests.testCertificateWithTrustedNameIsAccepted <<< FAILURES!
14:34:56    > Throwable #1: java.lang.AssertionError: handshake should have been successful, but failed with java.net.SocketException: Broken pipe (Write failed)
14:34:56    > 	at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.testCertificateWithTrustedNameIsAccepted(SSLTrustRestrictionsTests.java:163)
14:34:56    > 	at java.lang.Thread.run(Thread.java:748)Throwable #2: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{_cFfXO0rR1GeO-P4u6OtWw}{127.0.0.1}{127.0.0.1:30180}]]
14:34:56    > 	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
14:34:56    > 	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
14:34:56    > 	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60)
14:34:56    > 	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:373)
14:34:56    > 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405)
14:34:56    > 	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394)
14:34:56    > 	at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1247)
14:34:56    > 	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46)
14:34:56    > 	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53)
14:34:56    > 	at org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked(ElasticsearchAssertions.java:114)
14:34:56    > 	at org.elasticsearch.test.TestCluster.wipeIndices(TestCluster.java:141)
14:34:56    > 	at org.elasticsearch.test.TestCluster.wipe(TestCluster.java:78)
14:34:56    > 	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:577)
14:34:56    > 	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2081)
14:34:56    > 	at java.lang.Thread.run(Thread.java:748)

jdconrad · 2018-07-26T17:39:33Z

@tvernum I hope you don't mind me updating this issue to add these test failures as well. It didn't seem worth creating a separate issue for each.

org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests testCertificateWithUntrustedNameFails
org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests testRestrictionsAreReloaded

Failure Link:
(https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/200/console)

Reproduce with:

REPRODUCE WITH:

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=E88D976EA1519B74 \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testCertificateWithUntrustedNameFails" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-SN \
  -Dtests.timezone=PST8PDT

REPRODUCE WITH:

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=E88D976EA1519B74 \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testRestrictionsAreReloaded" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-SN \
  -Dtests.timezone=PST8PDT

danielmitterdorfer · 2018-07-31T06:57:13Z

We have another instance of these failures in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/208/console

with the following reproduction lines:

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=28197402351DB64D \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testRestrictionsAreReloaded" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-MP \
  -Dtests.timezone=Africa/Dakar

failed with:

ERROR   1.65s J1 | SSLTrustRestrictionsTests.testRestrictionsAreReloaded <<< FAILURES!
   > Throwable #1: javax.net.ssl.SSLException: readRecord
   >    at __randomizedtesting.SeedInfo.seed([28197402351DB64D:ED9D0A8C67A18C2F]:0)
   >    at java.base/sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:819)
   >    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:383)
   >    at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.tryConnect(SSLTrustRestrictionsTests.java:223)
   >    at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.lambda$testRestrictionsAreReloaded$2(SSLTrustRestrictionsTests.java:189)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:819)
   >    at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.testRestrictionsAreReloaded(SSLTrustRestrictionsTests.java:187)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)
   > Caused by: java.net.SocketException: Broken pipe (Write failed)
   >    at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
   >    at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
   >    at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
   >    at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:227)
   >    at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:81)
   >    at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:749)
   >    at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:429)
   >    at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:173)
   >    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:390)
   >    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445)
   >    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
   >    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:178)
   >    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
   >    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:877)
   >    at java.base/sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:810)

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=28197402351DB64D \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testCertificateWithUntrustedNameFails" \
  -Dtests.security.manager=true \
  -Dtests.locale=en-MP \
  -Dtests.timezone=Africa/Dakar

failed with

FAILURE 0.31s J1 | SSLTrustRestrictionsTests.testCertificateWithUntrustedNameFails <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: handshake should have failed, but was successful
   >    at __randomizedtesting.SeedInfo.seed([28197402351DB64D:3D61E9EF63673AFE]:0)
   >    at org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests.testCertificateWithUntrustedNameFails(SSLTrustRestrictionsTests.java:179)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >    at java.base/java.lang.Thread.run(Thread.java:834)

Both did not reproduce locally (JAVA_HOME: Java 10.0.2+13, RUNTIME_JAVA_HOME: Java 11-ea+22).

danielmitterdorfer · 2018-07-31T14:14:56Z

And we have another one in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/209/console

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=B0DEA59D380A135F \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testCertificateWithUntrustedNameFails" \
  -Dtests.security.manager=true \
  -Dtests.locale=guz-KE \
  -Dtests.timezone=Etc/GMT+5

and

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=B0DEA59D380A135F \
  -Dtests.class=org.elasticsearch.xpack.ssl.SSLTrustRestrictionsTests \
  -Dtests.method="testRestrictionsAreReloaded" \
  -Dtests.security.manager=true \
  -Dtests.locale=guz-KE \
  -Dtests.timezone=Etc/GMT+5

As the newer failures all appeared on Java 11, I'm gonna label this as Java 11 issue as well. In addition, I'll mute those tests on Java 11.

This commit adds an assumption to two test methods in SSLTrustRestrictionsTests that we are not on JDK 11 as the tests currently fail there. Relates #29989

danielmitterdorfer · 2018-07-31T14:44:39Z

The tests are muted on master in 5f30258 and on 6.x in 5b96abe.

In prior versions of Java, we expected to see a SSLHandshakeException when starting a handshake with a server that we do not trust. In JDK11, the exception has changed to a SSLException, which SSLHandshakeException extends. This is most likely a side effect of the TLS 1.3 changes in JDK11. This change updates the test to catch the SSLException instead of the SSLHandshakeException and enables the test to work on JDK8 through JDK11. Closes elastic#29989

In prior versions of Java, we expected to see a SSLHandshakeException when starting a handshake with a server that we do not trust. In JDK11, the exception has changed to a SSLException, which SSLHandshakeException extends. This is most likely a side effect of the TLS 1.3 changes in JDK11. This change updates the test to catch the SSLException instead of the SSLHandshakeException and enables the test to work on JDK8 through JDK11. Closes #29989

elasticmachine added :Security/TLS SSL/TLS, Certificates >test Issues or PRs that are addressing/adding tests >test-failure Triaged test failures from CI labels Apr 25, 2018

elasticmachine assigned tvernum Apr 25, 2018

$@polyfractal$ polyfractal removed the >test Issues or PRs that are addressing/adding tests label May 9, 2018

tvernum mentioned this issue Jun 4, 2018

[TEST] Make SSL restrictions update atomic #31050

Merged

jdconrad changed the title ~~[CI] SSLTrustRestrictionsTests#testCertificateWithTrustedNameIsAccepted~~ [CI] SSLTrustRestrictionsTests Jul 26, 2018

jdconrad added the v7.0.0 label Jul 26, 2018

danielmitterdorfer added the jdk11 label Jul 31, 2018

danielmitterdorfer added a commit that referenced this issue Jul 31, 2018

Mute SSLTrustRestrictionsTests on JDK 11

5f30258

This commit adds an assumption to two test methods in SSLTrustRestrictionsTests that we are not on JDK 11 as the tests currently fail there. Relates #29989

danielmitterdorfer added a commit that referenced this issue Jul 31, 2018

Mute SSLTrustRestrictionsTests on JDK 11

5b96abe

This commit adds an assumption to two test methods in SSLTrustRestrictionsTests that we are not on JDK 11 as the tests currently fail there. Relates #29989

jaymode mentioned this issue Sep 28, 2018

Update SSLTrustRestrictionTests for JDK11 #34131

Merged

jaymode closed this as completed in #34131 Sep 28, 2018

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI] SSLTrustRestrictionsTests #29989

[CI] SSLTrustRestrictionsTests #29989

elasticmachine commented Dec 12, 2017

ywelsch commented Jun 1, 2018

Uh oh!

jkakavas commented Jun 1, 2018

Uh oh!

jdconrad commented Jul 26, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!

[CI] SSLTrustRestrictionsTests #29989

[CI] SSLTrustRestrictionsTests #29989

Comments

elasticmachine commented Dec 12, 2017

ywelsch commented Jun 1, 2018

Uh oh!

jkakavas commented Jun 1, 2018

Uh oh!

jdconrad commented Jul 26, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!

danielmitterdorfer commented Jul 31, 2018

Uh oh!