Skip to content

[CI] ActiveDirectorySessionFactoryTests classMethod failing #105401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
edsavage opened this issue Feb 12, 2024 · 11 comments
Closed

[CI] ActiveDirectorySessionFactoryTests classMethod failing #105401

edsavage opened this issue Feb 12, 2024 · 11 comments
Labels
:Delivery/Build Build or test infrastructure low-risk An open issue or test failure that is a low risk to future releases Team:Delivery Meta label for Delivery team >test-failure Triaged test failures from CI

Comments

@edsavage
Copy link
Contributor

Seems docker related

Build scan:
https://gradle-enterprise.elastic.co/s/gkulbnzrdipfg/tests/:x-pack:qa:third-party:active-directory:test/org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactoryTests

Reproduction line:

null

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactoryTests#classMethod

Failure excerpt:

org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=<resolving>, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@27332164)

  at __randomizedtesting.SeedInfo.seed([7FA184586C7298D8]:0)
  at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1367)
  at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
  at org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:67)
  at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
  at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not build image: failed to get digest sha256:cc95a733c5d8e97e125081874caaaf90cc1e1bb5d68b2af165a989cc7673b7aa: open /var/lib/docker/image/overlay2/imagedb/content/sha256/cc95a733c5d8e97e125081874caaaf90cc1e1bb5d68b2af165a989cc7673b7aa: no such file or directory

    at com.github.dockerjava.api.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:78)
    at com.github.dockerjava.api.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:50)
    at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:159)
    at org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:40)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
    at org.testcontainers.shaded.com.google.common.util.concurrent.Futures$1.get(Futures.java:536)
    at org.testcontainers.images.RemoteDockerImage.getImageName(RemoteDockerImage.java:130)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:67)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:28)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1365)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
    at org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:67)
    at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
    at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1583)
@edsavage edsavage added :Search/Search Search-related issues that do not fall into other categories >test-failure Triaged test failures from CI labels Feb 12, 2024
@elasticsearchmachine elasticsearchmachine added blocker Team:Search Meta label for search team labels Feb 12, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@benwtrent benwtrent added :Delivery/Build Build or test infrastructure :Security/Security Security issues without another label and removed :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team labels Feb 12, 2024
@benwtrent
Copy link
Member

Not sure why this was tagged as :search/search

The failure was due to a docker image failing to download and if it wasn't due to that, it the test is located within the security plugin.

Tagging delivery & security.

@elasticsearchmachine elasticsearchmachine added Team:Security Meta label for security team Team:Delivery Meta label for Delivery team labels Feb 12, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-security (Team:Security)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

@mark-vieira
Copy link
Contributor

So far only a single occurrence. We'll keep an eye on this.

@benwtrent
Copy link
Member

eerily similar failure here: https://gradle-enterprise.elastic.co/s/zlqmdntkflpbs

active-directory test failure due to docker failure.

@slobodanadamovic slobodanadamovic added low-risk An open issue or test failure that is a low risk to future releases and removed blocker labels Apr 23, 2024
@slobodanadamovic
Copy link
Contributor

slobodanadamovic commented Apr 23, 2024 •
edited
Loading

There have been only 2 failures in the past 2 months and in the past 1 month 0 failures. Hence I marked this issue as low-risk, since this feels like a temporary issue.

@mark-vieira
Copy link
Contributor

Closing since there have been no failures in the last month.

@slobodanadamovic
Copy link
Contributor

Just found two recent failures with the same issue but for different tests:

@slobodanadamovic
Copy link
Contributor

We still see occasional failures for GroupMappingIT and ActiveDirectoryRunAsIT integration test:

This still looks like a transient error, but it might as well be a race condition!?

GroupMappingIT > classMethod FAILED
    org.testcontainers.containers.ContainerLaunchException: Container startup failed for image es-smb-fixture:latest
        at __randomizedtesting.SeedInfo.seed([ABF161D102ABA71E]:0)
        at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:362)
        at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333)
        at app//org.elasticsearch.test.fixtures.testcontainers.DockerEnvironmentAwareTestContainer.start(DockerEnvironmentAwareTestContainer.java:68)
        at app//org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1085)
        at app//org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at [email protected]/java.lang.Thread.run(Thread.java:1570)

        Caused by:
        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=<resolving>, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@1c350e95)
            at app//org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1367)
            at app//org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:674)
            at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
            ... 21 more

            Caused by:
            com.github.dockerjava.api.exception.DockerClientException: Could not build image: failed to get digest sha256:a543b761e853ed2141667038347abfe5b0683c620183d5ed320595cc83b254ed: open /var/lib/docker/image/overlay2/imagedb/content/sha256/a543b761e853ed2141667038347abfe5b0683c620183d5ed320595cc83b254ed: no such file or directory
                at app//com.github.dockerjava.api.command.BuildImageResultCallback.getImageId(BuildImageResultCallback.java:78)
                at app//com.github.dockerjava.api.command.BuildImageResultCallback.awaitImageId(BuildImageResultCallback.java:50)
                at app//org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:159)
                at app//org.testcontainers.images.builder.ImageFromDockerfile.resolve(ImageFromDockerfile.java:40)
                at app//org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at app//org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
                at app//org.testcontainers.shaded.com.google.common.util.concurrent.Futures$1.get(Futures.java:536)
                at app//org.testcontainers.images.RemoteDockerImage.getImageName(RemoteDockerImage.java:130)
                at app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:67)
                at app//org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:28)
                at app//org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
                at app//org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
                at app//org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1365)
                ... 23 more

This was referenced Oct 15, 2024
Closed
Closed
@slobodanadamovic slobodanadamovic removed :Security/Security Security issues without another label Team:Security Meta label for security team labels Nov 4, 2024
@mark-vieira
Copy link
Contributor

Duplicate of #117415

@mark-vieira mark-vieira marked this as a duplicate of #117415 Dec 27, 2024
@slobodanadamovic slobodanadamovic mentioned this issue Feb 3, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Delivery/Build Build or test infrastructure low-risk An open issue or test failure that is a low risk to future releases Team:Delivery Meta label for Delivery team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mark-vieira @benwtrent @slobodanadamovic @edsavage @elasticsearchmachine