Skip to content

Gradle task avoidance api should be used consistently in the build #56610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
breskeby opened this issue May 12, 2020 · 1 comment · Fixed by #66738
Closed

Gradle task avoidance api should be used consistently in the build #56610

breskeby opened this issue May 12, 2020 · 1 comment · Fixed by #66738
Assignees
Labels
:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team

Comments

@breskeby
Copy link
Contributor

breskeby commented May 12, 2020

At several places in the gradle build we still see usage of the old task creation api. This should be replaced by using the task avoidance api of gradle (tasks.register(...){}).

This makes the build more consistent and eventually gives us slightly better configuration time. Though this step alone might not result in many task creations being avoided yet as this also requires some further optimisations in how existing tasks are requested within the build scripts and different plugins. This will addressed later on in a different issue.

Update: Many tasks have been converted. Two major remainings:

  • RestIntegTest tasks are created eagerly. Tasks of type RestIntegTest created during task graph calculation causes
  1. the creation of clusters late after all build scripts have been evaluated.
  2. distributions late in the build after the current setupDistribution (see https://github.com/elastic/elasticsearch/blob/master/buildSrc/src/main/java/org/elasticsearch/gradle/DistributionDownloadPlugin.java#L82)

One fix for 1) would be, to configure test clusters that are created by a RestIntegTest in the context of the task like:

tasks.named('integTest').configure {
   testclusters.integTest {
       // test cluster configuration
   }
}

The benefit of this is that the test cluster configuration would only happen if the cluster is required by the build.

Fixing 2) is more complex as we need to find a different solution for delaying the distribution setup. We can probably replace the afterEvaluate usage by using configuration.defaultDependencies{} hooks

@breskeby breskeby added the :Delivery/Build Build or test infrastructure label May 12, 2020
@breskeby breskeby self-assigned this May 12, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Build)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label May 12, 2020
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jul 30, 2020
- One step closer to elastic#56610
- Still many tasks are created during configuration phase. Tackled in separate steps
breskeby added a commit that referenced this issue Jul 31, 2020
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jul 31, 2020
…#60071)

- Replace immediate task creations by using task avoidance api
- One step closer to elastic#56610
- Still many tasks are created during configuration phase. Tackled in separate steps
breskeby added a commit that referenced this issue Jul 31, 2020
…#60504)

- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
breskeby added a commit that referenced this issue Nov 11, 2020
- Avoid to rely on afterEvaluate lifecycle hook in DistributionDownloadPlugin
- Allows us to define test cluster only in the task creation which can be later than afterevaluate 
which allows us to move tasks that rely on test clusters using task avoidance api.
- This should result in test clusters only being defined when actually used in the build
- This unblocks further progress on #56610
breskeby added a commit to breskeby/elasticsearch that referenced this issue Nov 11, 2020
- Avoid to rely on afterEvaluate lifecycle hook in DistributionDownloadPlugin
- Allows us to define test cluster only in the task creation which can be later than afterevaluate
which allows us to move tasks that rely on test clusters using task avoidance api.
- This should result in test clusters only being defined when actually used in the build
- This unblocks further progress on elastic#56610
@mark-vieira mark-vieira added Team:Delivery Meta label for Delivery team and removed Team:Core/Infra Meta label for core/infra team labels Nov 11, 2020
breskeby added a commit that referenced this issue Nov 12, 2020
- Avoid to rely on afterEvaluate lifecycle hook in DistributionDownloadPlugin
- Allows us to define test cluster only in the task creation which can be later than afterevaluate
which allows us to move tasks that rely on test clusters using task avoidance api.
- This should result in test clusters only being defined when actually used in the build
- This unblocks further progress on #56610
breskeby added a commit that referenced this issue Jan 4, 2021
This finishes porting all tasks created in gradle build scripts and plugins to use 
the task avoidance api (see #56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
breskeby added a commit to breskeby/elasticsearch that referenced this issue Jan 4, 2021
This finishes porting all tasks created in gradle build scripts and plugins to use
the task avoidance api (see elastic#56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
breskeby added a commit that referenced this issue Jan 4, 2021
This finishes porting all tasks created in gradle build scripts and plugins to use
the task avoidance api (see #56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Delivery/Build Build or test infrastructure Team:Delivery Meta label for Delivery team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants