Add gradle task to build platform specific distro #56003

breskeby · 2020-04-30T08:42:20Z

This adds the gradle task ':installLocalDistribution' that installs a distribution locally matching the local architecture.

Closes #53682

elasticmachine · 2020-04-30T08:42:22Z

Pinging @elastic/es-core-infra (:Core/Infra/Build)

mark-vieira

I think we want to leverage our DistributionDownloadPlugin here. The plugin has all sort of smarts in it already to figure out how to resolve a requested distribution type to a project dependency or download. So I'd recommend we go that route so we don't have to duplicate the logic of which project to delegate to depending on the current os (that lives in the plugin already). Also, we'll probably want this task to extract the distro to some standard location. That way we can easily document that you can run this task, then look in /extracted/dir to find our distro. As it is, the build artifact is going to live in the build dir for the specific platform project, which is kind of confusing.

So let's do this:

Create a new script plugin for this and apply to the root project so we don't bloat the root build.gradle any further.
Apply the elasticsearch.distribution-download plugin.
Configure an ElasticsearchDistribution for the current platform, with type = 'archive' and flavor = 'default'.
Wire that distribution up to a copy task that copy the extracted distro to a known location see ElasticsearchDistribution.getExtracted()`.

breskeby · 2020-04-30T18:35:02Z

Just to get a better understanding, why would we package and unpackage instead of just having a plain copy task that might share the same copy spec with the distro archive tasks?

…

On 30. Apr 2020, at 20:25, Mark Vieira ***@***.***> wrote: @mark-vieira commented on this pull request. I think we want to leverage our DistributionDownloadPlugin here. The plugin has all sort of smarts in it already to figure out how to resolve a requested distribution type to a project dependency or download. So I'd recommend we go that route so we don't have to duplicate the logic of which project to delegate to depending on the current os (that lives in the plugin already). Also, we'll probably want this task to extract the distro to some standard location. That way we can easily document that you can run this task, then look in /extracted/dir to find our distro. As it is, the build artifact is going to live in the build dir for the specific platform project, which is kind of confusing. So let's do this: Create a new script plugin for this and apply to the root project so we don't bloat the root build.gradle any further. Apply the elasticsearch.distribution-download plugin. Configure an ElasticsearchDistribution for the current platform, with type = 'archive' and flavor = 'default'. Wire that distribution up to a copy task that copy the extracted distro to a known location see ElasticsearchDistribution.getExtracted()`. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

mark-vieira · 2020-04-30T18:40:38Z

Just to get a better understanding, why would we package and unpackage instead of just having a plain copy task that might share the same copy spec with the distro archive tasks?

That's a fantastic question 😉 . Mainly because we just simply haven't optimized for that yet but it is very much worth doing as tar/untar adds considerable overhead. Right now we are basically just doing this all via artifact dependencies, so we grab the archive artifact, then unpack it. It'll need some work to make it more efficient.

This adds the gradle task ':assemblePlatformArchive' that depends on the platform specific distro archive tasks and allows using the same task for building only the archive according to the local platform. This delegates to :distribution:archives:windows-zip:assemble on windows :distribution:archives:linux-tar:assemble on linux :distribution:archives:darwin-tar:assemble on osx Closes #53682

- reuse distribution download plugin - rename task to installLocalDistribution

breskeby · 2020-05-11T12:28:09Z

@mark-vieira I applied your feedback

mark-vieira

LGTM 👍

@rjernst want to contribute to some bike-shedding about naming here 😉

rjernst · 2020-05-11T18:44:43Z

gradle/local-distribution.gradle

+  }
+}
+
+tasks.register('installLocalDistribution', Copy) {


perhaps we could make this easier to type as just localDistro? Should we also use Sync instead of Copy? Can't older files that were removed be left in place with Copy?

localDistro it is. And switched to use Sync

rjernst · 2020-05-11T18:45:25Z

gradle/local-distribution.gradle

+tasks.register('installLocalDistribution', Copy) {
+  from(elasticsearch_distributions.local.extracted)
+  into("build/distribution/local")
+}


I think we should add a println that states the location of the expanded distribution

Good call, let's add a user friendly doLast which prints something like Elasticsearch distribution installed to {location}.

pugnascotia · 2020-05-11T20:16:14Z

gradle/local-distribution.gradle

+ */
+
+/**
+ * This script sets up a local distribution including.


Seems to be some words missing here.

fixed the comment

- rename task name to localDistro - use Sync instead of Copy task - add user friendly output to point user to right directory

breskeby · 2020-05-11T20:24:59Z

Example output https://gradle-enterprise.elastic.co/s/r7iukym2sivi4/console-log#L786

rjernst

LGTM

* Add gradle task 'localDistro' to build platform specific distro * reuse distribution download plugin Closes elastic#53682

backports #56003 to

backports #56003 to 7.8

mark-vieira · 2020-05-12T14:14:21Z

@breskeby FYI, I added the v7.9.0 label, corresponding to the 7.x branch.

backports #56003 to 7.7

breskeby added >enhancement :Delivery/Build Build or test infrastructure labels Apr 30, 2020

breskeby requested a review from mark-vieira April 30, 2020 08:42

breskeby self-assigned this Apr 30, 2020

elasticmachine added the Team:Core/Infra Meta label for core/infra team label Apr 30, 2020

mark-vieira reviewed Apr 30, 2020

View reviewed changes

breskeby mentioned this pull request May 5, 2020

Move bwcVersions extension property to BuildParams #56206

Merged

breskeby marked this pull request as draft May 7, 2020 10:24

breskeby added 2 commits May 7, 2020 14:07

Apply review feedback

1ccf383

- reuse distribution download plugin - rename task to installLocalDistribution

breskeby force-pushed the fix/53682 branch from a2f28b3 to 1ccf383 Compare May 7, 2020 12:07

breskeby marked this pull request as ready for review May 11, 2020 12:26

mark-vieira approved these changes May 11, 2020

View reviewed changes

rjernst reviewed May 11, 2020

View reviewed changes

pugnascotia reviewed May 11, 2020

View reviewed changes

breskeby added 2 commits May 11, 2020 22:17

Apply more feedback

d6c048c

- rename task name to localDistro - use Sync instead of Copy task - add user friendly output to point user to right directory

Update code comments

43957ef

Change log level to lifecycle

e1639c5

rjernst approved these changes May 11, 2020

View reviewed changes

breskeby merged commit 739ba5f into master May 12, 2020

breskeby deleted the fix/53682 branch May 12, 2020 07:27

breskeby added backport pending v7.7.1 v7.8.1 v8.0.0 labels May 12, 2020

breskeby mentioned this pull request May 12, 2020

Add gradle task to build platform specific distro (7.x backport) #56572

Merged

breskeby mentioned this pull request May 12, 2020

Add gradle task to build platform specific distro (7.8 backport) #56573

Merged

breskeby mentioned this pull request May 12, 2020

Add gradle task to build platform specific distro (7.7 backport) #56583

Merged

breskeby added a commit that referenced this pull request May 12, 2020

Add gradle task to build platform specific distro (#56003) (#56572)

376f46f

backports #56003 to

breskeby added a commit that referenced this pull request May 12, 2020

Add gradle task to build platform specific distro (#56003) (#56573)

800bd8e

backports #56003 to 7.8

mark-vieira added the v7.9.0 label May 12, 2020

breskeby added a commit that referenced this pull request May 13, 2020

Add gradle task to build platform specific distro (#56003) (#56583)

ef1415b

backports #56003 to 7.7

breskeby removed the backport pending label May 13, 2020

mark-vieira added Team:Delivery Meta label for Delivery team and removed Team:Core/Infra Meta label for core/infra team labels Nov 11, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Add gradle task to build platform specific distro #56003

Add gradle task to build platform specific distro #56003

Uh oh!

Conversation

breskeby commented Apr 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Apr 30, 2020

Uh oh!

mark-vieira left a comment

Choose a reason for hiding this comment

Uh oh!

breskeby commented Apr 30, 2020 via email

Uh oh!

mark-vieira commented Apr 30, 2020

Uh oh!

breskeby commented May 11, 2020

Uh oh!

mark-vieira left a comment

Choose a reason for hiding this comment

Uh oh!

rjernst May 11, 2020

Choose a reason for hiding this comment

Uh oh!

breskeby May 11, 2020

Choose a reason for hiding this comment

Uh oh!

rjernst May 11, 2020

Choose a reason for hiding this comment

Uh oh!

mark-vieira May 11, 2020

Choose a reason for hiding this comment

Uh oh!

breskeby May 11, 2020

Choose a reason for hiding this comment

Uh oh!

pugnascotia May 11, 2020

Choose a reason for hiding this comment

Uh oh!

breskeby May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

breskeby commented May 11, 2020

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

mark-vieira commented May 12, 2020

Uh oh!

Uh oh!

breskeby commented Apr 30, 2020 •

edited

Loading

breskeby May 11, 2020 •

edited

Loading