Test upgrading to the default distribution

[nik9000]

Adds scenarios to the rolling restart and full cluster restart upgrade
test cases to test case involving the default distribution that we
believe will be fairly common. Before this we'd just test upgrading from
the OSS distribution to the OSS distribution. Now we'll test
OSS -> OSS and Default -> Default, and, if the version we're
upgrading from is before x-pack was part of the default distribution,
then we'll also test OSS -> Default because we believe that'll be
somewhat more common. In fact, we believe that this is the only time
when it'll be common to significantly change the configuration of
Elasticsearch and upgrade at the same time. Such upgrades come with
extra risk because you are changing two things at once so we want to be
super, mega, ultra sure that we test it.

[nik9000]

I've opened this as a WIP because I'd like to get CI running on it. I have to step away for a bit but I'll come back and self review before removing the label.

[elasticmachine]

Pinging @elastic/es-core-infra

[nik9000]

I believe this is ready for review now.

[rjernst]

I'm apprehensive about doubling the number of tests run for full cluster and rolling restart. When we had discussed testing of OSS vs default distributions before, we had agreed to adding a flag to the build which would mean "run qa with oss build" and have the default be the default distribution. Right now, the qa tests override the default explicitly to use the oss distribution. I think if we update this to use the default distribution (by default) then we automatically get this testing. Then with the flag controlling using oss, we would have another CI job which sets this flag to test oss only.

[nik9000]

I'm apprehensive about doubling the number of tests run for full cluster and rolling restart.

It triples sometimes.

I think if we update this to use the default distribution (by default) then we automatically get this testing. Then with the flag controlling using oss, we would have another CI job which sets this flag to test oss only.

We wouldn't get the oss -> default upgrades which I think is super important. Also I really don't want treat the default distribution as more important than the OSS distribution which is what we'd do by adding that flag that way. In the scenario you describe we'd still run the OSS tests but I worry that they'd get a ton less coverage because we'd treat them like windows and set it on one build and do all the distro coverage in default mode.

I don't like the additional build time either but I don't know what to do about it that doesn't make me feel terrible.

[rjernst]

We wouldn't get the oss -> default upgrades

Why wouldn't we? 6.3.0 would be default, 6.2.0 would be oss (as it must be, as there was no distribution including xpack there).

[nik9000]

Ah! I see now. I'd confused myself. zip and oss-zip are the same for the pre-6.3 stuff which makes sense.

Then we're missing upgrading x-pack basic to the default distribution. We kind of have it in x-pack/qa/rolling-upgrade-basic but it doesn't run most of the tests so far as I can tell. I think that isn't going to be as common a thing as going from oss to default so I'm happy to skip it.

That still leaves us with the doubling of test runs. I'm happy to do whatever you think is right for the bwcTest task but I really do want to do the oss to oss and oss to default and default to default test runs as part of the build. I'd be happy with having something like -DbwcFlavor=oss and -DbwcFlavor=default allow you to opt out of half of the tests. If you don't set it then you'd get both tests. We could use that on the bwcTest builds to parallelize them.

[nik9000]

So it has been a few days and I still think this is a blocker for 6.3 so we should talk this through. What I've proposed lets you opt in to OSS or default upgrades and by default will run both. We can use it to parallelize bwc testing if we'd like. And it doesn't make the OSS distribution into a second class citizen.

[nik9000]

I talked to @rjernst on Friday and we came up with something that makes us both happy. It is basically what he wanted all along but I didn't understand. I'm going to test it locally and close this PR when I open a working one for the new solution.

[nik9000]

I'm closing this in favor of #30591.