RestSqlIT: Attempt to fix #80089

[craigtaverner]

Fixes #80089

Based on the comment regarding a leftover async task:

The leftover task is indices:data/read/sql/async/get.

I've added a waitForPendingTasks for this. The test has been muted for over two years, so I think it's time to unmute it and try this out. If we're wrong, we'll see some failures and can mute it again. But leaving it muted forever seems self-defeating.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[craigtaverner]

It failed once in part-4 and I'm running it again, but even a single failure does not bode well for merging this.

[bpintea]

It failed once in part-4 and I'm running it again, but even a single failure does not bode well for merging this.

Wondering if the failure was related, though. The change in the cleanup shouldn't affect the results of the tests? They might still fail, but that might another issue.

[bpintea]

LGTM.

[craigtaverner]

It failed once in part-4 and I'm running it again, but even a single failure does not bode well for merging this.

Wondering if the failure was related, though. The change in the cleanup shouldn't affect the results of the tests? They might still fail, but that might another issue.

Getting empty results (which is what we saw in the failed test) was something I seem to remember from some of the failures in previous cases, so perhaps this is exactly this same issue. I'll run CI once more, and if it passes again, we could consider merging, watching CI for a while, and muting/reverting if we see this again.

[craigtaverner]

Failed again with:

testAsyncTextPaginated FAILED
--
  | org.junit.ComparisonFailure: expected:<[text5          \|5
  | text6          \|6
  | text7          \|7
  | text8          \|8
  | text9          \|9
  | ]> but was:<[]>

So it gets an empty result instead of the expected table of results. The numbers make me think this is the second page, with the first having 0..4, and the second 5..9. So one page works and the second page fails. This has failed a couple of times now in the PR for main, but the PR for 7.17 keeps passing on repeated runs of CI. That is curious.

[bpintea]

@craigtaverner

So it gets an empty result instead of the expected table of results.

Was going to try a replication, but the logs in buildkite point to a(n unrelated?) failure in:
org.elasticsearch.xpack.core.security.authc.AuthenticationSerializationTests.testWriteToWithCrossClusterAccessThrowsOnUnsupportedVersion

Did the failure link change somehow??