Include reason in TaskCancelledException

[dimitris-athanasiou]

The task framework allows cancelling a task. The cancel task request also accepts a reason. However, that reason is not a part of the TaskCancelledException. Incorporating the reason into the TaskCancelledException would improve the information a user receives when their request fails because the corresponding task was cancelled.

There is a number of places where TaskCancelledException is thrown. For example, here it is thrown because a child task is tried to be started but the parent has been cancelled. The user will see "The parent task was cancelled, shouldn't start any child tasks" but not the reason the task has been cancelled, which would be greatly more informative as to what is going on.

I will also mention an ML use case to highlight the benefit of including the reason.

For ML anomaly detection jobs, we have a reset and a delete API. They both result in the deletion of the results and state of the job, which could be a long running task. If a user calls the delete API while a reset task is ongoing, it makes no sense to fail the call. Delete should take precedence. Thus, the delete API checks if there's a reset task running and cancels it with a reason deleting job. Then the caller of the reset API receives a response that contains the TaskCancelledException. However, that does not include the reason so the user does not get the critical piece of information that their reset was cancelled because some other user deleted the job entirely.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

As things stand today, the task framework only creates a TaskCancelledException in two places and the only one that doesn't include the reason is the one you linked:

elasticsearch/server/src/main/java/org/elasticsearch/tasks/TaskManager.java

Line 567 in 68817d7

throw new TaskCancelledException("The parent task was cancelled, shouldn't start any child tasks");

I think including the reason makes sense here, although in fact deleting job looks to be pretty much the only place that we fill in a meaningful reason for cancellation which I guess is why this hasn't come up much before.

There's a broader question about whether we should leave it up to callers to check for cancellation and construct their own TaskCancelledException as much as we do today, since almost none of these exceptions check the reason for cancellation either.