Compound tasks

[tcompa]

We have two cases with a create&fill structure:

• Global task that creates a zarr file with the right structure/metadata but without data (e.g. create_zarr_structure, or replicate_zarr_structure);
• Well-parallelized task that writes data (e.g. yokogawa_to_zarr, or maximum_intensity_projection).

We could find a new way to define them together.

Note that this only concerns fractal, while the tasks would remain as they are now.

[jluethi]

That's an interesting topic. Would also be interesting from a user side if there is a compound task to call that does both. i.e. a user should not need to know that parsing form yokogawa to zarr needs to task, all they think is "I want my yokogawa data in a Zarr file".
So below the hood, they are distinct tasks, because they have different parallelisation. But the user just sees the compound task as something they want to use.

[tcompa]

For the record, yesterday we re-discussed the option of a compound task, possibly made of pre/main/post phases.
This was not only related to the create/fill tasks discussed above, but also to the fact that the future import-ome-zarr task (ref fractal-analytics-platform/fractal-tasks-core#521) could be seen as the pre phase of most tasks.

For the moment we did not go too far into this direction, apart from noting that this "compound task" would still need to correspond to two actual tasks being run (independently from how transparent this is for the user), or it would be difficult to parallelize the second task (the "fill" one, in a create&fill scenario).

[jluethi]

Good summary. On the priority of this: Compound tasks in the way described above may offer a useful simplification from the users perspective. It's not a current priority to push those though, as we first want to build the features that would make up the useful pre-phases.
Even after that, TBD when this would come a priority

[jluethi]

Additional notes on what we could do with it were made here: #514

[tcompa]

This issue in the current form is made obsolete by ongoing V2 work.