Better support for OME-Zarr with singleton T dimension

[BioinfoTongLI]

Downstream processing after #662. I've manually rename image_ROI_table to FOV_ROI_table and it seems to be working.
Then I've prepared a fake correction profile for all channels

dict_corr = {
    "PhenoVue Fluor 488": "fake_ffc_mat.tif",
    "Brightfield": "fake_ffc_mat.tif",
    "PhenoVue Hoechst 33342": "fake_ffc_mat.tif",
    "PhenoVue 641 Mito Stain": "fake_ffc_mat.tif",
    "PhenoVue Fluor 568": "fake_ffc_mat.tif",
}

and ran with

illumination_correction(
    input_paths=[f"{r}"],
    output_path=meas,
    metadata=metadata,
    component=component,
    illumination_profiles_folder=f"./parameters",
    dict_corr=dict_corr,
    background=100,
)

Here's the error msg I've got:

Cell In[30], [line 1](vscode-notebook-cell:?execution_count=30&line=1)
----> [1](vscode-notebook-cell:?execution_count=30&line=1) illumination_correction(
     [2](vscode-notebook-cell:?execution_count=30&line=2)     input_paths=[f"{r}"],
     [3](vscode-notebook-cell:?execution_count=30&line=3)     output_path=meas,
     [4](vscode-notebook-cell:?execution_count=30&line=4)     metadata=metadata,
     [5](vscode-notebook-cell:?execution_count=30&line=5)     component=component,
     [6](vscode-notebook-cell:?execution_count=30&line=6)     illumination_profiles_folder=f"[./parameters](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/parameters)",
     [7](vscode-notebook-cell:?execution_count=30&line=7)     dict_corr=dict_corr,
     [8](vscode-notebook-cell:?execution_count=30&line=8)     background=100,
     [9](vscode-notebook-cell:?execution_count=30&line=9) )

File [~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:40](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:40), in pydantic.decorator.validate_arguments.validate.wrapper_function()

File [~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:134](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:134), in pydantic.decorator.ValidatedFunction.call()

File [~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:206](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/pydantic/decorator.py:206), in pydantic.decorator.ValidatedFunction.execute()

File [~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:276](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:276), in illumination_correction(input_paths, output_path, component, metadata, illumination_profiles_folder, dict_corr, background, overwrite_input, new_component)
   [271](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:271) logger.info(
   [272](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:272)     f"Now processing ROI {i_ROI+1}[/](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/){num_ROIs} "
   [273](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:273)     f"for channel {i_c+1}[/](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/){num_channels}"
   [274](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:274) )
   [275](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:275) # Execute illumination correction
...
    [68](https://vscode-remote+ssh-002dremote-002b172-002e27-002e24-002e164.vscode-resource.vscode-cdn.net/lustre/scratch126/cellgen/team283/tl10/hcs_analysis/bin/notebooks/~/mambaforge/envs/fractal/lib/python3.10/site-packages/fractal_tasks_core/tasks/illumination_correction.py:68) dtype = img_stack.dtype

ValueError: Error in illumination_correction:
img_stack.shape=(1, 5, 5, 2160, 2160)
corr_img.shape=(2160, 2160)

In this case, is the shape of 5D image_stack unexpected or the corr_img?

[jluethi]

Hey @BioinfoTongLI
Thanks for opening the issue and describing it so well! Our correction function expects a 4D image array and a 2D correction array. Your image array still appears to be 5D (=> still with a time dimension?).

You mentioned somewhere you managed to drop the time dimension, right? I guess our illumination correction task is another example of a task not handling varying image dimensions well!

[BioinfoTongLI]

Our correction function expects a 4D image array and a 2D correction array.

Thanks for the reply! Do you mean there the 2D profile is applied to each FOV across all Zs? Are you planning on doing 3D correction? basically one Z position -> one profile. It is natively supported by BaSiCpy and I am working on a nf-core version of that. should be possible to merge into Fractal later.

You mentioned somewhere you managed to drop the time dimension, right?

Nope, it seems too much work. The whole folder needs to be recreated...Still trying to figure out how to walk around this issue

[jluethi]

Do you mean there the 2D profile is applied to each FOV across all Zs? Are you planning on doing 3D correction? basically one Z position -> one profile

That is interesting. I wasn't aware that BaSiCpy had such a correction. @adrtsc has developed some BaSiCpy Fractal tasks for handling 2D illumination correction (see https://github.com/Apricot-Therapeutics/APx_fractal_task_collection).

If you are building something broader for BaSiCpy processing of OME-Zarrs, would be great to also get this into Fractal of course! We are in the process of refactoring our Task API and are hoping to release Fractal 2.0 with more flexible server and simpler Task API, so may be worth to revisit it in the new version :)

Nope, it seems too much work. The whole folder needs to be recreated...Still trying to figure out how to walk around this issue

Ah, that's a pity! Unfortunately, we won't have a chance to work on making our tasks more flexible in axes handling before our Fractal 2.0 transition that we're currently focusing on, but you nicely highlight the importance of picking this up soon after!

[jluethi]

Nope, it seems too much work. The whole folder needs to be recreated...Still trying to figure out how to walk around this issue

@BioinfoTongLI I've also started looking into this, as we just had some users with the same issue. I also can't find a way to do this without rewriting the whole Zarr, which is a pity. I'll nonetheless try my luck at creating a Fractal task that copies a Zarr & drops a singleton T dimension to enable our users to use our other tasks.

Still, it will be helpful once we update more tasks to use Zarr loading that is robust to this of course! Given the current state of OME-Zarr reading & writing libraries, I'm not sure whether I expect everyone to adopt the same system though. We currently often fall back on the raw Zarr API, which means we'd have to explicitly handle such cases and others may be doing similar things.

[jluethi]

Hey @BioinfoTongLI

I now have a working (very early) prototype of a python function that drops the T dimension from an OME-Zarr. It creates a second Zarr with a suffix in the name with the same image, just no T dimension. Lot's of improvements still TBD of course, but it's working on our test data. Maybe a way for you to check out some of the Fractal tasks as well :)

See https://github.com/jluethi/fractal-helper-tasks/blob/main/src/fractal_helper_tasks/drop_t_dimension.py

[BioinfoTongLI]

Thanks a lot @jluethi for helping out here!
I will have a try!

[BioinfoTongLI]

Hi @jluethi, sorry for being radio silent for a while. I've tried the helper function is it's working great!
I've noticed that this is for processing one single FOV (or well), right?
If I want to loop over all FOVs/wells, is there such a function/object in fractal already?
I can definitely parse OME/.zattrs, but don't want to re-created the wheel either :)

[jluethi]

Hey @BioinfoTongLI
The helper function has just recently been updated to the Fractal 2.0 API, so just needs the "zarr_url" of the individual images now.

The way we handle this in Fractal is that users/developers write tasks that run e.g. per image and the Fractal server then submits this task to all available images. If you have a pre-existing OME-NGFF plate, the Import OME-Zarr task is what we use to get the list of all the available Zarr images (either within Fractal or run separately). If you run it outside of Fractal, you could probably just run import_ome_zarr with an empty list of zarr_urls and the directory where the Zarr is found.

e.g. if you have a Zarr plate located like this: /path/to/my/zarr/plate.zarr

Then running:

import_ome_zarr(
    zarr_urls=[],
    zarr_dir="/path/to/my/zarr"
    zarr_name="plate.zarr"
)

You can turn on or off the update_omero_metadata, add_image_ROI_table & add_grid_ROI_table depending on whether you run more Fractal tasks afterwards (if you do, it's helpful to add at least the first 2 options).

This function returns a dict with image_list_changes (i.e. what are the images in that plate, what are their attributes and types). If it's run within a Fractal 2.0 server, Fractal then keeps track of those. If you run it as a Python script, you can extra all the zarr_urls from there and loop over them in the drop_t_dimension task :)

[jluethi]

See more info about these concepts in Fractal 2.0 in the 2 pages here:
The Fractal task API: https://fractal-analytics-platform.github.io/version_2/tasks/
The Fractal image list: https://fractal-analytics-platform.github.io/version_2/image_list/

[BioinfoTongLI]

Thanks so much @jluethi ! I think I've made it working. - I couldn't wait to try it out when I saw your answers and keep forgetting sending feedback. sorry about that!

[jluethi]

That's awesome, glad to hear that! And no worries, Github is for async conversations anyway. The speed of my answers also vary ;)