Processing 2D only dataset fails

[jluethi]

WIP, just taking some notes on this issue here.

I've added a 2D only dataset to the tests: /data/active/fractal/2D/hiPSC_Slice/Cycle1_subset
I've been trying to process it through our current pipeline and much of it runs, it generates OME-Zarr files etc. The pipeline typically fails when I try to do labeling. I've tried to do MIPs on this dataset (even though it already has only 1 Z plane). But something in the parsing seems off and the MIP seems to make it worse. I get varying number of chunks per y folder, some with barely any content, which I wouldn't expect for a grid like this. And they aren't numbered consistently.

Plus, for some reason, the chunk setup changes when doing MIPs, which is also unexpected

As a consequence, it can't be opened in napari, because it's looking for some of the missing chunks I think. I get this error message when trying to open it:

File ~/opt/miniconda3/envs/napari-ome-zarr/lib/python3.9/site-packages/zarr/core.py:208, in Array._load_metadata_nosync(self=<class 'zarr.core.Array'> instance)
    206     meta_bytes = self._store[mkey]
    207 except KeyError:
--> 208     raise ArrayNotFoundError(self._path)
        Exception trying to inspect frame. No more locals available.
    209 else:
    210 
    211     # decode and store metadata as instance members
    212     meta = self._store._metadata_class.decode_array_metadata(meta_bytes)

ArrayNotFoundError: array not found at path %r' '1'

I'll have to recheck the metadata to make sure it is correctly "faked" (because I manually adapted the metadata to pretend it's just a 2D dataset).

Here's the branch I'm working on: https://github.com/fractal-analytics-platform/fractal/tree/example_2D/examples/14_2D_only_example

[jluethi]

This may be a problem with the metadata or the image data in the test set. I parsed it again with the current version and get the images parsed into the OME-Zarr like this now:

This isn't what I'd be expecting for this dataset, so I'll need to validate that the image data and corresponding metadata are actually correct (or trace where this mismatch may happen)

[jluethi]

Ok, it's actually a problem with the test dataset (I must have edited the metadata wrongly. It's a synthetic test dataset, not a real 2D acquisition). I'll be creating a new test dataset and testing that one.

[jluethi]

Ok, I updated the test dataset (now /data/active/fractal/2D/hiPSC_Slice/2D_test_set, also in the test cases issue). This now has correct metadata and is parsed correctly.
See fractal-analytics-platform/fractal-demos@7cfe363

And here is how the output then looks: