Yokogawa Parsing chunk size mismatch

[jluethi]

I've been trying to run the /data/active/fractal/Liberali/1_well_15_fields_20_planes_SF_w_errors/D10_R1/220304_172545_220304_175557 search first example and ran into the following error:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 797, in _unwrap_futures
    new_inputs.extend([dep.result()])
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 288, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 483, in _unwrap_remote_exception
_wrapper
    result.reraise()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 138, in reraise
    reraise(t, v, v.__traceback__)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 176, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/fractal_server/app/runner/__init__.py", line 105, in _task_para
llel_fun
    _callable(
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/fractal_3repo/fractal-tasks-core/fractal_tasks_core/yokogawa_to_zarr.py
", line 136, in yokogawa_to_zarr
    da.array(FOV_4D).to_zarr(
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/core.py", line 2905, in to_zarr
    return to_zarr(self, *args, **kwargs)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/core.py", line 3638, in to_zarr
    arr = arr.rechunk(chunks)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/core.py", line 2721, in rechunk
    return rechunk(self, chunks, threshold, block_size_limit, balance)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/rechunk.py", line 297, in rechunk
    chunks = normalize_chunks(
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/dask/array/core.py", line 3084, in normalize_chunks
    raise ValueError(
ValueError: Chunks do not add up to shape. Got chunks=((1,), (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), (2160,), (2559,)), shape=(1, 19, 2160, 2560)

It looks like there is a shape mismatch here. I first thought this was related to #91, but looking closer at the error message, I'm not sure. Which one describes regions ROI parsed from the metadata which may get rounded wrong?
Also, if it was an issue with the single pixel error due to rounding of the metadata, shouldn't we have also hit it with the 10 well example?

A quick check of the raw files doesn't show issues there either though.

I'll need to dig a bit deeper. I'll have a look at whether any ROI tables have already been written.

This is run on fractal-server==0.1.4, fractal-client==0.2.1, fractal-tasks-core==0.1.6

[tcompa]

A quick check shows that some image in the metadata has a rounding error as in #91. This script

import numpy as np
import pandas as pd
# NOTE: probably this also requires "pip install lxml"

fname = "/data/active/fractal/Liberali/1_well_15_fields_20_planes_SF_w_errors/D10_R1/220304_172545_220304_175557/MeasurementData.mlf"
mlf = pd.read_xml(fname)
X = np.array(sorted(mlf["X"].unique()))
print("X sizes:")
print(X[1:] - X[:-1])

prints

X sizes:
[832.  831.9 832.  832. ]

I did not check too carefully inside the mlf file, but that would be my first guess for where this issue comes from.

[tcompa]

Also, if it was an issue with the single pixel error due to rounding of the metadata, shouldn't we have also hit it with the 10 well example?

This is a good question, since in that case the file /data/active/fractal/3D/PelkmansLab/CardiacMultiplexing/Cycle1_5x5_10wells_constantZ/MeasurementData.mlf also returns (with the same script):

X sizes:
[416.  416.  416.  415.9]

[tcompa]

Here is my test on the same (search-first) dataset.

TL;DR

• I don't hit the same shape/chunk error.. why?
• I do hit other errors, but they seem unrelated (and possibly harmless). To be checked.
• I do get an output zarr which looks reasonable.
• Parsing of metadata (correctly?) identifies some errors, and skips 4 sites out of 15.

---

First, the metadata parsing says that

Succesfully parsed 11 sites, could not parse 4 sites due to errors (see warnings).

and indeed I can see some stderr messages like

0: error was: 'AF Error at A06 No. 3 (220304_172545_220304_175557)'.The entry is skipped.
0:   warnings.warn(
0: /data/homes/fractal/.conda/envs/fractal/lib/python3.8/site-packages/fractal_tasks_core/metadata_parsing.py:193: UserWarning: When parsing the yokogawa metadata, found an 'ERR' entry at line '536' in the MLF meta file! The error was: 'AF Error at A06 No. 9 (220304_172545_220304_175557)'.The entry is skipped.
0:   warnings.warn(
0: /data/homes/fractal/.conda/envs/fractal/lib/python3.8/site-packages/fractal_tasks_core/metadata_parsing.py:193: UserWarning: When parsing the yokogawa metadata, found an 'ERR' entry at line '537' in the MLF meta file! The error was: 'AF Error at A06 No. 10 (220304_172545_220304_175557)'.The entry is skipped.
0:   warnings.warn(
0: /data/homes/fractal/.conda/envs/fractal/lib/p
0: ython3.8/site-packages/fractal_tasks_core/metadata_parsing.py:193: UserWarning: When parsing the yokogawa metadata, found an 'ERR' entry at line '690' in the MLF meta file! The error was: 'AF Error at A06 No. 13 (220304_172545_220304_175557)'.The entry is skipped.
0:   warnings.warn(

(both the stdout and stderr are currently in the server folder, under runinfo/XXX/submit_scripts)

I did see two errors, but they seem unrelated to the current issue (one is slurmstepd: Exceeded job memory limit at some point, and the other one is Exception ignored in: <Finalize object, dead>, which comes from multiprocessing/synchronize.py). Those are to be better understood.

Still, I weirdly didn't see the 2559!=2560 error. My guess is that this error takes place in one of the 4 FOVs which were skipped due to some other errors, but then I don't why @jluethi's run would behave differently and process one of those.
I'm on versions

fractal_server.__VERSION__='0.1.4'
fractal.__VERSION__='0.2.1'
fractal_tasks_core.__VERSION__='0.1.7'

but the 0.1.7 did not introduce any change to the metadata parsing, as seen in

$ git diff 0.1.6 0.1.7 --name-only
fractal_tasks_core/__init__.py
fractal_tasks_core/create_zarr_structure.py
fractal_tasks_core/image_labeling.py
fractal_tasks_core/image_labeling_whole_well.py
fractal_tasks_core/lib_pyramid_creation.py
fractal_tasks_core/lib_regions_of_interest.py
fractal_tasks_core/lib_to_zarr_custom.py
fractal_tasks_core/maximum_intensity_projection.py
fractal_tasks_core/measurement.py
fractal_tasks_core/replicate_zarr_structure.py
fractal_tasks_core/yokogawa_to_zarr.py
pyproject.toml
tests/data/napari_workflows/regionprops.yaml
tests/data/parameters_workflow_on_fake_data/wf_params.json
tests/test_unit_ROI_indices.py
tests/test_unit_image_labeling.py

To conclude, here is the output I obtain:

[tcompa]

Just to have it clear, the FOVs which are skipped are in the top-right and bottom-left corners:

[tcompa]

but the 0.1.7 did not introduce any change to the metadata parsing, as seen in

I stand corrected, 0.1.7 removed the get_bounding_box logic, and this has high chances of being the relevant difference at play here.

[jluethi]

Yup, tested it in 0.1.7 now and it also runs through for me now! 🎉

This is also the pattern we were expecting, see: #8

Once I run a few extra tests (do Illumination correction, MIP, segmentation & measurements still work as expected on search-first data), we can also close that issue & will have a search-first implementation!