Get search first zebrafish test set to pass

[jluethi]

EDIT: This may be an issue regarding overlap. If so, that's currently out-of-scope.

When running the search-first zebrafish test set /data/active/fractal/3D/PelkmansLab/ZebrafishMultiplexing/cycle0 (number 2 gridless of fractal-analytics-platform/fractal-client#213), parsing fails with the following error message:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 783, in _unwrap_futures
    kwargs[key] = dep.result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 288, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/dataflow/dflow.py", line 483, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 138, in reraise
    reraise(t, v, v.__traceback__)
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/six.py", line 719, in reraise
    raise value
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/parsl/app/errors.py", line 176, in wrapper
    return func(*args, **kwargs)  # type: ignore
  File "/data/homes/jluethi/.conda/envs/fractal/lib/python3.8/site-packages/fractal_server/app/runner/__init__.py", line 55, in _task_fun
    metadata_update = _callable(
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/fractal_3repo/fractal-tasks-core/fractal_tasks_core/create_zarr_structure.py", line 
157, in create_zarr_structure
    metadata = parse_metadata(fn.name)
  File "/net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/fractal_3repo/fractal-tasks-core/fractal_tasks_core/lib_parse_filename_metadata.py",
 line 58, in parse_metadata
    raise Exception(
Exception: Failure in metadata parsing of AssayPlate_Greiner_#655090, with img_date={img_date} and img_time={img_time}

Debug the parsing to figure out what's failing. Could be an issue when the metadata was created for this test set, could be a parsing problem.

[jluethi]

I updated the parsing logic to speed it up (see here: )

As part of the refactor, the base problem appears to have been solved. But now, the overlap correction runs into an issue:

2022-10-14 15:44:30,935; INFO; removing FOV overlaps for well='B02'
2022-10-14 15:44:30,936; INFO; removing FOV overlaps for well='B03'
2022-10-14 15:44:30,937; INFO; removing FOV overlaps for well='B04'
2022-10-14 15:44:30,939; INFO; iteration=1, removing overlap between fov_id_1=2.0 and fov_id_2=1.0
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In [9], line 1
----> 1 site_metadata, total_files = parse_yokogawa_metadata(mrf_path, mlf_path)
      2 total_files

File ~/Dropbox/Joel/FMI/Code/fractal/fractal-tasks-core/fractal_tasks_core/metadata_parsing.py:81, in parse_yokogawa_metadata(mrf_path, mlf_path)
     75 total_files = len(mlf_frame)
     76 # TODO: Check whether the total_files correspond to the number of
     77 # relevant input images in the input folder. Returning it for now
     78 # Maybe return it here for further checks and produce a warning if it does
     79 # not match
---> 81 site_metadata = remove_FOV_overlaps(site_metadata)
     83 return site_metadata, total_files

File ~/Dropbox/Joel/FMI/Code/fractal/fractal-tasks-core/fractal_tasks_core/lib_remove_FOV_overlaps.py:119, in remove_FOV_overlaps(df)
    117 is_y_equal = abs(ymin_1 - ymin_2) < tol and (ymax_1 - ymax_2) < tol
    118 if is_x_equal + is_y_equal != 1:
--> 119     raise Exception(
    120         "Two overlapping FOVs MUST share either a "
    121         "vertical boundary or an horizontal one, but "
    122         f"{is_x_equal=} and {is_y_equal=}"
    123     )
    125 # Compute and apply shift (either along x or y)
    126 if is_y_equal:

Exception: Two overlapping FOVs MUST share either a vertical boundary or an horizontal one, but is_x_equal=False and is_y_equal=False

I'll have to dig a bit deeper to understand why this would occur

Edit: It does not occur in all wells. Looks like there is some well in this dataset that triggers this issue. I'll see whether I can trace that down to check whether it's a metadata problem or a processing problem

[jluethi]

My impression is that this is an issue that arises when there is overlap between images that can't be corrected with a simple shift in one direction. Unsure whether we want to get into this at all, but I wasn't expecting to hit it here.

The well B04, which is the first one triggering this issue, does have a weird FOV setup. Here are the FOVs:

FOV 1 is at -1907.4; 514.5
FOV 2 is at -2385.3; 385.4

The pixel size for this experiment is 0.322101 in x & y, 1 in z, according to the metadata. @MaksHess is this correct? That's a fairly weird pixel size! At the CV7K, I expect to see 0.1625 and 0.33. Is that correct?

Given the pixel size and the image size of 2000x2000 pixels, FOV 2 thus extends into FOV1 (overlap in both X & Y) => not a regular overlap, but overlap like in scenario 4 here that we're not planning to cover: #11

@MaksHess Is this dataset acquired with overlap in some field of views?

[MaksHess]

I think Yokogawa does some internal scaling of the pixel size based on who-knows-what (theoretically it would be 0.325um).
Regarding the overlap: I did not acquire overlapping regions on purpose, but it's theoretically possible if two embryos detected by searchFirst are close together...

[jluethi]

Ok, weird that the Yokogawa does some kind of pixel size scaling here. The CV7K returns pixel sizes like we would expect them. Do you think 0.322101 is actually a wrong entry or the actual truth for this dataset?

Alternatively, we could look into manually changing the metadata to avoid such overlaps (but misplace a few FOVs then). But I'd have some concerns with that, see below

Regarding the overlap: I did not acquire overlapping regions on purpose, but it's theoretically possible if two embryos detected by searchFirst are close together...

Two points here:

1. Do you have a dataset without any such overlap that we could test? Alternatively, I'd just test with the B02 & B03 wells only, because they don't appear to have any overlap
2. Broader discussion: Is imaging with potential overlap desirable? The way the search-first is implemented in the Liberali lab is the following: If two objects are very close to each other, it still acquires non-overlapping images by just shifting one FOV slightly off-center. My concern with imaging with overlap has multiple levels:
   a) It becomes really hard to process whole wells when we have unpredictable regions of overlap. We could work around this by shifting FOVs to pretend they don't have overlap. We already do this for simple cases (e.g. a grid with predictable overlap). But in unpredictable setups, it's quite tricky to handle this
   b) On the scientific side: you may have partial bleaching of one of the objects you image that you wouldn't account for. Also, elution may work worse if some regions are imaged multiple times, leading unpredicted elution issues in the actual experiment that don't show up in any controls. Thus, I'd strongly argue towards imaging non-overlapped were possible.

[MaksHess]

I think the scaling might be right (mb taking into account some deviation from 20x for their custom objective), in any case, an error of <1% sounds reasonable...

1. No unfortunately the dataset is on archive, if you can work with B02 & B03 that would be great... Another option would be to delete the single acquisition that is overlapping from the well.
2. I share your concerns with overlapping FOV's in general, mb long-term we should add constraints on that in the initial search-first detection. With my data it's possible that only edges of FOV's overlap without imaging an embryo twice though. Ideally those cases would be covered, could also be solved by a small shift :)

[jluethi]

Makes sense. I'll test with B02 & B03 for the moment then (next week). And will think about a temporary mitigation strategy (e.g. a quick check we could run over the metadata file and then whether we could work around issues that come up with some small hack)

[jluethi]

Ok, noticed another issue we're having with this test set: Our filename parsing currently makes assumptions of the filename structure that don't hold up here. We'll need to correct those. In the meantime, I'm renaming a (linked version of) these files into something without _ in the first part of the filename.

See this issue for details: #189

[jluethi]

I now got the zebrafish data to be parsed successfully for the well B02 (a subset of the data, because the B02 well doesn't have any overlaps between ROIs), see: fractal-analytics-platform/fractal-demos@1a78f16

We can look at it in napari both in 3D with single cell segmentation (not optimized, but decent shot with default cellpose at 4x downsampled level):

As well as at an MIP representation:

We can interactively browse the well in napari, zoom in to different zebrafish embryos to interactively load the full res data. It handles the positions of this search-first acquisition well, parsing positions from metadata and placing FOVs in the well context.

@MaksHess If you want to try it out as well, the data is here: /data/active/jluethi/Fractal/20221108-zebrafish-subset.

There are 2 remaining issues, but I'll close this overall issue while we work on the more specific issues we discovered here:

1. How do we handle "unexpected" overlap, i.e. not a regular overlap in a grid, but overlap that can happen somewhat rarely: Handling unexpected overlap #170
2. More robust image filename parsing: Image filename parsing assumes date case in wrong case #189 We'll have a fix for this soon :)