Define working example of whole project (from project creation to workflow execution).

[tcompa]

It would be useful to have a working example of an entire project (from its creation to workflow execution), to simplify debugging when something goes wrong (like the ModuleNotFoundError: No module named 'wrappers.compressionTaskWrap error found by @gusqgm today, but it could be something else).

By looking at the README, a list of instructions would look like this (from the mwe_fractal/fractal folder):

poetry install
mkdir -p ../test-proj
echo {"datasets": {"dstest": {"resources": ["absolute/path/of/resource"], "type": "tif"}}, "workflows": {}} > ../test-proj/mwe-test.json
poetry run python fractal_cmd.py project new mwe-test ../test-proj dstest
poetry run python fractal_cmd.py dataset add-resources mwe-test dstest absolute/path/of/resource
poetry run python fractal_cmd.py dataset update-type mwe-test dstest tif
poetry run python fractal_cmd.py task add compression_tif tif tif
poetry run python fractal_cmd.py workflow new mwe-test wftest compression_tif
poetry run python fractal_cmd.py apply test_apply.json

I need some help to complete this list and turn it into something fully working. This requires at least specifying a path where I can find some appropriate resources (in the fractal_cmd.py dataset add-resources line), and possibly also specifying some more information in test_apply.json. Any other tip is welcome.

After I figure out how it works, I can turn it into a test script and add it to the CI (more precisely: to that part of the CI which is currently only run locally on pelkmanslab, as it requires access to the data folder and to a slurm cluster).

[mfranzon]

Hi @tcompa, thank you for pointing out that the README needs more informations.

Minimal test case, 2x2 grid, single well, 3 channels could be found here:
/data/active/fractal/3D/PelkmansLab/CardiacMultiplexing/Cycle1_subset_extras/

Regarding test_apply.json, at the moment I am writing this message, a good example of it could be:

{"arguments": {"project_name": "test",
     "workflow_name": "wftest",
      "input_dataset": ["dstest"],
      "output_dataset": ["dstest"],
      "resource_in": ["/data/active/fractal/3D/PelkmansLab/CardiacMultiplexing/Cycle1_subset_extras/"],
      "resource_out": ["/data/homes/rmurri/mwe_fractal/fractal/result/"],
      "delete": ["False"],
      "scheduler": "slurm",
      "slurm_params": {"mem" : 10000, "cores": 1, "nodes":1},
      "ext": "png",
      "other_params": {"dims" : "2,2"}
      }
}

[tcompa]

The following version seems to work. Note that this should run from the mwe_fractal/fractal folder, and that luigi daemon should be active.

PATH_INPUT=/data/active/fractal/3D/PelkmansLab/CardiacMultiplexing/Cycle1_subset_extras/
PATH_OUTPUT=/data/homes/rmurri/mwe_fractal/fractal/result/

echo 'Re-install poetry'
poetry install
echo 'Clean up old JSON files'
mkdir -p ../test-proj
rm -f fractal.json test.json
echo 'Create JSON files'
echo "{\"datasets\": {\"dstest\": {\"resources\": [], \"type\": \"\"}}, \"workflows\": {}}" > ../test-proj/mwe-test.json
echo "\
    {\"arguments\": {\"project_name\": \"mwe-test\",
         \"workflow_name\": \"wftest\",
          \"input_dataset\": [\"dstest\"],
          \"output_dataset\": [\"dstest\"],
          \"resource_in\": [\"$PATH_INPUT\"],
          \"resource_out\": [\"$PATH_OUTPUT\"],
          \"delete\": [\"False\"],
          \"scheduler\": \"slurm\",
          \"slurm_params\": {\"mem\" : 10000, \"cores\": 1, \"nodes\":1},
          \"ext\": \"png\",
          \"other_params\": {\"dims\" : \"2,2\"}
          }
    }" > test_apply.json
echo 'Create project'
poetry run python fractal_cmd.py project new mwe-test ../test-proj dstest
echo
echo 'Add dataset to project'
poetry run python fractal_cmd.py dataset add-resources mwe-test dstest $PATH_INPUT
poetry run python fractal_cmd.py dataset list mwe-test
echo
echo 'Update dataset type'
poetry run python fractal_cmd.py dataset update-type mwe-test dstest tif
poetry run python fractal_cmd.py dataset list mwe-test
echo
echo 'Add yokogawa_to_zarr task'
poetry run python fractal_cmd.py task add yokogawa_to_zarr tif zarr
poetry run python fractal_cmd.py task list
echo
echo 'Create workflow'
poetry run python fractal_cmd.py workflow new mwe-test wftest yokogawa_to_zarr
poetry run python fractal_cmd.py workflow list mwe-test
echo
echo 'Execute workflow'
poetry run python fractal_cmd.py apply test_apply.json
echo

[tcompa]

I rewrote the regression test above as a stand-alone python script, see code below. Once again, this should be run from the mwe_fractal/fractal folder, and with the luigi daemon running. The first three lines should be modified, to correctly set resource_in, tmp_path and resource_out. After that change, the script can be run as

poetry run python regression_test.py

Here's the full regression_test.py code:

import os
import sys
import json

from fractal_cmd import project_new
from fractal_cmd import projects_list
from fractal_cmd import datasets_add_resources
from fractal_cmd import dataset_update_type
from fractal_cmd import datasets_list
from fractal_cmd import task_add
from fractal_cmd import task_list
from fractal_cmd import workflow_new
from fractal_cmd import workflow_list
from fractal_cmd import apply as _apply


# General variables and paths, to be set by user
resource_in = "/data/active/fractal-temp/3D/PelkmansLab/CardiacMultiplexing/Cycle1_subset_extras/"
tmp_path = "/data/homes/rmurri/mwe_fractal/fractal/Temporary_folder_for_tests/"
resource_out = tmp_path + "result/"


####################################
# Do not modify code under this line

# Quick&dirty way to ignore function decorators (which are otherwise used for CLI)
project_new = project_new.__dict__['callback']
projects_list = projects_list.__dict__['callback']
datasets_add_resources = datasets_add_resources.__dict__['callback']
dataset_update_type = dataset_update_type.__dict__['callback']
datasets_list = datasets_list.__dict__['callback']
task_add = task_add.__dict__['callback']
task_list = task_list.__dict__['callback']
workflow_new = workflow_new.__dict__['callback']
workflow_list = workflow_list.__dict__['callback']
_apply = _apply.__dict__['callback']


# Check that tmp_path folder is not there
if os.path.isdir(tmp_path):
    sys.exit(f"ERROR: {tmp_path} folder already exists, "
             f"you should remove it or change the tmp_path variable.")

# General variables and path, part 2 (do not modify this)
project_name = "mwe-test"
dataset_name = "dstest"
workflow_name = "wftest"
ds_type = "tif"
task_name = "yokogawa_to_zarr"
input_type = "tif"
output_type = "zarr"

# Prepare and execute a workflow
project_new(project_name, tmp_path, dataset_name)
datasets_add_resources(project_name, dataset_name, [resource_in])
dataset_update_type(project_name, dataset_name, ds_type)
task_add(task_name, input_type, output_type)
workflow_new(project_name, workflow_name, [task_name])
filename_test_apply = tmp_path + "test_apply.json"
test_apply = {
    "arguments": {
        "project_name": f"{project_name}",
        "workflow_name": f"{workflow_name}",
        "input_dataset": [f"{dataset_name}"],
        "output_dataset": [f"{dataset_name}"],
        "resource_in": [f"{resource_in}"],
        "resource_out": [f"{resource_out}"],
        "delete": ["False"],
        "scheduler": "slurm",
        "slurm_params": {"mem" : 10000, "cores": 1, "nodes":1},
        "ext": "png",
        "other_params": {"dims" : "2,2"}
        }
    }
with open(filename_test_apply, 'w') as xx:
    json.dump(test_apply, xx)

_apply(filename_test_apply)

[tcompa]

I still need to clean it up a bit (for instance the apply function now requires a JSON file, but it should be consistent with the others), and then I'll add this script to the repository.

[tcompa]

With commit 4311c15, we now have two regression tests (one in bash and one in python).

Note that the python script is a bit simpler, but it requires an additional block of code to remove all the click decorators (the ones that allow us to use the CLI) for the functions imported from fractal_cmd.py. At the moment I'm not aware of a better version to do this.

By now this is all in a dev branch (https://github.com/fractal-analytics-platform/mwe_fractal/tree/dev-newcmd), because I also modified fractal_cmd.py in a backward-incompatible way (I removed the special input form of apply, which is now similar to the other workflow-preparation functions in that it takes a series of arguments - instead of a JSON file).

I will close the issue once we merge 4311c15.

[jluethi]

@tcompa great work! We will need to parse more of the parameters to the command line interface (things like number of rows, columns, pyramid levels are currently hard-coded in the fractal_cmd.py file (=> #49).
Other than that, this issue can be closed.