Merge branch 'master' into 000-add-spellcheck

Author: Panaetius

.github/workflows/acceptance-tests.yml

@@ -27,6 +27,7 @@ jobs:
       renku-notebooks: ${{ steps.deploy-comment.outputs.renku-notebooks}}
       renku-ui: ${{ steps.deploy-comment.outputs.renku-ui}}
       test-enabled: ${{ steps.deploy-comment.outputs.test-enabled}}
+      extra-values: ${{ steps.deploy-comment.outputs.extra-values}}
     steps:
       - id: deploy-comment
         uses: SwissDataScienceCenter/renku-actions/[email protected]
@@ -64,6 +65,7 @@ jobs:
         renku_graph: "${{ needs.check-deploy.outputs.renku-graph }}"
         renku_notebooks: "${{ needs.check-deploy.outputs.renku-notebooks }}"
         renku_ui: "${{ needs.check-deploy.outputs.renku-ui }}"
+        extra_values: "${{ needs.check-deploy.outputs.extra-values }}"
     - name: Check existing renkubot comment
       uses: peter-evans/find-comment@v1
       id: findcomment

design/001-new-workflow-commands/001-new-workflow-commands.md

@@ -146,14 +146,14 @@ renku workflow execute [<options>] <name or uuid>
 
 ```
 
-#### renku workflow loop
+#### renku workflow iterate
 
 Has similar semantics to `renku workflow execute`, but allows doing multiple executions in a single go with variable inputs/outputs/parameters.
 
 Syntax example:
 
 ```
-renku workflow loop <workflow name> --mapping <mapping>
+renku workflow iterate <workflow name> --mapping <mapping>
 ```
 
 Where mapping is a list of mappings for input parameters/files that gets iterated over. This can contain values that are the same for all executions and then a list of parameters that vary.
@@ -163,7 +163,7 @@ Otherwise supports the same flags as `renku workflow execute`
 ##### Detailed Parameter Description
 
 ```
-renku workflow loop [<options>] <name or uuid>
+renku workflow iterate [<options>] <name or uuid>
 
 <name or uuid>              The name of a workflow or its id
 
@@ -172,10 +172,10 @@ renku workflow loop [<options>] <name or uuid>
 
 -c|--config                    YAML file containing config for the provider
 
--s|--set <parameter>=[<value> | <value>,<values...>]   Set <value> or value range for a <parameter>
+-m|--map <parameter>=[<value> | <value>,<values...>]   Set <value> or value range for a <parameter>
                                                     to be used in execution
 
---values <file>             YAML file containing parameter mappings to be used (See
+--mapping <file>            YAML file containing parameter mappings to be used (See
                             `Input/Output/Argument Mappings`)
 
 ```
@@ -318,35 +318,49 @@ myworkflow:
 
 At the top level, it specifies the parameter values of the workflow (run or grouped run) to be executed. It can also contain values for steps contained in a grouped run, by specifying the name of the step and setting parameters for that step. In case of the grouped run mapping its values to child steps, setting values for child steps directly overwrites those values for the step and all its (potential) children.
 
-For `renku workflow loop` the values file looks like:
+For `renku workflow iterate` the values file looks like:
 
 ```
 learning_rate: 0.9
 dataset_input: dataset.csv
-chart_output: mychart.{loop_index}.png
-looped_parameters:
-    - alpha: 0.1
-      beta: 0.7
-    - alpha: 0.2
-      beta: 0.6
-    - alpha: 0.5
-      beta: 0.2
+chart_output: mychart.{iter_index}.png
+alpha: [0.1, 0.2, 0.5]
+beta: [0.7, 0.6]
+gamma@tag1: [1.0, 2.0, 3.0]
+delta@tag1: [10.0, 20.0, 30.0]
 myworkflow:
     lr: 0.8
     lookuptable: lookup.xml
     myotherworkflow:
         language: en
 ```
 
-Where the workflow is run for each entry in `looped_parameters` and the templated variable `{loop}` is substituted for the loop index (1, 2, 3, ...).
+where the the paramaters with list values are considered as the iteration parameters and the templated variable `{iter_index}` is substituted with the iteration index (0, 1, 2, ...).
+There are two distinct type of paramaters:
+ 1. The parameter's value is a simple list, `alpha` and `beta` in the above example. These list values are simply the possible values of the given parameter. The iterate command will generate the cartesian product of these possible values and will be used in each iteration. Hence, the length of the list of possible values is not constrained, i.e. it can contain arbitrary amount of values.
+ 2. Tagged iteration parameters, `gamma` and `delta` in the above example. A parameter can be tagged in the values file by adding the `@<TAG>` suffix to the parameter's name. Parameters of the same tag shall have the same number of possible values. This is because these values are going to be iterated over as a tuple, in the provided order. For example in case of `gamma` and `delta` the `renku workflow iterate` command would simply result in three iterations: `[(1.0, 10.0), (2.0, 20.0), (3.0, 30.0)]`
+
+One can mix these two types of parameters.
+
+Based on the this, the first six iterations using the values file above will result in the following workflow parametrization and execution:
+```
+chart_output      alpha    beta    gamma    delta
+--------------  -------  ------  -------  -------
+mychart.0.png       0.1     0.7        1       10
+mychart.1.png       0.1     0.7        2       20
+mychart.2.png       0.1     0.7        3       30
+mychart.3.png       0.1     0.6        1       10
+mychart.4.png       0.1     0.6        2       20
+mychart.5.png       0.1     0.6        3       30
+```
 
 On the commandline, values can be specified with `--set learning_rate=0.9 --set myworkflow.lr=0.8`.
 
 #### Templated Values
 
 Parameter values can contain one of several templated values or glob patterns:
 
-- `{loop_index}`: Loop index in the case of a `renku workflow loop` call.
+- `{iter_index}`: Iteration index in the case of a `renku workflow iterate` call.
 - `{%m-%d-%Y_%H-%M-%S}`: Current date/time (Following python date formatting).
 - `a*b?`: Glob/`fnmatch` patterns for paths.
 
@@ -466,7 +480,7 @@ Outputs:
 $ echo "
 static:
   epochs: 200
-  output_file: result.{loop_index}.json
+  output_file: result.{iter_index}.json
 variable:
 - lr:0.1
   batch-size: 32
@@ -477,7 +491,7 @@ variable:
 - lr:1.0
   batch-size: 1024
 " > mapping.yaml
-$ renku workflow loop --provider cwl --mapping mapping.yaml train_model
+$ renku workflow iterate --provider cwl --mapping mapping.yaml train_model
 Preparing environment....
 Executing workflows
 Run 1...
@@ -559,7 +573,7 @@ The following outputs were produced:
             data.2021-02-24.csv
 ```
 #### Workflow execution with extensive parameter search
-See detailed user story to `renku workflow loop` [here](https://github.com/SwissDataScienceCenter/renku-python/edit/rfc-001-workflow-ux/design/001-new-workflow-commands/user-story-workflow-execution-with-extensive-parameter-space.md)
+See detailed user story to `renku workflow iterate` [here](https://github.com/SwissDataScienceCenter/renku-python/edit/rfc-001-workflow-ux/design/001-new-workflow-commands/user-story-workflow-execution-with-extensive-parameter-space.md)
 
 ## Drawbacks
 

docs/_static/asciicasts/iterate_plan.gif

@@ -185,9 +185,15 @@ def cli(ctx, path, external_storage_requested):
     """Check common Renku commands used in various situations."""
     from renku.core.management import RENKU_HOME
     from renku.core.management.client import LocalClient
+    from renku.core.management.migrations.utils import OLD_METADATA_PATH
+    from renku.core.management.repository import RepositoryApiMixin
+    from renku.core.metadata.database import Database
 
     renku_path = Path(path) / RENKU_HOME
-    if not renku_path.exists() and not is_allowed_command(ctx):
+    old_metadata = renku_path / OLD_METADATA_PATH
+    new_metadata = renku_path / RepositoryApiMixin.DATABASE_PATH / Database.ROOT_OID
+
+    if not old_metadata.exists() and not new_metadata.exists() and not is_allowed_command(ctx):
         raise UsageError(
             (
                 "`{0}` is not a renku repository.\n"

renku/cli/__init__.py

@@ -121,42 +121,10 @@ def migrate(check, skip_template_update, skip_docker_update, strict):
 @click.command(hidden=True)
 def migrationscheck():
     """Check status of the project and current renku-python version."""
-    from renku.core.commands.migrate import migrations_check, migrations_versions
-
-    latest_version, project_version = migrations_versions().build().execute().output
-    (
-        migration_required,
-        project_supported,
-        template_update_possible,
-        current_template_version,
-        latest_template_version,
-        template_source,
-        template_ref,
-        template_id,
-        automated_update,
-        docker_update_possible,
-    ) = (
-        migrations_check().lock_project().build().execute().output
-    )
+    from renku.core.commands.migrate import migrations_check
 
-    click.echo(
-        json.dumps(
-            {
-                "latest_version": latest_version,
-                "project_version": project_version,
-                "migration_required": migration_required,
-                "project_supported": project_supported,
-                "template_update_possible": template_update_possible,
-                "current_template_version": str(current_template_version),
-                "latest_template_version": str(latest_template_version),
-                "template_source": template_source,
-                "template_ref": template_ref,
-                "template_id": template_id,
-                "automated_update": automated_update,
-                "docker_update_possible": docker_update_possible,
-            }
-        )
-    )
+    result = migrations_check().lock_project().build().execute().output
+    click.echo(json.dumps(result))
 
 
 @click.command(hidden=True)

renku/cli/migrate.py

@@ -111,6 +111,55 @@
 values YAML file and passing that using ``--values``. Provider specific
 settings can be passed as file using the ``--config`` parameter.
 
+Iterate Plans
+*************
+
+.. image:: ../_static/asciicasts/iterate_plan.gif
+   :width: 850
+   :alt: Iterate Plans
+
+For executing a Plan with different parameterization ``renku workflow iterate``
+could be used. This sub-command is basically conducting a 'grid search'-like
+execution of a Plan, with parameter-sets provided by the user.
+
+.. code-block:: console
+
+    $ renku workflow iterate --map parameter-1=[1,2,3] \
+            --map parameter-2=[10,20] my-run
+
+The set of possible values for a parameter can be given by ``--map`` command
+line argument or by specifying them in a values YAML file and passing that
+using ``--mapping``.
+
+By default ``renku workflow iterate`` will execute all the combination of the
+given parameters' list of possible values. Sometimes it is desired that instead
+of all the combination of possible values, a specific tuple of values are
+executed. This could be done by marking the parameters that should be bound
+together with the ``@tag`` suffix in their names.
+
+.. code-block:: console
+
+    $ renku workflow iterate --map parameter-1@tag1=[1,2,3] \
+            --map parameter-1@tag1=[10,5,30] my-run
+
+This will result in only three distinct execution of the ``my-run`` Plan,
+with the following parameter combinations: ``[(1,10), (2,5), (3,30)]``. It is
+important to note that parameters that have the same tag, should have the same
+number of possible values, i.e. the values list should have the same length.
+
+There's a special template variable for parameter values ``{iter_index}``, which
+can be used to mark each iteration's index in a value of a parameter. The template
+variable is going to be substituted with the iteration index (0, 1, 2, ...).
+
+.. code-block:: console
+
+    $ renku workflow iterate --map parameter-1=[10,20,30] \
+            --map output=output_{iter_index}.txt my-run
+
+This would execute ``my-run`` three times, where ``parameter-1`` values would be
+``10``, `20`` and ``30`` and the producing output files ``output_0.txt``,
+``output_1.txt`` and ``output_2.txt`` files in this order.
+
 Exporting Plans
 ***************
 
@@ -1060,3 +1109,58 @@ def visualize(sources, columns, exclude_files, ascii, interactive, no_color, pag
         text_output, navigation_data, result.output.vertical_space, use_color=not no_color
     )
     viewer.run()
+
+
+@workflow.command()
+@click.option(
+    "mapping_path",
+    "--mapping",
+    metavar="<file>",
+    type=click.Path(exists=True, dir_okay=False),
+    help="YAML file containing parameter mappings to be used.",
+)
+@click.option(
+    "--dry-run",
+    "-n",
+    is_flag=True,
+    default=False,
+    help="Print the generated plans with their parameters instead of executing.",
+    show_default=True,
+)
+@click.option(
+    "provider",
+    "-p",
+    "--provider",
+    default="cwltool",
+    show_default=True,
+    type=click.Choice(Proxy(_available_workflow_providers), case_sensitive=False),
+    help="The workflow engine to use.",
+)
+@click.option("mappings", "-m", "--map", multiple=True, help="Mapping for a workflow parameter.")
+@click.option("config", "-c", "--config", metavar="<config file>", help="YAML file containing config for the provider.")
+@click.argument("name_or_id", required=True)
+def iterate(name_or_id, mappings, mapping_path, dry_run, provider, config):
+    """Execute a workflow by iterating through a range of provided parameters."""
+    from renku.core.commands.view_model.plan import PlanViewModel
+    from renku.core.commands.workflow import iterate_workflow_command, show_workflow_command
+
+    if len(mappings) == 0 and mapping_path is None:
+        raise errors.UsageError("No mapping has been given for the iteration!")
+
+    plan = show_workflow_command().build().execute(name_or_id=name_or_id).output
+
+    if plan:
+        if isinstance(plan, PlanViewModel):
+            _print_plan(plan)
+        else:
+            _print_composite_plan(plan)
+
+    communicator = ClickCallback()
+    iterate_workflow_command().with_communicator(communicator).build().execute(
+        name_or_id=name_or_id,
+        mapping_path=mapping_path,
+        mappings=mappings,
+        dry_run=dry_run,
+        provider=provider,
+        config=config,
+    )

renku/cli/workflow.py

@@ -36,6 +36,7 @@
 from renku.core.management.interface.client_dispatcher import IClientDispatcher
 from renku.core.management.interface.database_dispatcher import IDatabaseDispatcher
 from renku.core.management.interface.database_gateway import IDatabaseGateway
+from renku.core.management.migrations.utils import OLD_METADATA_PATH
 from renku.core.management.repository import INIT_APPEND_FILES, INIT_KEEP_FILES
 from renku.core.metadata.repository import Repository
 from renku.core.models.tabulate import tabulate
@@ -326,6 +327,14 @@ def _init(
     database_gateway = inject.instance(IDatabaseGateway)
     database_gateway.initialize()
 
+    # add metadata.yml for backwards compatibility
+    metadata_path = client.renku_path.joinpath(OLD_METADATA_PATH)
+    with open(metadata_path, "w") as f:
+        f.write(
+            "# Dummy file kept for backwards compatibility, does not contain actual version\n"
+            "'http://schema.org/schemaVersion': '9'"
+        )
+
     # NOTE: clone the repo
     communication.echo("Initializing new Renku repository... ")
     with client.lock: