Merge remote-tracking branch 'github/develop' into fix/colang-2-runtime-issues

Author: schuellc-nvidia

CHANGELOG-Colang.md

@@ -8,12 +8,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ### Added
 
+* [#673](https://github.com/NVIDIA/NeMo-Guardrails/pull/673) Add support for new Colang 2 keyword `deactivate`.
+
 ### Changed
 
 * [#669](https://github.com/NVIDIA/NeMo-Guardrails/pull/669) Merged (and removed) utils library file with core library.
 
 ### Fixed
 
+* [#672](https://github.com/NVIDIA/NeMo-Guardrails/pull/672) Fixes a event group match bug (e.g. `match $flow_ref.Finished() or $flow_ref.Failed()`)
 * [#699](https://github.com/NVIDIA/NeMo-Guardrails/pull/699) Fix issues with ActionUpdated events and user utterance action extraction.
 
 ## [2.0-beta.2] - 2024-07-25

README.md

@@ -289,7 +289,7 @@ NeMo Guardrails integrates seamlessly with LangChain. You can easily wrap a guar
 
 Evaluating the safety of a LLM-based conversational application is a complex task and still an open research question. To support proper evaluation, NeMo Guardrails provides the following:
 
-1. An [evaluation tool](./nemoguardrails/eval/README.md), i.e. `nemoguardrails evaluate`, with support for topical rails, fact-checking, moderation (jailbreak and output moderation) and hallucination.
+1. An [evaluation tool](nemoguardrails/evaluate/README.md), i.e. `nemoguardrails evaluate`, with support for topical rails, fact-checking, moderation (jailbreak and output moderation) and hallucination.
 2. An experimental [red-teaming interface](https://docs.nvidia.com/nemo/guardrails/security/red-teaming.html).
 3. Sample LLM Vulnerability Scanning Reports, e.g, [ABC Bot - LLM Vulnerability Scan Results](https://docs.nvidia.com/nemo/guardrails/evaluation/llm-vulnerability-scanning.html)
 

docs/colang_2/language_reference/more-on-flows.rst

@@ -12,11 +12,13 @@ More on Flows
 
 In section :ref:`Defining Flows<defining-flows>` we learned the core mechanisms of flows. In this section will look at more advanced topics that are related to flows.
 
+.. _more-on-flows-activate-a-flow:
+
 ----------------------------------------
 Activate a Flow
 ----------------------------------------
 
-We already have seen the ``start`` and ``await`` keywords to trigger a flow. We are now introducing the third keyword ``activate`` that can start a flow. The difference to ``start`` lies in the behavior of the flow when it has finished or failed. If a flow was activated it will always automatically restart a new instance of the flow as soon as it has ended.
+We already have seen the ``start`` and ``await`` keywords to trigger a flow. We are now introducing the third keyword ``activate`` that can start a flow. The difference to ``start`` lies in the behavior of the flow when it has finished or failed. If a flow was activated it will always automatically restart a new instance of the flow as soon as it has ended. Furthermore, a specific flow configuration (with identical flow parameters) can only be activated once and will not start new instances, even if activated multiple times.
 
 .. important::
     Flow activation statement syntax definition:
@@ -25,8 +27,8 @@ We already have seen the ``start`` and ``await`` keywords to trigger a flow. We
 
         activate <Flow> [and <Flow>]…
 
-    - Currently, reference assignments for activated flows is not supported since the instance will change after a restart
-    - Only flow and-groups are supported
+    - Reference assignments for activated flows is not supported since the instance will change after a restart
+    - Only and-groups are supported and not or-groups
 
     Examples:
 
@@ -35,6 +37,10 @@ We already have seen the ``start`` and ``await`` keywords to trigger a flow. We
         # Activate a single flow
         activate handling user presents
 
+        # Activate two different instances of the same flow with parameters
+        activate handling user said "Hi"
+        activate handling user said "Bye"
+
         # Activate a group of flows
         activate handling user presents and handling question repetition 5.0
 
@@ -82,10 +88,10 @@ Running this example you will see the bot responding with "Hello again" as long
 
 In contrast, you can only say "Bye" once before you restart the story.
 
-Activating a flow enables you to keep matching the interaction event sequence against the pattern defined in the flow, even if the pattern previously successfully matched the interaction event sequence or failed. This is often used for passively observing a pattern to update a state or to trigger actions that don't compete with the main interaction pattern.
+Activating a flow enables you to keep matching the interaction event sequence against the pattern defined in the flow, even if the pattern previously successfully matched the interaction event sequence (finished) or failed. Since the same flow configuration can only be activated once, you can use the flow activation directly wherever you require the flow's functionality. This on demand pattern is better than activating it once in the beginning before you actually know if it is needed.
 
 .. important::
-    Activating a flow will start a flow and automatically restart it when it has ended to match to reoccurring interaction patterns.
+    Activating a flow will start a flow and automatically restart it when it has ended (finished or failed) to match to reoccurring interaction patterns.
 
 .. important::
     The main flow behaves also like an activated flow. As soon as it reaches the end it will restart automatically.
@@ -120,6 +126,8 @@ See, how the main flow does not require any match statement at the end and will
 .. important::
     An activated flow that immediately finished (does not wait for any event) will only be run once and will stay activated.
 
+.. _more-on-flows-start-a-new-flow-instance:
+
 ----------------------------------------
 Start a new Flow Instance
 ----------------------------------------
@@ -206,6 +214,39 @@ Since the first instance already started a new instance (second one) it will not
 .. note::
     You can think of the ``start_new_flow_instance`` label being at the end of each activated flow. Defining it in a different position will move it up from the default position at the end.
 
+.. _more-on-flows-deactivate-a-flow:
+
+----------------------------------------
+Deactivate a Flow
+----------------------------------------
+
+An activated flow will usually stay alive since it always restarts when it finishes or fails. To deactivate an activated flow you can use the `deactivate` keyword:
+
+.. important::
+    Flow deactivation statement syntax definition:
+
+    .. code-block:: text
+
+        deactivate <Flow>
+
+    Examples:
+
+    .. code-block:: colang
+
+        # Deactivate a single flow
+        deactivate handling user presents
+
+        # Deactivate two different instances of the same flow with different parameters
+        deactivate handling user said "Hi"
+        deactivate handling user said "Bye"
+
+Under the hood the `deactivate` keyword will abort the flow and disable the restart. It is a shortcut for this statement:
+
+.. code-block:: colang
+
+    send StopFlow(flow_id="flow name", deactivate=True)
+
+
 .. _more-on-flows-override-flows:
 
 ---------------

docs/evaluation/README.md

@@ -45,7 +45,7 @@ For the initial evaluation experiments for dialog rails, we have used two datase
 
 The datasets were transformed into a NeMo Guardrails app by defining canonical forms for each intent, specific dialogue flows, and even bot messages (for the _chit-chat_ dataset alone).
 The two datasets have a large number of user intents, thus dialog rails. One of them is very generic and has higher-grained intents (_chit-chat_), while the _banking_ dataset is domain-specific and more fine-grained.
-More details about running the dialog rails evaluation experiments and the evaluation datasets are available [here](./../../nemoguardrails/eval/data/topical/README.md).
+More details about running the dialog rails evaluation experiments and the evaluation datasets are available [here](../../nemoguardrails/evaluate/data/topical/README.md).
 
 Preliminary evaluation results follow next. In all experiments, we have chosen to have a balanced test set with at most 3 samples per intent.
 For both datasets, we have assessed the performance for various LLMs and also for the number of samples (`k = all, 3, 1`) per intent that are indexed in the vector database.
@@ -163,15 +163,15 @@ Here is a list of arguments that you can use to configure the fact-checking rail
 - `output-dir`: The directory to save the output to. The default is `eval_outputs/factchecking`.
 - `write-outputs`: Whether to write the outputs to a file or not. The default is `True`.
 
-More details on how to set up the data in the right format and run the evaluation on your own dataset can be found [here](./../../nemoguardrails/eval/data/factchecking/README.md).
+More details on how to set up the data in the right format and run the evaluation on your own dataset can be found [here](../../nemoguardrails/evaluate/data/factchecking/README.md).
 
 #### Evaluation Results
 
 Evaluation Date - Nov 23, 2023 (Mar 7 2024 for `gemini-1.0-pro`).
 
 We evaluate the performance of the fact-checking rail on the [MSMARCO](https://huggingface.co/datasets/ms_marco) dataset using the Self-Check and the AlignScore approaches. To build the dataset, we randomly sample 100 (question, correct answer, evidence) triples, and then, for each triple, build a non-factual or incorrect answer to yield 100 (question, incorrect answer, evidence) triples.
 
-We breakdown the performance into positive entailment accuracy and negative entailment accuracy. Positive entailment accuracy is the accuracy of the model in correctly identifying answers that are grounded in the evidence passage. Negative entailment accuracy is the accuracy of the model in correctly identifying answers that are **not** supported in the evidence. Details on how to create synthetic negative examples can be found [here](./../../nemoguardrails/eval/data/factchecking/README.md)
+We breakdown the performance into positive entailment accuracy and negative entailment accuracy. Positive entailment accuracy is the accuracy of the model in correctly identifying answers that are grounded in the evidence passage. Negative entailment accuracy is the accuracy of the model in correctly identifying answers that are **not** supported in the evidence. Details on how to create synthetic negative examples can be found [here](../../nemoguardrails/evaluate/data/factchecking/README.md)
 
 | Model                  | Positive Entailment Accuracy | Negative Entailment Accuracy | Overall Accuracy | Average Time Per Checked Fact (ms) |
 |------------------------|------------------------------|------------------------------|------------------|------------------------------------|
@@ -224,7 +224,7 @@ To evaluate the output moderation rail only, use the following command:
 
 ```nemoguardrails evaluate moderation --check-input False --config=path/to/guardrails/config```
 
-More details on how to set up the data in the right format and run the evaluation on your own dataset can be found [here](./../../nemoguardrails/eval/data/moderation/README.md).
+More details on how to set up the data in the right format and run the evaluation on your own dataset can be found [here](../../nemoguardrails/evaluate/data/moderation/README.md).
 
 #### Evaluation Results
 
@@ -315,7 +315,7 @@ To evaluate the hallucination rail on your own dataset, you can follow the creat
 
 #### Evaluation Results
 
-To evaluate the hallucination rail, we manually curate a set of [questions](./../../nemoguardrails/eval/data/hallucination/sample.txt) which mainly consists of questions with a false premise, i.e., questions that cannot have a correct answer.
+To evaluate the hallucination rail, we manually curate a set of [questions](../../nemoguardrails/evaluate/data/hallucination/sample.txt) which mainly consists of questions with a false premise, i.e., questions that cannot have a correct answer.
 
 For example, the question "What is the capital of the moon?" has a false premise since the moon does not have a capital. Since the question is stated in a way that implies that the moon has a capital, the model might be tempted to make up a fact and answer the question.
 

docs/getting_started/installation-guide.md

@@ -91,7 +91,7 @@ NeMo Guardrails is under active development and the main branch always contains
 The `nemoguardrails` package also defines the following extra dependencies:
 
 - `dev`: packages required by some extra Guardrails features for developers, such as the **autoreload** feature.
-- `eval`: packages used for the Guardrails [evaluation tools](../../nemoguardrails/eval/README.md).
+- `eval`: packages used for the Guardrails [evaluation tools](../../nemoguardrails/evaluate/README.md).
 - `openai`: installs the latest `openai` package supported by NeMo Guardrails.
 - `sdd`: packages used by the [sensitive data detector](../user_guides/guardrails-library.md#sensitive-data-detection) integrated in NeMo Guardrails.
 - `all`: installs all extra packages.

docs/user_guides/advanced/embedding-search-providers.md

@@ -16,6 +16,7 @@ core:
       use_batching: False
       max_batch_size: 10
       max_batch_hold: 0.01
+      search_threshold: None
     cache:
       enabled: False
       key_generator: md5
@@ -31,6 +32,7 @@ knowledge_base:
       use_batching: False
       max_batch_size: 10
       max_batch_hold: 0.01
+      search_threshold: None
     cache:
       enabled: False
       key_generator: md5

docs/user_guides/cli.md

@@ -3,17 +3,20 @@
 **NOTE: THIS SECTION IS WORK IN PROGRESS.**
 
 ## Guardrails CLI
+
 For testing purposes, the Guardrails toolkit provides a command line chat that can be used to interact with the LLM.
+
 ```
 > nemoguardrails chat --config examples/ [--verbose] [--verbose-llm-calls]
 ```
+
 ## Options
+
 - `--config`: The configuration that should be used. Can be a folder or a .co/.yml file.
 - `--verbose`: In verbose mode, detailed debugging information is also shown.
 - `--verbose-llm-calls`: In verbose LLM calls mode, the debugging information includes the entire prompt that is sent to the LLM and the completion.
 
-
-7. You should now be able to invoke the `nemoguardrails` CLI.
+You should now be able to invoke the `nemoguardrails` CLI.
 
  ```bash
  > nemoguardrails --help
@@ -31,11 +34,14 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
  Commands:
   actions-server  Starts a NeMo Guardrails actions server.
   chat            Starts an interactive chat session.
+  convert         Convert a Colang 1.0 directory to Colang 2.0 format.
+  evaluate        Run an evaluation task.
   server          Starts a NeMo Guardrails server.
  ```
 
  You can also use the `--help` flag to learn more about each of the `nemoguardrails` commands:
 
+#### actions-server
  ```bash
  > nemoguardrails actions-server --help
 
@@ -48,6 +54,7 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
   --help          Show this message and exit.
  ```
 
+#### chat
  ```bash
  > nemoguardrails chat --help
 
@@ -82,15 +89,54 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
                                                        [default: None]
   --help                                               Show this message and exit.
  ```
+#### server
+```bash
+> nemoguardrails server --help
+
+Usage: nemoguardrails server [OPTIONS]
+
+Starts a NeMo Guardrails server.
+
+Options:
+--port                                        INTEGER  The port that the server should listen on. [default: 8000]
+--config                                      TEXT     Path to a directory containing multiple configuration sub-folders.
+--verbose           --no-verbose:                      If the server should be verbose and output detailed logs including prompts. [default: no-verbose]
+--disable-chat-ui   --no-disable-chat-ui               Weather the ChatUI should be disabled [default: no-disable-chat-ui]
+--auto-reload       --no-auto-reload                   Enable auto reload option. [default: no-auto-reload]
+--prefix                                      TEXT     A prefix that should be added to all server paths. Should start with '/'.
+--help                                                Show this message and exit.
+```
 
- ```bash
- > nemoguardrails server --help
+#### evaluate
+```bash
+> nemoguardrails evaluate --help
 
- Usage: nemoguardrails server [OPTIONS]
+Usage: nemoguardrails evaluate [OPTIONS] COMMAND [ARGS]...
 
-  Starts a NeMo Guardrails server.
+Options:
+--help:          Show this message and exit.
 
- Options:
-  --port INTEGER  The port that the server should listen on.   [default: 8000]
-  --help          Show this message and exit.
- ```
+Commands:
+fact-checking:   Evaluate the performance of the fact-checking rails defined in a Guardrails application.
+hallucination:   Evaluate the performance of the hallucination rails defined in a Guardrails application.
+moderation:      Evaluate the performance of the moderation rails defined in a Guardrails application.
+topical:         Evaluates the performance of the topical rails defined in a Guardrails application. Computes accuracy for canonical form detection, next step generation, and next bot message generation. Only a single Guardrails application can be specified in the config option.
+```
+
+#### convert
+```bash
+> nemoguardrails convert --help
+
+Usage: nemoguardrails convert [OPTIONS] PATH
+
+Convert a Colang 1.0 directory to Colang 2.0.
+
+Arguments:
+  path TEXT The path to the file or directory to migrate. [default: None] [required]
+
+Options:
+--verbose                       --no-verbose                If the migration should be verbose and output detailed logs. [default: no-verbose]
+--validate                      --no-validate               If the migration should validate the output using Colang Parser. [default: no-validate]
+--use-active-decorator          --no-use-active-decorator   If the migration should use the active decorator. [default: use-active-decorator]
+--help                                                      Show this message and exit.
+```