Merge pull request #689 from NVIDIA/feature/colang-2.0-library

Colang 2.0 - Guardrails Library

Author: drazvan

docs/user_guides/cli.md

@@ -3,17 +3,20 @@
 **NOTE: THIS SECTION IS WORK IN PROGRESS.**
 
 ## Guardrails CLI
+
 For testing purposes, the Guardrails toolkit provides a command line chat that can be used to interact with the LLM.
+
 ```
 > nemoguardrails chat --config examples/ [--verbose] [--verbose-llm-calls]
 ```
+
 ## Options
+
 - `--config`: The configuration that should be used. Can be a folder or a .co/.yml file.
 - `--verbose`: In verbose mode, detailed debugging information is also shown.
 - `--verbose-llm-calls`: In verbose LLM calls mode, the debugging information includes the entire prompt that is sent to the LLM and the completion.
 
-
-7. You should now be able to invoke the `nemoguardrails` CLI.
+You should now be able to invoke the `nemoguardrails` CLI.
 
  ```bash
  > nemoguardrails --help
@@ -31,11 +34,14 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
  Commands:
   actions-server  Starts a NeMo Guardrails actions server.
   chat            Starts an interactive chat session.
+  convert         Convert a Colang 1.0 directory to Colang 2.0 format.
+  evaluate        Run an evaluation task.
   server          Starts a NeMo Guardrails server.
  ```
 
  You can also use the `--help` flag to learn more about each of the `nemoguardrails` commands:
 
+#### actions-server
  ```bash
  > nemoguardrails actions-server --help
 
@@ -48,6 +54,7 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
   --help          Show this message and exit.
  ```
 
+#### chat
  ```bash
  > nemoguardrails chat --help
 
@@ -82,15 +89,54 @@ For testing purposes, the Guardrails toolkit provides a command line chat that c
                                                        [default: None]
   --help                                               Show this message and exit.
  ```
+#### server
+```bash
+> nemoguardrails server --help
+
+Usage: nemoguardrails server [OPTIONS]
+
+Starts a NeMo Guardrails server.
+
+Options:
+--port                                        INTEGER  The port that the server should listen on. [default: 8000]
+--config                                      TEXT     Path to a directory containing multiple configuration sub-folders.
+--verbose           --no-verbose:                      If the server should be verbose and output detailed logs including prompts. [default: no-verbose]
+--disable-chat-ui   --no-disable-chat-ui               Weather the ChatUI should be disabled [default: no-disable-chat-ui]
+--auto-reload       --no-auto-reload                   Enable auto reload option. [default: no-auto-reload]
+--prefix                                      TEXT     A prefix that should be added to all server paths. Should start with '/'.
+--help                                                Show this message and exit.
+```
 
- ```bash
- > nemoguardrails server --help
+#### evaluate
+```bash
+> nemoguardrails evaluate --help
 
- Usage: nemoguardrails server [OPTIONS]
+Usage: nemoguardrails evaluate [OPTIONS] COMMAND [ARGS]...
 
-  Starts a NeMo Guardrails server.
+Options:
+--help:          Show this message and exit.
 
- Options:
-  --port INTEGER  The port that the server should listen on.   [default: 8000]
-  --help          Show this message and exit.
- ```
+Commands:
+fact-checking:   Evaluate the performance of the fact-checking rails defined in a Guardrails application.
+hallucination:   Evaluate the performance of the hallucination rails defined in a Guardrails application.
+moderation:      Evaluate the performance of the moderation rails defined in a Guardrails application.
+topical:         Evaluates the performance of the topical rails defined in a Guardrails application. Computes accuracy for canonical form detection, next step generation, and next bot message generation. Only a single Guardrails application can be specified in the config option.
+```
+
+#### convert
+```bash
+> nemoguardrails convert --help
+
+Usage: nemoguardrails convert [OPTIONS] PATH
+
+Convert a Colang 1.0 directory to Colang 2.0.
+
+Arguments:
+  path TEXT The path to the file or directory to migrate. [default: None] [required]
+
+Options:
+--verbose                       --no-verbose                If the migration should be verbose and output detailed logs. [default: no-verbose]
+--validate                      --no-validate               If the migration should validate the output using Colang Parser. [default: no-validate]
+--use-active-decorator          --no-use-active-decorator   If the migration should use the active decorator. [default: use-active-decorator]
+--help                                                      Show this message and exit.
+```

docs/user_guides/migration_guide.md

@@ -0,0 +1,108 @@
+# Migrating from Colang 1 to Colang 2
+
+The NeMo Guardrails CLI provides a tool (`nemoguardrails convert ...`) for converting guardrail configurations from Colang 1.0 format to Colang 2.x.
+It performs several syntax transformations on the content of the files, such as converting certain keywords and changing the case of certain identifiers.
+
+```{warning}
+There are edge cases not convered by the conversion tool. You should always manually review a guardrail configuration after conversion.
+```
+
+## How it Works
+
+The tool walks through the directory specified by the `path` argument and applies the transformations to each file ending with `.co`. The transformations include:
+
+- Converting `define flow` to `flow`.
+- Converting `define subflow` to `flow`.
+- Converting `define bot` to `flow bot`.
+- Converting `define user` to `flow user`.
+- Converting `execute` to `await`.
+- Converting snake_case identifiers after `await` to CamelCase and append it with key word `Action` while preserving any arguments used..
+- Converting `else when` to `or when`.
+- Converting `stop` to `abort`.
+- Converting quoted strings after `bot` to `bot say` or `or bot say`.
+- Converting quoted strings after `user` to `user said` or `or user said`.
+- Convert anonymous flows.
+- Use `active` decorator for root flows (flows that need activation).
+- Add `global` keyword to global variables.
+- Converting `...` to corresponding syntax in Colang 2.
+
+### Generation operator
+
+The syntax for the generation operation has changed slightly between Colang 1.0 and 2.0. The following Colang 1.0 snippet:
+
+```colang
+# some instruction in natural language
+$var_name = ...
+```
+
+It is translated to:
+
+```colang
+$name = ... "some instruction in natural language"
+```
+
+### Generic Matching
+
+In Colang 1.0 it was possible to use generic matching using `user ...` and `bot ...`.
+These have a different equivalent in Colang 2.0. The `...` can no longer be used for matching, only for generation.
+
+The following changes are applied during conversion:
+- `user ...` is translated to `user said something`
+- `bot ...` is translated to `bot said something`
+
+### Rails Configuration
+
+Colang 1.0 configuration can define certain rails in the `rails` field.
+- If rails are defined in `config.yml` a `_rails.co` file is generated with the rails defined in it.
+
+### Example Flow conversion
+
+As an example, the following flow:
+
+```colang
+define flow
+  user express greeting
+  bot express greeting
+```
+
+is converted to:
+
+```colang
+@active
+flow express_greeting
+  user express greeting
+  bot express greeting
+```
+
+```{note}
+It is a convention to use past tense for user flows and present tense for bot flows. However, the migration tool does not enforce this convention.
+```
+
+The tool keeps track of the number of lines processed and the number of changes made in each file. It also counts the total number of files changed.
+
+```{warning}
+The tool modifies the original files. It is recommended to use version control to track the changes made by the tool. It also enables you to see the differences between the original and modified files.
+```
+
+## Usage
+
+To use the conversion tool, use the following command:
+
+```bash
+nemoguardrails convert /path/to/directory
+```
+
+The `convert` command has several options:
+
+- `--verbose` or `--no-verbose`: If the migration should be verbose and output detailed logs. Default is `no-verbose`.
+- `--validate` or `--no-validate`: If the migration should validate the output using Colang Parser. Default is `no-validate`.
+- `--use-active-decorator` or `--no-use-active-decorator`: If the migration should use the `active` decorator. Default is `use-active-decorator`.
+
+## Assumptions and Limitations
+
+- The tool assumes that the input files are correctly formatted. If a file is not correctly formatted, the tool may not work as expected.
+- The tool uses regular expressions to find and replace certain patterns in the text. If the input files contain text that matches these patterns but should not be replaced, the tool may produce incorrect results.
+- The tool renames the original files and writes the transformed content to new files with the original names. Use version control to track the changes made by the tool.
+- The tool does not handle errors that may occur during file reading and writing operations. If an error occurs, the tool logs the error and continues with the next file.
+- Using characters like `-`, `+` and tokens like `or`, `and`, etc. is not supported in flow definition, the migration tool does not handle this conversion.
+- It is a better practice to define global variables at the beginning of the flow. However, the migration tool does not enforce this.

examples/bots/abc/_rails.co

@@ -0,0 +1,8 @@
+import guardrails
+import nemoguardrails.library
+
+flow input rails $input_text
+    self check input
+
+flow output rails $output_text
+    self check output

examples/bots/abc/config.yml

@@ -7,29 +7,24 @@ instructions:
       If the bot does not know the answer to a question, it truthfully says it does not know.
 
 sample_conversation: |
-  user "Hi there. Can you help me with some questions I have about the company?"
-    express greeting and ask for assistance
-  bot express greeting and confirm and offer assistance
-    "Hi there! I'm here to help answer any questions you may have about the ABC Company. What would you like to know?"
-  user "What's the company policy on paid time off?"
-    ask question about benefits
-  bot respond to question about benefits
-    "The ABC Company provides eligible employees with up to two weeks of paid vacation time per year, as well as five paid sick days per year. Please refer to the employee handbook for more information."
+  user action: user said "Hi there. Can you help me with some questions I have about the company?"
+  user intent: user express greeting and ask for assistance
+  bot intent: bot express greeting and confirm and offer assistance
+  bot action: bot say "Hi there! I'm here to help answer any questions you may have about the ABC Company. What would you like to know?"
+  user action: user said "What's the company policy on paid time off?"
+  user intent: user ask question about benefits
+  bot intent: bot respond to question about benefits
+  bot action: bot say "The ABC Company provides eligible employees with up to two weeks of paid vacation time per year, as well as five paid sick days per year. Please refer to the employee handbook for more information."
+
 
 models:
   - type: main
     engine: openai
     model: gpt-3.5-turbo-instruct
 
 rails:
-  input:
-    flows:
-      - self check input
-
-  output:
-    flows:
-      - self check output
-
   dialog:
     single_call:
       enabled: False
+
+colang_version: "2.x"

examples/bots/abc/main.co

@@ -0,0 +1,5 @@
+import core
+import llm
+
+flow main
+  activate llm continuation