[FR] [DAC] Import Exceptions from API Export and ndjson

[eric-forte-elastic]

Issues

#3674

Summary

This PR adds support for exporting exception lists from Kibana that are attached to rules that we export. The existing API functionality included exporting exceptions. However, these still needed to be parsed. The main logic is to handle the API response that returns the exception objects in a list after the rule objects. Some of these objects are containers while others are the exceptions in the containers, and these have to be put together after the API call. The current naming convention for rule exception list files is {exceptions_list_id}_exceptions.toml.

This PR is the export part of that functionality.
Note: the prior exception list logic does not support importing to Kibana. This PR contains only the export logic, a separate PR: #3870 will be needed to support importing into Kibana.

Details

Current output if you attempt to import rules with exceptions list (the exceptions list does not get moved with the rule).

2 rule(s) failed to import!
 - 794d2fc0-ecd0-4963-99da-fd587666b80d: (400) Rule with rule_id: "794d2fc0-ecd0-4963-99da-fd587666b80d" references a non existent exception list of list_id: "1ce8efca-877d-4d4a-a069-6d41aa77e0d9". Reference has been removed.
 - 7c22a9d2-5910-4da2-92af-7ff7481bd0f7: (400) Rule with rule_id: "7c22a9d2-5910-4da2-92af-7ff7481bd0f7" references a non existent exception list of list_id: "ad78032a-6730-44c1-8ec3-129ff1dc2ad9". Reference has been removed.

Additional note: while it may be desirable to also provide support for pulling exception lists when one has a .ndjson export from Kibana, this may be blocked by #3863.

Kibana API Docs

https://www.elastic.co/guide/en/security/current/exceptions-api-get-item.html
https://www.elastic.co/guide/en/security/current/exceptions-api-create-container.html

Testing

To test this code:
0. If you have not already please update your config to include actions and exception directories.

Example Config

bbr_rules_dirs:
- rules_building_block
files:
  deprecated_rules: etc/deprecated_rules.json
  packages: etc/packages.yaml
  stack_schema_map: etc/stack-schema-map.yaml
  version_lock: etc/version.lock.json
directories:
  action_dir: actions
  exception_dir: exceptions
rule_dirs:
- rules
testing:
  config: etc/test_config.yaml
bypass_version_lock: True
normalize_kql_keywords: False

1. Create a rule with one or more exceptions in the Kibana API.

Details

2. Next, run the export rules command with the new -e flag to also export the exceptions.
   Example Command: python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv -e
   Note: Depending on how many custom rules you have in your Kibana instance, your numbers may be slightly different.

Expected Output

detection-rules on  3674-frdac-add-exceptions-importing-from-ndjson [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s
❯ python -m detection_rules kibana export-rules -d custom_rules/rules -s -sv -e
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄ ▄ █▀▀▄ ▄ ▄ ▄ ▄▄▄ ▄▄▄
█ █ █▄▄ █ █▄▄ █ █ █ █ █ █▀▄ █ █▄▄▀ █ █ █ █▄▄ █▄▄
█▄▄▀ █▄▄ █ █▄▄ █▄▄ █ ▄█▄ █▄█ █ ▀▄█ █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

8 rules exported
8 rules converted
2 exceptions exported
8 saved to custom_rules/rules

3. Check to see if your exception toml file exists in the exceptions/ folder in your custom rules directory.

Example Exception File Contents

[metadata]
creation_date = "2024/07/01"
list_name = "Exceptions for rule - Test Exception List"
rule_ids = ["7c22a9d2-5910-4da2-92af-7ff7481bd0f7"]
rule_names = ["Test Exception List"]
updated_date = "2024/07/10"

[[exceptions]]

[exceptions.container]
description = "Exception list containing exceptions for rule with id: 77260f65-d17e-468b-8fe9-305048404e95"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
name = "Exceptions for rule - Test Exception List"
namespace_type = "single"
tags = ["not_default_rule_exception_list"]
type = "rule_default"

[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "8dffbaa5-8068-4b7c-b4cf-ef17f627dc00"
name = "TestShareItem"
namespace_type = "single"
tags = []
type = "simple"

[[exceptions.items.entries]]
field = "Effective_process.pid"
type = "match"
operator = "included"
value = "1"

[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "970945dd-71d5-4128-89a8-7e8689326a19"
name = "Pid not One"
namespace_type = "single"
tags = []
type = "simple"

[[exceptions.items.entries]]
field = "Effective_process.pid"
type = "match"
operator = "included"
value = "1"

[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
item_id = "d6a0e21c-bf41-4758-a522-cca5df3a2332"
name = "FakeRoot"
namespace_type = "single"
tags = []
type = "simple"

[[exceptions.items.entries]]
field = "process.name"
type = "match"
operator = "included"
value = "FakeRootUpdatedFour"

Example Shared Exceptions List

[metadata]
creation_date = "2024/07/10"
list_name = "TestShareList"
rule_ids = ["7c22a9d2-5910-4da2-92af-7ff7481bd0f7", "521629d1-61e4-4203-8a16-a08d5dc20281", "222e1b03-fdc9-42a5-911e-2e3e0533549a"]
rule_names = ["Test Exception List", "INFOSEC CSS Test Rule", "DAC Demo Dev Rule 13"]
updated_date = "2024/07/10"

[[exceptions]]

[exceptions.container]
description = "TestShareList Desc"
list_id = "dbc9b368-5d39-41fa-9a16-bfcb995fc866"
name = "TestShareList"
namespace_type = "single"
tags = []
type = "detection"

[[exceptions.items]]
comments = []
description = "Exception list item"
list_id = "dbc9b368-5d39-41fa-9a16-bfcb995fc866"
item_id = "7c823cd0-ca30-46ba-af35-3633219eed1f"
name = "AllOSTestShare"
namespace_type = "single"
tags = []
type = "simple"

[[exceptions.items.entries]]
field = "Effective_process.name"
type = "match"
operator = "included"
value = "BadRooTFour"

Example Rule File Toml (in rules directory of custom rules dir) that is attached to the above exception list

[metadata]
creation_date = "2024/07/03"
maturity = "production"
updated_date = "2024/07/03"

[rule]
actions = []
author = ["Elastic"]
description = "Test Exception List"
enabled = true
false_positives = []
filters = []
from = "now-18060s"
index = [
    "apm-*-transaction*",
    "auditbeat-*",
    "endgame-*",
    "filebeat-*",
    "logs-*",
    "packetbeat-*",
    "traces-apm*",
    "winlogbeat-*",
    "-*elastic-cloud-logs-*",
]
interval = "5h"
language = "eql"
license = ""
max_signals = 100
name = "Test Exception List"
references = []
related_integrations = []
required_fields = []
risk_score = 21
risk_score_mapping = []
rule_id = "7c22a9d2-5910-4da2-92af-7ff7481bd0f7"
setup = ""
severity = "low"
severity_mapping = []
tags = []
threat = []
to = "now"
type = "eql"

query = '''
process where true
'''


[[rule.exceptions_list]]
id = "222e1466-6dee-49ed-bb40-b7791891dc90"
list_id = "ad78032a-6730-44c1-8ec3-129ff1dc2ad9"
namespace_type = "single"
type = "rule_default"

[rule.meta]
from = "1m"
kibana_siem_app_url = "https://dev-deployment-2c684a.kb.us-central1.gcp.cloud.es.io:9243/app/security"

Testing from ndjson

To test importing from ndjson run the following command with an appropriate rule export for a rule(s) that container exceptions, and see that an exceptions TOML file is created. The -e flag is used to specify importing exceptions

python -m detection_rules import-rules-to-repo ~/Downloads/rules_export_exception_list.ndjson -s custom_rules/rules --required-only -e

Example ndjson Export

rules_export_exception_list.ndjson.txt

Example Output

❯ python -m detection_rules import-rules-to-repo ~/Downloads/rules_export_exception_list.ndjson -s custom_rules/rules --required-only -e
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building exception(s) for custom_rules/exceptions/Test Exception List_exceptions.toml
[+] Building rule for custom_rules/rules/test_exception_list.toml

[eric-forte-elastic]

Note this PR has a minor update to the schemas and GenericLoader. The prior _assert_new for the GenericLoader assumes that each item will have a unique name and ID, which the Kibana schema does not enforce. If you write a new Exceptions TOML File you do not need to provide an id as one will be generated for you. However, you do need to provide a name. As such, we remove the ID check and just kept the name check. Additionally since each Toml File represents a single Exceptions List, the list's name was added to the metadata for ease of use.

[terrancedejesus]

Approving ✅
Reviewed with Eric and no further questions or concerns.