[FR] Add support for New Terms Fields and Window Start History

[terrancedejesus]

Issues

• [FR] Add Support for "New Terms" Field in Rules #2346
• [Security Solution][Alerts] New terms security rule type kibana#134526

Summary

Adds support for new_terms_field and window_start_history fields in detection rules. These rules are compatible with stack versions 8.4+. The following needs to be completed for this PR.

• Update Definitions.py to include new fields and minimum stack requirements
• Add NewTermsRuleData data class in rule.py for the new rule type
• Create validate method in NewTermsRuleData class to add specific validation for New Terms rule types
• Create transform method in NewTermsRuleData class to pop new terms key:value to root of JSON after validation
• Add multiple assertions in validate method

Testing

The following rule was made locally to test this by using the command -> python -m detection_rules view-rule /Users/tdejesus/code/src/detection-rules/rules/network/command_and_control_uncommon_well_known_port.toml

Example Rule

[metadata]
creation_date = "2020/10/16"
maturity = "production"
min_stack_comments = "New fields available: new_terms_fields and window_start_history"
min_stack_version = "8.4.0"
updated_date = "2022/10/16"

[rule]
author = ["Elastic"]
description = "This rule detects outbound network connections to a well known port that has not been seen within the last 30 days."
false_positives = [
    """
    Introducing new services that used well-known ports may cause outbound traffic to specific ports that would meet the
    criteria of this rule.
    """,
]
from = "now-9m"
index = ["logs-endpoint.events.*"]
language = "kuery"
license = "Elastic License v2"
name = "Outbound Traffic to Uncommon Well Known Port"
risk_score = 21
rule_id = "ecaa877a-6abc-11ed-82c5-f661ea17fbcd"
severity = "low"
tags = ["Elastic", "Host", "Network", "Threat Detection", "Command and Control", "Host"]
timestamp_override = "event.ingested"
type = "new_terms"

query = '''
event.category:(network or network_traffic) and destination.port < 1024
'''


[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1021"
name = "Remote Services"
reference = "https://attack.mitre.org/techniques/T1021/"


[rule.threat.tactic]
id = "TA0011"
name = "Command and Control"
reference = "https://attack.mitre.org/tactics/TA0011/"


[rule.new_terms]
field = "new_terms_fields"
value = ["destination.port"]

[[rule.new_terms.history_window_start]]
field = "history_window_start"
value = "now-30d"

The view-rule CLI command from devtools was used to view this rule which will then transform it to API format and produce the expected JSON ouput. This process calls on existing data validators and schemas which would fail if the new fields were unrecognized or did not follow proper schema.

View Rule Output

detection-rules on  2346-fr-add-support-for-new-terms-field-in-rules [$!?] via 🐍 v3.9.13 (detection-rules-dev) on ☁️   
❯  cd /Users/tdejesus/code/src/detection-rules ; /usr/bin/env /Users/tdejesus/.virtualenvs/detection-rules-dev/bin/python /Users/tdejesus/.vscode/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 60439 -- -m detection_rules view-rule /Users/tdejesus/code/src/detection-rules/rules/network/command_and_control_uncommon_well_known_port.toml 
Loaded config file: /Users/tdejesus/code/src/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule detects outbound network connections to a well known port that has not been seen within the last 30 days.",
  "false_positives": [
    "Introducing new services that used well-known ports may cause outbound traffic to specific ports that would meet the criteria of this rule."
  ],
  "from": "now-9m",
  "history_window_start": "now-30d",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Outbound Traffic to Uncommon Well Known Port",
  "new_terms_fields": [
    "destination.port"
  ],
  "query": "event.category:(network or network_traffic) and destination.port < 1024\n",
  "required_fields": [
    {
      "ecs": true,
      "name": "destination.port",
      "type": "long"
    },
    {
      "ecs": true,
      "name": "event.category",
      "type": "keyword"
    }
  ],
  "risk_score": 21,
  "rule_id": "ecaa877a-6abc-11ed-82c5-f661ea17fbcd",
  "severity": "low",
  "tags": [
    "Elastic",
    "Host",
    "Network",
    "Threat Detection",
    "Command and Control",
    "Host"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0011",
        "name": "Command and Control",
        "reference": "https://attack.mitre.org/tactics/TA0011/"
      },
      "technique": [
        {
          "id": "T1021",
          "name": "Remote Services",
          "reference": "https://attack.mitre.org/techniques/T1021/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "new_terms",
  "version": 1
}

[terrancedejesus]

@brokensound77 the following marshmallow error appears when attempting to run the following command (TOML rule is in the original comment).

Command: python -m detection_rules view-rule rules/network/command_and_control_new_terms_destination_port.toml

marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.'], 'new_terms': ['Unknown field.']}), ValidationError({'threshold': ['Missing data for required field.'], 'type': ['Must be equal to threshold.'], 'new_terms': ['Unknown field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.'], 'new_terms': ['Unknown field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'machine_learning_job_id': ['Missing data for required field.'], 'anomaly_threshold': ['Missing data for required field.'], 'index': ['Unknown field.'], 'new_terms': ['Unknown field.'], 'query': ['Unknown field.'], 'language': ['Unknown field.']}), ValidationError({'type': ['Must be equal to query.'], 'new_terms': ['Unknown field.']}), ValidationError({'new_terms': ['Unknown field.']})]}

I assume it is related to the NewTermsRuleData class in rule.py but I am unable to determine what exactly is the issue. Any ideas?

[terrancedejesus]

I have identified the issue with the NewTermsRuleData class. Addressed issues and the view-rule CLI command returns the rule in JSON format, meaning it passed validation and converted the rule.

Results

❯ python -m detection_rules view-rule rules/network/command_and_control_new_terms_destination_port.toml                                               
Loaded config file: /Users/tdejesus/code/src/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule detects outbound network connections to a well known port that has not been seen within the last 30 days.",
  "false_positives": [
    "Introducing new services that used well-known ports may cause outbound traffic to specific ports that would meet the criteria of this rule."
  ],
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Outbound Traffic to Uncommon Well Known Port",
  "new_terms": {
    "field": "new_terms_field",
    "history_window_start": [
      {
        "field": "history_window_start",
        "value": "now-30d"
      }
    ],
    "value": [
      "destination.port"
    ]
  },
  "query": "event.category:(network or network_traffic) and destination.port < 1024\n",
  "required_fields": [
    {
      "ecs": true,
      "name": "destination.port",
      "type": "long"
    },
    {
      "ecs": true,
      "name": "event.category",
      "type": "keyword"
    }
  ],
  "risk_score": 21,
  "rule_id": "34fde489-94b0-4500-a76f-b8a157cf9269",
  "severity": "low",
  "tags": [
    "Elastic",
    "Host",
    "Network",
    "Threat Detection",
    "Command and Control",
    "Host"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0011",
        "name": "Command and Control",
        "reference": "https://attack.mitre.org/tactics/TA0011/"
      },
      "technique": [
        {
          "id": "T1021",
          "name": "Remote Services",
          "reference": "https://attack.mitre.org/techniques/T1021/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "new_terms",
  "version": 101
}

[terrancedejesus]

Updated this to be current and match the stack version.

[eric-forte-elastic]

Great walk-through and discussion from @terrancedejesus and @Mikaayenson about this on Friday 12/02. Local test results on my machine match expected output. All looks good to me!

Local Test Output

detection-rules on  2346-fr-add-support-for-new-terms-field-in-rules [?] via  v3.8.10 (venv) on  eric.forte
❯ python -m detection_rules view-rule test_rule.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "This rule detects outbound network connections to a well known port that has not been seen within the last 30 days.",
  "false_positives": [
    "Introducing new services that used well-known ports may cause outbound traffic to specific ports that would meet the criteria of this rule."
  ],
  "from": "now-9m",
  "history_window_start": "now-30d",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "kuery",
  "license": "Elastic License v2",
  "name": "Outbound Traffic to Uncommon Well Known Port",
  "new_terms_fields": [
    "destination.port"
  ],
  "query": "event.category:(network or network_traffic) and destination.port < 1024\n",
  "required_fields": [
    {
      "ecs": true,
      "name": "destination.port",
      "type": "long"
    },
    {
      "ecs": true,
      "name": "event.category",
      "type": "keyword"
    }
  ],
  "risk_score": 21,
  "rule_id": "ecaa877a-6abc-11ed-82c5-f661ea17fbcd",
  "severity": "low",
  "tags": [
    "Elastic",
    "Host",
    "Network",
    "Threat Detection",
    "Command and Control",
    "Host"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0011",
        "name": "Command and Control",
        "reference": "https://attack.mitre.org/tactics/TA0011/"
      },
      "technique": [
        {
          "id": "T1021",
          "name": "Remote Services",
          "reference": "https://attack.mitre.org/techniques/T1021/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "new_terms",
  "version": 1
}

[Mikaayenson]

1. tested with sample rule new_terms load as expected
2. tested field name is correct (recommending add an assert)
3. tested schemas with no min_stack (recommending add a default)
4. tested the values (no validation on window)
5. tested number of new terms for 8.4.0
6. tested duplicated of terms
7. tested number of terms (more than 3, less than 1)
8. observed new_terms_field added to root of output.

Left a couple small recommendations.

[Mikaayenson]

Nit: It's a shame that we don't add validation to the value.

[terrancedejesus]

Digging into this at the moment. In L188 in rule.py we only ensure that the value of this field is a string.

In reference to L478 of rule.py, it seems that we have some additional logic for EQL rules because the possibility of conflicts between maxspax, lookback, and interval fields which is why they are methods of the EQLRuleData class and not inside QueryRuleData.

The look_back method of the EQLRuleData class is only called in the unit test test_all_rules.test_eql_lookback and converts, for example now-9m into milliseconds, ultimately to compare it with the maxspan value of an EQL rule to ensure it is greater than.

So in all, it appears we have no validation of the from field for rules, but we have comparison logic to ensure rule devs have entered maxspan and interval ranges that would not conflict. It is up to the rule developer to determine. If we do add logic, it should probably be in another PR where we determine what is compatible with Kibana, cover test/edge cases, etc while referencing https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math.

[terrancedejesus]

Added new issue for this -> #2416

[Mikaayenson]

LGTM Great work!