[FR] Adapt PyPi semver Library and Remove Custom

[terrancedejesus]

Issues

• [FR] Adapt PyPi semver Library and Remove Custom #2502

Summary

Remove custom semver library from Detection Rules and replace with 3rd-party semver library. Review all uses of custom library and adjust logic to pypi library.

Additional Information

• Replaces custom semver library with pypi library
• Adjusts semver imports
• Replaces typing use of Version with semver.VersionInfo
• Replaces use of Version with semver.VersionInfo or semver.VersionInfo.parse where applicable and necessary
• Adjusts logic of bump-versions to bump-pkg-versions in alignment with new detection rules release
• Adds find_latest_integration_version function to integrations.py - Used to dynamically determine what is bumped for patch releases (OOB)
• Hurdle - Since our version.lock.json file object relies on major.minor, we have to use lstrip and splitting to create the expected variables when passing parameters for some parts of the code. While adjustments are made, we will need to revisit the version.lock.json file and it's associating schema at a later date to be fully semantic versioned for consistency.

Testing

Build Release Packages and Update Rule Versions
Importance: The build-release --update-version-lock command tests our pre-built rules package building workflow. This workflow relies on version_lock.py, rule.py, schema validation, version.lock.json file references and more. It is expected that when running this release, the version.lock.json file will have changes that reflect expected rule updates with no double bumps, no forked rules adjustments and if versions are changed, the SHA256 was different.
Command: python -m detection_rules dev build-release --update-version-lock
Expected Results:

1. No errors
2. If rule changes, version bumps 1+ for current versions ONLY
3. Compare to running the same command from main branch, they should be identical

View Rule with Integration Tag
Importance: The view-rule command touches several parts of the code including to_api_format, related_integrations, version_lock.py and more. Parts of this code rely on semantic versioning to do related integration build time field comparisons, correctly load and interpret the version.lock.file file and convert a TOML rule to the Kibana API format which is necessary for package building and rule loading.
Command: python -m detection_rules view-rule /Users/tdejesus/code/src/detection-rules/rules/cross-platform/defense_evasion_elastic_agent_service_terminated.toml

Expected Results

{
  "author": [
    "Elastic"
  ],
  "description": "Identifies the Elastic endpoint agent has stopped and is no longer running on the host. Adversaries may attempt to disable security monitoring tools in an attempt to evade detection or prevention capabilities during an intrusion. This may also indicate an issue with the agent itself and should be addressed to ensure defensive measures are back in a stable state.",
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "eql",
  "license": "Elastic License v2",
  "name": "Elastic Agent Service Terminated",
  "note": "",
  "query": "process where\n/* net, sc or wmic stopping or deleting Elastic Agent on Windows */\n(event.type == \"start\" and\n  process.name : (\"net.exe\", \"sc.exe\", \"wmic.exe\",\"powershell.exe\",\"taskkill.exe\",\"PsKill.exe\",\"ProcessHacker.exe\") and\n  process.args : (\"stopservice\",\"uninstall\", \"stop\", \"disabled\",\"Stop-Process\",\"terminate\",\"suspend\") and\n  process.args : (\"elasticendpoint\", \"Elastic Agent\",\"elastic-agent\",\"elastic-endpoint\"))\nor\n/* service or systemctl used to stop Elastic Agent on Linux */\n(event.type == \"end\" and\n  (process.name : (\"systemctl\", \"service\") and\n    process.args : \"elastic-agent\" and\n    process.args : \"stop\")\n  or\n  /* Unload Elastic Agent extension on MacOS */\n  (process.name : \"kextunload\" and\n    process.args : \"com.apple.iokit.EndpointSecurity\" and\n    event.action : \"end\"))\n",
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "rule_id": "b627cd12-dac4-11ec-9582-f661ea17fbcd",
  "setup": "If enabling an EQL rule on a non-elastic-agent index (such as beats) for versions <8.2, events will not define `event.ingested` and default fallback for EQL rules was not added until 8.2, so you will need to add a custom pipeline to populate `event.ingested` to @timestamp for this rule to work.",
  "severity": "medium",
  "tags": [
    "Elastic",
    "Host",
    "Linux",
    "Windows",
    "macOS",
    "Threat Detection",
    "Defense Evasion"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0005",
        "name": "Defense Evasion",
        "reference": "https://attack.mitre.org/tactics/TA0005/"
      },
      "technique": [
        {
          "id": "T1562",
          "name": "Impair Defenses",
          "reference": "https://attack.mitre.org/techniques/T1562/",
          "subtechnique": [
            {
              "id": "T1562.001",
              "name": "Disable or Modify Tools",
              "reference": "https://attack.mitre.org/techniques/T1562/001/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "eql",
  "version": 101
}

Unit Tests
Importance: Unit testing is important for this PR to touch several parts of the code that could potentially rely on semantic versioning.
Command: pytest tests

Expected Results

❯ pytest tests                                                                                                                                           
============================================================================================================================================ test session starts =============================================================================================================================================
platform darwin -- Python 3.9.15, pytest-7.1.1, pluggy-1.0.0
rootdir: /Users/tdejesus/code/src/detection-rules, configfile: pyproject.toml
plugins: typeguard-2.13.3
collected 117 items                                                                                                                                                                                                                                                                                          

tests/test_all_rules.py ..................s..........                                                                                                                                                                                                                                                  [ 24%]
tests/test_gh_workflows.py .                                                                                                                                                                                                                                                                           [ 25%]
tests/test_mappings.py ...                                                                                                                                                                                                                                                                             [ 28%]
tests/test_packages.py .....                                                                                                                                                                                                                                                                           [ 32%]
tests/test_schemas.py ..........                                                                                                                                                                                                                                                                       [ 41%]
tests/test_toml_formatter.py ...                                                                                                                                                                                                                                                                       [ 43%]
tests/test_utils.py ....                                                                                                                                                                                                                                                                               [ 47%]
tests/test_version_locking.py .                                                                                                                                                                                                                                                                        [ 47%]
tests/kuery/test_dsl.py .......                                                                                                                                                                                                                                                                        [ 53%]
tests/kuery/test_eql2kql.py .........                                                                                                                                                                                                                                                                  [ 61%]
tests/kuery/test_evaluator.py .............                                                                                                                                                                                                                                                            [ 72%]
tests/kuery/test_kql2eql.py ..........                                                                                                                                                                                                                                                                 [ 81%]
tests/kuery/test_lint.py ............                                                                                                                                                                                                                                                                  [ 91%]
tests/kuery/test_parser.py ..........                                                                                                                                                                                                                                                                  [100%]

====================================================================================================================================== 116 passed, 1 skipped in 19.87s =======================================================================================================================================

Find Latest Integration Version
Importance: The find_latest_integration_version function was added to integrations.py to identify the latest integration version, either ga or beta depending on parameters passed. This is important for updating the integrations-pr command and eventually the release-fleet workflow. It is directly referenced in the bump-pkg-versions CLI command to help dynamically identify what the next beta or ga patch release version is by referencing EPR.
Commands:
Expected Results:

Bump Package Versions
Importance: The bump-pkg-versions will update the packages.yaml file contents and make necessary updates if major, minor or patch releases. Major and minor releases will save the changes to the packages.yaml as this will be used in the prep-for-next-release workflow so this file will be updated and then the packages.yaml file saved. Additionally it will be used for the integrations-pr to dynamically determine the correct package version
Command:
- python -m detection_rules dev bump-pkg-versions --minor-release --maturity="ga" (updates packages.yml)
- python -m detection_rules dev bump-pkg-versions --major-release --maturity="ga" (updates packages.yml)
- python -m detection_rules dev bump-pkg-versions --patch-release --maturity="ga"

Expected Results

Build Integrations PR
Importance: The integrations-pr command is used to make the necessary changes to the security_detection_engine package in your local integrations repository and create a PR. First a release package must exist locally in Detection Rules.
Command:

• python -m detection_rules dev build-release --update-version-lock
• python -m detection_rules dev integrations-pr /Users/tdejesus/code/src/integrations --assign terrancedejesus
  Expected Results: [Testing][Security Rules] Update security rules package to v8.7.0-beta.1 - DO NOT MERGE integrations#5176

Download ECS Schemas
Importance: ECS and beats schemas are important to doing rule validation. The ecs.py library now uses semver for version comparisons and downloading certain ECS packages.
Command:

from detection_rules.beats import download_latest_beats_schema
download_latest_beats_schema()

Expected Results

Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.6.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.6.0-rc1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.5.2: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.5.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.5.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.5.0-rc1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.4.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.4.0-rc1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.3.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.3.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.2.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.2.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.1.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.0.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/8.0.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.12.2: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.12.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.12.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.11.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.10.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.9.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.8.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.7.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.6.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.5.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.4.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.3.1: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.3.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.2.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to /Users/tdejesus/code/src/detection-rules/detection_rules/etc/ecs_schemas/1.1.0: 
        - ecs_flat.json.gz
        - ecs_nested.json.gz
Saved files to master_8.8.0-dev: 
        - ecs_flat.json.gz

Downgrade Functions
Importance: Downgrading allows a rule to be converted to the appropriate format depending on the stack version. This functionality exists in schemas.__init__.py where multiple migrate functions exist for each stack version. We add these functions at each "Prep for Next Branch" PR. The API format of the rule is passed in along with the target stack version and existing stack version which returns the appropriate API format of the rule at that target stack version.
Commands:

from detection_rules.rule_loader import RuleCollection
from detection_rules.schemas import downgrade
rules = RuleCollection.default()
rule = list(rules)[100]
rule_api = rule.contents.to_api_format()
downgrade(rule_api, "8.3", "8.4")

Expected Results

{'author': ['Elastic'],
 'description': 'Identifies the deletion of diagnostic settings in Azure, which send platform logs and metrics to different destinations. An adversary may delete diagnostic settings in an attempt to evade defenses.',
 'false_positives': ['Deletion of diagnostic settings may be done by a system or network administrator. Verify whether the username, hostname, and/or resource name should be making changes in your environment. Diagnostic settings deletion from unfamiliar users or hosts should be investigated. If known behavior is causing false positives, it can be exempted from the rule.'],
 'from': 'now-25m',
 'index': ['filebeat-*', 'logs-azure*'],
 'language': 'kuery',
 'license': 'Elastic License v2',
 'name': 'Azure Diagnostic Settings Deletion',
 'note': '',
 'query': 'event.dataset:azure.activitylogs and azure.activitylogs.operation_name:"MICROSOFT.INSIGHTS/DIAGNOSTICSETTINGS/DELETE" and event.outcome:(Success or success)\n',
 'references': ['https://docs.microsoft.com/en-us/azure/azure-monitor/platform/diagnostic-settings'],
 'risk_score': 47,
 'rule_id': '5370d4cd-2bb3-4d71-abf5-1e1d0ff5a2de',
 'severity': 'medium',
 'tags': ['Elastic',
  'Cloud',
  'Azure',
  'Continuous Monitoring',
  'SecOps',
  'Monitoring'],
 'threat': [{'technique': [{'name': 'Impair Defenses',
     'subtechnique': [{'name': 'Disable or Modify Tools',
       'reference': 'https://attack.mitre.org/techniques/T1562/001/',
       'id': 'T1562.001'}],
     'reference': 'https://attack.mitre.org/techniques/T1562/',
     'id': 'T1562'}],
   'framework': 'MITRE ATT&CK',
   'tactic': {'name': 'Defense Evasion',
    'reference': 'https://attack.mitre.org/tactics/TA0005/',
    'id': 'TA0005'}}],
 'timestamp_override': 'event.ingested',
 'type': 'query'}

Refresh ATT&CK Mappings
Importance: The semver library is used to compare ATT&CK package versions.
Command:
Rename detection_rules/etc/attack-v12.1.json.gz -> attack-v11.1.json.gz

• python -m detection_rules dev attack refresh-data
• python -m detection_rules dev attack refresh-redirect-mappings

Expected Results

❯ python -m detection_rules dev attack refresh-data                                                                   

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Replaced file: /Users/tdejesus/code/src/detection-rules/detection_rules/etc/attack-v11.1.json.gz with /Users/tdejesus/code/src/detection-rules/detection_rules/etc/attack-v12.1.0.json.gz

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom [✘!?] is 📦 v0.1.0 via 🐍 v3.9.15 (detection-rules-dev) on ☁️  terdeje50@gmail.com took 5s 
❯ python -m detection_rules dev attack refresh-data

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

No versions newer than the current detected: 12.1.0

[terrancedejesus]

Update: 1-29-2023

All Version() references have been replaced with semver.VersionInfo() or semver.VersionInfo.parse(). Since the old custom library has so many call references throughout the code, I will need to review all of these and ensure the expected results exist as they were before. As we adjust for the pypi semver library, we may need to adjust any major.minor references and have these be full semantic versioning so the code and static references are consistent, such as the version lock file and packages.yml file. This may require a separate PR considering the potential challenges and changes necessary from editing such important files in this repository.

Notes:

• To parse values that are only major.minor a list can be casted to semver.VersionInfo() which will by default give it a patch version of 0. Example - ver = "8.7" -> semver.VersionInfo(*ver.split(".")) -> VersionInfo(major=8, minor=7, patch=0, prerelease=None, build=None)
• To parse a full semantic version string, use semver.VersionInfo.parse(). Example = ver = "8.7.1" -> semver.VersionInfo.parse(ver) -> VersionInfo(major=8, minor=7, patch=1, prerelease=None, build=None)
• Major, Minor and Patch can be called simply as a method of the VersionInfo class and returns int -> ver.major or ver.minor
• Bumping a version will reset anything proceeding it, so ver.bump_major() will bump the major 1+ and reset minor, patch and release to 0.
• While we are using the 2 major version, 3 is currently only in development but changes some of the importing to from semver.version import Version for example.

There is a lot more functionality that can be read in the docs.

[terrancedejesus]

Thanks for the thorough explanation and walkthrough of testing, that helps a lot (I had been avoiding this refactor for a while because of all the nuances of versions used)!

I see a lot of unrelated refactor of code (imports, formatting, etc.). Can you roll all the non semver related changes back to gives us a clearer delta of these changes since they are so sensitive (and if desired, you can push the reformatting to another PR).

Certainly, I'll review the comments and re-formatting this morning! Thanks for the review. By chance, do we have a specific preference when sorting imports? I assumed as long as they are sorted (which I do via VSCode) then there is no harm done as it follows the standard pythonic import hygiene.

[Mikaayenson]

🥇 Fantastic work @terrancedejesus

Here's a summary of the tests I replicated in addition to the code review.

🟢 : python -m detection_rules dev build-release --update-version-lock
🟢 : python -m detection_rules view-rule /Users/stryker/workspace/Elastic/detection-rules/rules/cross-platform/defense_evasion_elastic_agent_service_terminated.toml
🟢 : pytest tests
🟢 : python -m detection_rules dev bump-pkg-versions --major-release --maturity="ga"
🟢 : Download ECS Schemas
🟢 : Downgrade Functions
🟢 : python -m detection_rules dev attack refresh-data

[eric-forte-elastic]

Looks great! 🥳 ✔️

In addition to considerable peer review and manual review of code also performed the following checks

Tested

✔️ Build Release Packages and Update Rule Versions

Output

• No errors
• +1 Bump for current version only

• Has the same result on main

✔️ View Rule with Integration Tag

Output

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom [!] is  v0.1.0 via  v3.8.10 (detection-rules-build) on  eric.forte took 24s 
❯ python -m detection_rules view-rule rules/cross-platform/defense_evasion_elastic_agent_service_terminated.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

{
  "author": [
    "Elastic"
  ],
  "description": "Identifies the Elastic endpoint agent has stopped and is no longer running on the host. Adversaries may attempt to disable security monitoring tools in an attempt to evade detection or prevention capabilities during an intrusion. This may also indicate an issue with the agent itself and should be addressed to ensure defensive measures are back in a stable state.",
  "from": "now-9m",
  "index": [
    "logs-endpoint.events.*"
  ],
  "language": "eql",
  "license": "Elastic License v2",
  "name": "Elastic Agent Service Terminated",
  "note": "",
  "query": "process where\n/* net, sc or wmic stopping or deleting Elastic Agent on Windows */\n(event.type == \"start\" and\n  process.name : (\"net.exe\", \"sc.exe\", \"wmic.exe\",\"powershell.exe\",\"taskkill.exe\",\"PsKill.exe\",\"ProcessHacker.exe\") and\n  process.args : (\"stopservice\",\"uninstall\", \"stop\", \"disabled\",\"Stop-Process\",\"terminate\",\"suspend\") and\n  process.args : (\"elasticendpoint\", \"Elastic Agent\",\"elastic-agent\",\"elastic-endpoint\"))\nor\n/* service or systemctl used to stop Elastic Agent on Linux */\n(event.type == \"end\" and\n  (process.name : (\"systemctl\", \"service\") and\n    process.args : \"elastic-agent\" and\n    process.args : \"stop\")\n  or\n  /* Unload Elastic Agent extension on MacOS */\n  (process.name : \"kextunload\" and\n    process.args : \"com.apple.iokit.EndpointSecurity\" and\n    event.action : \"end\"))\n",
  "related_integrations": [
    {
      "package": "endpoint",
      "version": "^8.2.0"
    }
  ],
  "required_fields": [
    {
      "ecs": true,
      "name": "event.action",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "event.type",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.args",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "process.name",
      "type": "keyword"
    }
  ],
  "risk_score": 47,
  "rule_id": "b627cd12-dac4-11ec-9582-f661ea17fbcd",
  "setup": "If enabling an EQL rule on a non-elastic-agent index (such as beats) for versions <8.2, events will not define `event.ingested` and default fallback for EQL rules was not added until 8.2, so you will need to add a custom pipeline to populate `event.ingested` to @timestamp for this rule to work.",
  "severity": "medium",
  "tags": [
    "Elastic",
    "Host",
    "Linux",
    "Windows",
    "macOS",
    "Threat Detection",
    "Defense Evasion"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0005",
        "name": "Defense Evasion",
        "reference": "https://attack.mitre.org/tactics/TA0005/"
      },
      "technique": [
        {
          "id": "T1562",
          "name": "Impair Defenses",
          "reference": "https://attack.mitre.org/techniques/T1562/",
          "subtechnique": [
            {
              "id": "T1562.001",
              "name": "Disable or Modify Tools",
              "reference": "https://attack.mitre.org/techniques/T1562/001/"
            }
          ]
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "eql",
  "version": 101
}

✔️ Unit Tests

Output

LINTING
./env/detection-rules-build/bin/python -m flake8 tests detection_rules --ignore D203 --max-line-length 120
./env/detection-rules-build/bin/python -m detection_rules test

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

============================== test session starts ===============================
platform linux -- Python 3.8.10, pytest-7.2.0, pluggy-1.0.0 -- /home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/bin/python
cachedir: .pytest_cache
rootdir: /home/forteea1/Code/elastic/detection-rules, configfile: pyproject.toml
plugins: typeguard-2.13.3
collected 117 items                                                              

tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized PASSED [  0%]
tests/test_all_rules.py::TestValidRules::test_duplicate_file_names PASSED  [  1%]
tests/test_all_rules.py::TestValidRules::test_file_names PASSED            [  2%]
tests/test_all_rules.py::TestValidRules::test_production_rules_have_rta PASSED [  3%]
tests/test_all_rules.py::TestValidRules::test_rule_type_changes PASSED     [  4%]
tests/test_all_rules.py::TestValidRules::test_schema_and_dupes PASSED      [  5%]
tests/test_all_rules.py::TestThreatMappings::test_duplicated_tactics PASSED [  5%]
tests/test_all_rules.py::TestThreatMappings::test_tactic_to_technique_correlations PASSED [  6%]
tests/test_all_rules.py::TestThreatMappings::test_technique_deprecations PASSED [  7%]
tests/test_all_rules.py::TestRuleTags::test_casing_and_spacing PASSED      [  8%]
tests/test_all_rules.py::TestRuleTags::test_primary_tactic_as_tag PASSED   [  9%]
tests/test_all_rules.py::TestRuleTags::test_required_tags PASSED           [ 10%]
tests/test_all_rules.py::TestRuleTimelines::test_timeline_has_title PASSED [ 11%]
tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED  [ 11%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED    [ 12%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED     [ 13%]
tests/test_all_rules.py::TestRuleMetadata::test_updated_date_newer_than_creation PASSED [ 14%]
tests/test_all_rules.py::TestIntegrationRules::test_all_min_stack_rules_have_comment PASSED [ 15%]
tests/test_all_rules.py::TestIntegrationRules::test_integration_guide SKIPPED [ 16%]
tests/test_all_rules.py::TestIntegrationRules::test_rule_demotions PASSED  [ 17%]
tests/test_all_rules.py::TestRuleTiming::test_eql_interval_to_maxspan PASSED [ 17%]
tests/test_all_rules.py::TestRuleTiming::test_eql_lookback PASSED          [ 18%]
tests/test_all_rules.py::TestRuleTiming::test_event_override PASSED        [ 19%]
tests/test_all_rules.py::TestRuleTiming::test_required_lookback PASSED     [ 20%]
tests/test_all_rules.py::TestLicense::test_elastic_license_only_v2 PASSED  [ 21%]
tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields PASSED [ 22%]
tests/test_all_rules.py::TestBuildTimeFields::test_build_fields_min_stack PASSED [ 23%]
tests/test_all_rules.py::TestRiskScoreMismatch::test_rule_risk_score_severity_mismatch PASSED [ 23%]
tests/test_all_rules.py::TestOsqueryPluginNote::test_note_guide PASSED     [ 24%]
tests/test_gh_workflows.py::TestWorkflows::test_matrix_to_lock_version_defaults PASSED [ 25%]
tests/test_mappings.py::TestMappings::test_false_positives PASSED          [ 26%]
tests/test_mappings.py::TestMappings::test_true_positives PASSED           [ 27%]
tests/test_mappings.py::TestRTAs::test_rtas_with_triggered_rules_have_uuid PASSED [ 28%]
tests/test_packages.py::TestPackages::test_package_loader_default_configs PASSED [ 29%]
tests/test_packages.py::TestPackages::test_package_loader_production_config PASSED [ 29%]
tests/test_packages.py::TestPackages::test_package_summary PASSED          [ 30%]
tests/test_packages.py::TestPackages::test_rule_versioning PASSED          [ 31%]
tests/test_packages.py::TestRegistryPackage::test_registry_package_config PASSED [ 32%]
tests/test_schemas.py::TestSchemas::test_eql_validation PASSED             [ 33%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_7_x PASSED        [ 34%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_8_x PASSED        [ 35%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_7_x PASSED    [ 35%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_8_x PASSED    [ 36%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_7_x PASSED    [ 37%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_8_x PASSED    [ 38%]
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_has_nested_previous PASSED [ 39%]
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_no_previous PASSED [ 40%]
tests/test_schemas.py::TestVersions::test_stack_schema_map PASSED          [ 41%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_deep PASSED [ 41%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_rule PASSED [ 42%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_normalization PASSED [ 43%]
tests/test_utils.py::TestTimeUtils::test_caching PASSED                    [ 44%]
tests/test_utils.py::TestTimeUtils::test_event_class_normalization PASSED  [ 45%]
tests/test_utils.py::TestTimeUtils::test_schema_multifields PASSED         [ 46%]
tests/test_utils.py::TestTimeUtils::test_time_normalize PASSED             [ 47%]
tests/test_version_locking.py::TestVersionLock::test_previous_entries_gte_current_min_stack PASSED [ 47%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_and_query PASSED               [ 48%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_exists PASSED            [ 49%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_inequality PASSED        [ 50%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_match PASSED             [ 51%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_not_query PASSED               [ 52%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_optimizations PASSED           [ 52%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_or_query PASSED                [ 53%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_and_query PASSED            [ 54%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_boolean_precedence PASSED   [ 55%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_equals PASSED         [ 56%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_inequality PASSED     [ 57%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_ip_checks PASSED            [ 58%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_list_of_values PASSED       [ 58%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_not_query PASSED            [ 59%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_or_query PASSED             [ 60%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_wildcard_field PASSED       [ 61%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_expr PASSED        [ 62%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_values PASSED      [ 63%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_cidr_match PASSED      [ 64%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_field_exists PASSED    [ 64%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_flattening PASSED      [ 65%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_list_value PASSED      [ 66%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_not_value PASSED       [ 67%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_expr PASSED         [ 68%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_values PASSED       [ 69%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_quoted_wildcard PASSED [ 70%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_range PASSED           [ 70%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_single_value PASSED    [ 71%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_wildcard PASSED        [ 72%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_and_query PASSED            [ 73%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_boolean_precedence PASSED   [ 74%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_equals PASSED         [ 75%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_inequality PASSED     [ 76%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_list_of_values PASSED       [ 76%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_lone_value PASSED           [ 77%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_nested_query PASSED         [ 78%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_not_query PASSED            [ 79%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_or_query PASSED             [ 80%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_schema PASSED               [ 81%]
tests/kuery/test_lint.py::LintTests::test_and_not PASSED                   [ 82%]
tests/kuery/test_lint.py::LintTests::test_compound PASSED                  [ 82%]
tests/kuery/test_lint.py::LintTests::test_double_negate PASSED             [ 83%]
tests/kuery/test_lint.py::LintTests::test_extract_not PASSED               [ 84%]
tests/kuery/test_lint.py::LintTests::test_ip PASSED                        [ 85%]
tests/kuery/test_lint.py::LintTests::test_lint_field PASSED                [ 86%]
tests/kuery/test_lint.py::LintTests::test_lint_precedence PASSED           [ 87%]
tests/kuery/test_lint.py::LintTests::test_merge_fields PASSED              [ 88%]
tests/kuery/test_lint.py::LintTests::test_mixed_demorgans PASSED           [ 88%]
tests/kuery/test_lint.py::LintTests::test_not_demorgans PASSED             [ 89%]
tests/kuery/test_lint.py::LintTests::test_not_or PASSED                    [ 90%]
tests/kuery/test_lint.py::LintTests::test_upper_tokens PASSED              [ 91%]
tests/kuery/test_parser.py::ParserTests::test_conversion PASSED            [ 92%]
tests/kuery/test_parser.py::ParserTests::test_date PASSED                  [ 93%]
tests/kuery/test_parser.py::ParserTests::test_keyword PASSED               [ 94%]
tests/kuery/test_parser.py::ParserTests::test_list_equals PASSED           [ 94%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_fail PASSED   [ 95%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_success PASSED [ 96%]
tests/kuery/test_parser.py::ParserTests::test_number_exists PASSED         [ 97%]
tests/kuery/test_parser.py::ParserTests::test_number_wildcard_fail PASSED  [ 98%]
tests/kuery/test_parser.py::ParserTests::test_type_family_fail PASSED      [ 99%]
tests/kuery/test_parser.py::ParserTests::test_type_family_success PASSED   [100%]

================================ warnings summary ================================
env/detection-rules-build/lib/python3.8/site-packages/_pytest/config/__init__.py:1171
  /home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/_pytest/config/__init__.py:1171: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: typeguard
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================== 116 passed, 1 skipped, 1 warning in 22.76s ===================

✔️ Bump Package Versions

Output

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom is  v0.1.0 via  v3.8.10 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules dev bump-pkg-versions --minor-release --maturity="ga"

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Kibana version: 8.8
Package Kibana version: ^8.8.0
Package version: 8.8.0-beta.1

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom [!] is  v0.1.0 via  v3.8.10 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules dev bump-pkg-versions --major-release --maturity="ga"

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Kibana version: 9
Package Kibana version: ^9.0.0
Package version: 9.0.0-beta.1

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom [!] is  v0.1.0 via  v3.8.10 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules dev bump-pkg-versions --patch-release --maturity="ga"

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/forteea1/Code/elastic/detection-rules/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/home/forteea1/Code/elastic/detection-rules/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/forteea1/Code/elastic/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/forteea1/Code/elastic/detection-rules/detection_rules/devtools.py", line 181, in bump_versions
    latest_patch_release_ver = find_latest_integration_version("security_detection_engine",
  File "/home/forteea1/Code/elastic/detection-rules/detection_rules/integrations.py", line 243, in find_latest_integration_version
    existing_pkgs = get_integration_manifests(integration, prerelease, str(stack_version))
  File "/home/forteea1/Code/elastic/detection-rules/detection_rules/integrations.py", line 232, in get_integration_manifests
    raise ValueError(f"EPR search for {integration} integration package returned empty list")
ValueError: EPR search for security_detection_engine integration package returned empty list

✔️ Refresh ATT&CK Mappings

Output

detection-rules on  2502-fr-adapt-pypi-semver-library-and-remove-custom [✘?] is  v0.1.0 via  v3.8.10 (detection-rules-build) on  eric.forte took 3m39s 
❯ python -m detection_rules dev attack refresh-data

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Replaced file: /home/forteea1/Code/elastic/detection-rules/detection_rules/etc/attack-v11.1.json.gz with /home/forteea1/Code/elastic/detection-rules/detection_rules/etc/attack-v12.1.0.json.gz