Skip to content

[FR] [DAC] Add Flag to Enable All Prebuilt Tests #3684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
May 16, 2024
Merged

[FR] [DAC] Add Flag to Enable All Prebuilt Tests #3684

merged 3 commits into from
May 16, 2024

Conversation

eric-forte-elastic
Copy link
Contributor

@eric-forte-elastic eric-forte-elastic commented May 16, 2024 •
edited
Loading

Issues

#3682

Summary

This PR changed the behavior of setup-config to by bypass the following unit tests (specific to Elastic package release version locking)

  - tests.test_gh_workflows.TestWorkflows.test_matrix_to_lock_version_defaults
  - tests.test_schemas.TestVersionLockSchema.test_version_lock_has_nested_previous
  - tests.test_packages.TestRegistryPackage.test_registry_package_config
  - tests.test_all_rules.TestValidRules.test_schema_and_dupes

It also introduces a flag -e to allow one to change this behavior to have these bypasses commented out to enable all unit tests.

Testing

To test try making configs without the -e flag and with the flag. After this, run make test with your non--e config specified through the CUSTOM_RULES_DIR environment variable. Move at least one rule into the directory. (Optional but for more through testing) All tests should pass with rules or without any rules.

python -m detection_rules custom-rules setup-config test-dir

Output 

detection-rules on  3682-frdac-unit-test-locked-version-support-for-custom-config [!] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules custom-rules setup-config test-dir

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Created directory: test-dir/actions
Created directory: test-dir/exceptions
Created directory: test-dir/rules
Created directory: test-dir/rules_building_block
Created directory: test-dir/etc
Created file with default content: test-dir/etc/deprecated_rules.json
Created file with default content: test-dir/etc/version.lock.json
Created file with default content: test-dir/etc/packages.yaml
Created file with default content: test-dir/etc/stack-schema-map.yaml
Created file with default content: test-dir/etc/test_config.yaml
Created file with default content: test-dir/_config.yaml

# For details on how to configure the _config.yaml file,
# consult: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/etc/_config.yaml
# or the docs: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/docs/custom-rules.md

python -m detection_rules custom-rules setup-config -e test-dir-all

Output 

detection-rules on  3682-frdac-unit-test-locked-version-support-for-custom-config [!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules custom-rules setup-config -e test-dir-all

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Created directory: test-dir-all/actions
Created directory: test-dir-all/exceptions
Created directory: test-dir-all/rules
Created directory: test-dir-all/rules_building_block
Created directory: test-dir-all/etc
Created file with default content: test-dir-all/etc/deprecated_rules.json
Created file with default content: test-dir-all/etc/version.lock.json
Created file with default content: test-dir-all/etc/packages.yaml
Created file with default content: test-dir-all/etc/stack-schema-map.yaml
Created file with default content: test-dir-all/etc/test_config.yaml
Created file with default content: test-dir-all/_config.yaml

# For details on how to configure the _config.yaml file,
# consult: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/detection_rules/etc/_config.yaml
# or the docs: /home/forteea1/Code/clean_mains/additional_mains/detection-rules/docs/custom-rules.md

make test

Output 

❯ make test
Installing all dependencies...
./env/detection-rules-build/bin/pip install .[dev]
Looking in indexes: https://pypi.org/simple, https://eric.forte%40elastic.co:****@artifactory.elastic.dev/artifactory/api/pypi/pypi-endgame/simple
Processing /home/forteea1/Code/clean_mains/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /tmp/pip-install-fo8yst3g/detection-rules-kql_c2dcda17993c4919bfb0d9ac52295c0e
  Running command git clone --filter=blob:none --quiet https://github.com/elastic/detection-rules.git /tmp/pip-install-fo8yst3g/detection-rules-kql_c2dcda17993c4919bfb0d9ac52295c0e
  Resolved https://github.com/elastic/detection-rules.git to commit 79f575b33c747e0c3c5f7293c95f3ddab611e683
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kibana@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kibana (from detection_rules==0.1.0)
  Cloning https://github.com/elastic/detection-rules.git to /tmp/pip-install-fo8yst3g/detection-rules-kibana_18082b79500c483fb4a234c2e7af26bf
  Running command git clone --filter=blob:none --quiet https://github.com/elastic/detection-rules.git /tmp/pip-install-fo8yst3g/detection-rules-kibana_18082b79500c483fb4a234c2e7af26bf
  Resolved https://github.com/elastic/detection-rules.git to commit 79f575b33c747e0c3c5f7293c95f3ddab611e683
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: Click~=8.1.7 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (8.1.7)
Requirement already satisfied: elasticsearch~=8.12.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (8.12.1)
Requirement already satisfied: eql==0.9.19 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.9.19)
Requirement already satisfied: jsl==0.2.4 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.2.4)
Requirement already satisfied: jsonschema>=4.21.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (4.22.0)
Requirement already satisfied: marko==2.0.3 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (2.0.3)
Requirement already satisfied: marshmallow-dataclass~=8.6.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from marshmallow-dataclass[union]~=8.6.0->detection_rules==0.1.0) (8.6.1)
Requirement already satisfied: marshmallow-jsonschema~=0.13.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.13.0)
Requirement already satisfied: marshmallow-union~=0.1.15 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.1.15.post1)
Requirement already satisfied: marshmallow~=3.21.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (3.21.2)
Requirement already satisfied: pytoml==0.1.21 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.1.21)
Requirement already satisfied: PyYAML~=6.0.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (6.0.1)
Requirement already satisfied: requests~=2.31.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (2.31.0)
Requirement already satisfied: toml==0.10.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.10.2)
Requirement already satisfied: typing-inspect==0.9.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.9.0)
Requirement already satisfied: typing-extensions==4.10.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (4.10.0)
Requirement already satisfied: XlsxWriter~=3.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (3.2.0)
Requirement already satisfied: semver==3.0.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (3.0.2)
Requirement already satisfied: lark-parser~=0.12.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from eql==0.9.19->detection_rules==0.1.0) (0.12.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from typing-inspect==0.9.0->detection_rules==0.1.0) (1.0.0)
Requirement already satisfied: pep8-naming==0.13.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (0.13.0)
Requirement already satisfied: PyGithub==2.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (2.2.0)
Requirement already satisfied: flake8==7.0.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (7.0.0)
Requirement already satisfied: pyflakes==3.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (3.2.0)
Requirement already satisfied: pytest>=8.1.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (8.2.0)
Requirement already satisfied: pre-commit==3.6.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==0.1.0) (3.6.2)
Requirement already satisfied: mccabe<0.8.0,>=0.7.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from flake8==7.0.0->detection_rules==0.1.0) (0.7.0)
Requirement already satisfied: pycodestyle<2.12.0,>=2.11.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from flake8==7.0.0->detection_rules==0.1.0) (2.11.1)
Requirement already satisfied: cfgv>=2.0.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pre-commit==3.6.2->detection_rules==0.1.0) (3.4.0)
Requirement already satisfied: identify>=1.0.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pre-commit==3.6.2->detection_rules==0.1.0) (2.5.36)
Requirement already satisfied: nodeenv>=0.11.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from pre-commit==3.6.2->detection_rules==0.1.0) (1.8.0)
Requirement already satisfied: virtualenv>=20.10.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pre-commit==3.6.2->detection_rules==0.1.0) (20.26.1)
Requirement already satisfied: pynacl>=1.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==0.1.0) (1.5.0)
Requirement already satisfied: pyjwt>=2.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pyjwt[crypto]>=2.4.0->PyGithub==2.2.0->detection_rules==0.1.0) (2.8.0)
Requirement already satisfied: urllib3>=1.26.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==0.1.0) (2.2.1)
Requirement already satisfied: Deprecated in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==0.1.0) (1.2.14)
Requirement already satisfied: elastic-transport<9,>=8 in ./env/detection-rules-build/lib/python3.12/site-packages (from elasticsearch~=8.12.1->detection_rules==0.1.0) (8.13.0)
Requirement already satisfied: attrs>=22.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==0.1.0) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==0.1.0) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==0.1.0) (0.35.1)
Requirement already satisfied: rpds-py>=0.7.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==0.1.0) (0.18.1)
Requirement already satisfied: packaging>=17.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from marshmallow~=3.21.1->detection_rules==0.1.0) (24.0)
Requirement already satisfied: typeguard<4.0.0,>=2.4.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from marshmallow-dataclass[union]~=8.6.0->detection_rules==0.1.0) (3.0.2)
Requirement already satisfied: iniconfig in ./env/detection-rules-build/lib/python3.12/site-packages (from pytest>=8.1.1->detection_rules==0.1.0) (2.0.0)
Requirement already satisfied: pluggy<2.0,>=1.5 in ./env/detection-rules-build/lib/python3.12/site-packages (from pytest>=8.1.1->detection_rules==0.1.0) (1.5.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==0.1.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==0.1.0) (3.7)
Requirement already satisfied: certifi>=2017.4.17 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==0.1.0) (2024.2.2)
Requirement already satisfied: setuptools in ./env/detection-rules-build/lib/python3.12/site-packages (from nodeenv>=0.11.1->pre-commit==3.6.2->detection_rules==0.1.0) (69.5.1)
Requirement already satisfied: cryptography>=3.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pyjwt[crypto]>=2.4.0->PyGithub==2.2.0->detection_rules==0.1.0) (42.0.7)
Requirement already satisfied: cffi>=1.4.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from pynacl>=1.4.0->PyGithub==2.2.0->detection_rules==0.1.0) (1.16.0)
Requirement already satisfied: distlib<1,>=0.3.7 in ./env/detection-rules-build/lib/python3.12/site-packages (from virtualenv>=20.10.0->pre-commit==3.6.2->detection_rules==0.1.0) (0.3.8)
Requirement already satisfied: filelock<4,>=3.12.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from virtualenv>=20.10.0->pre-commit==3.6.2->detection_rules==0.1.0) (3.14.0)
Requirement already satisfied: platformdirs<5,>=3.9.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from virtualenv>=20.10.0->pre-commit==3.6.2->detection_rules==0.1.0) (4.2.1)
Requirement already satisfied: wrapt<2,>=1.10 in ./env/detection-rules-build/lib/python3.12/site-packages (from Deprecated->PyGithub==2.2.0->detection_rules==0.1.0) (1.16.0)
Requirement already satisfied: pycparser in ./env/detection-rules-build/lib/python3.12/site-packages (from cffi>=1.4.1->pynacl>=1.4.0->PyGithub==2.2.0->detection_rules==0.1.0) (2.22)
Building wheels for collected packages: detection_rules
  Building wheel for detection_rules (pyproject.toml) ... done
  Created wheel for detection_rules: filename=detection_rules-0.1.0-py3-none-any.whl size=39968505 sha256=50b05129788c01de14bbf8bd287c92ce7037d6ba2e2d748dfa8e78fd509ce5e4
  Stored in directory: /home/forteea1/.cache/pip/wheels/33/0b/6f/442542fc0e808e368b4dfaf768ed0b61a5d8281942974600d4
Successfully built detection_rules
Installing collected packages: detection_rules
  Attempting uninstall: detection_rules
    Found existing installation: detection_rules 0.1.0
    Uninstalling detection_rules-0.1.0:
      Successfully uninstalled detection_rules-0.1.0
Successfully installed detection_rules-0.1.0
LINTING
./env/detection-rules-build/bin/python -m flake8 tests detection_rules --ignore D203,N815 --max-line-length 120
./env/detection-rules-build/bin/python -m detection_rules test
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Tests skipped per config (3):
tests/kuery/test_lint.py::LintTests::test_upper_tokens
tests/test_gh_workflows.py::TestWorkflows::test_matrix_to_lock_version_defaults
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_has_nested_previous
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.12.3, pytest-8.2.0, pluggy-1.5.0 -- /home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/bin/python
cachedir: .pytest_cache
rootdir: /home/forteea1/Code/clean_mains/detection-rules
configfile: pyproject.toml
plugins: typeguard-3.0.2
collected 142 items                                                                                                                                                                          

tests/kuery/test_dsl.py::TestKQLtoDSL::test_and_query PASSED                                                                                                                           [  0%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_exists PASSED                                                                                                                        [  1%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_inequality PASSED                                                                                                                    [  2%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_field_match PASSED                                                                                                                         [  2%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_not_query PASSED                                                                                                                           [  3%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_optimizations PASSED                                                                                                                       [  4%]
tests/kuery/test_dsl.py::TestKQLtoDSL::test_or_query PASSED                                                                                                                            [  4%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_and_query PASSED                                                                                                                        [  5%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_boolean_precedence PASSED                                                                                                               [  6%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_equals PASSED                                                                                                                     [  7%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_field_inequality PASSED                                                                                                                 [  7%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_ip_checks PASSED                                                                                                                        [  8%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_list_of_values PASSED                                                                                                                   [  9%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_not_query PASSED                                                                                                                        [  9%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_or_query PASSED                                                                                                                         [ 10%]
tests/kuery/test_eql2kql.py::TestEql2Kql::test_wildcard_field PASSED                                                                                                                   [ 11%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_expr PASSED                                                                                                                    [ 11%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_and_values PASSED                                                                                                                  [ 12%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_cidr_match PASSED                                                                                                                  [ 13%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_field_exists PASSED                                                                                                                [ 14%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_flattening PASSED                                                                                                                  [ 14%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_list_value PASSED                                                                                                                  [ 15%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_not_value PASSED                                                                                                                   [ 16%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_expr PASSED                                                                                                                     [ 16%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_or_values PASSED                                                                                                                   [ 17%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_quoted_wildcard PASSED                                                                                                             [ 18%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_range PASSED                                                                                                                       [ 19%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_single_value PASSED                                                                                                                [ 19%]
tests/kuery/test_evaluator.py::EvaluatorTests::test_wildcard PASSED                                                                                                                    [ 20%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_and_query PASSED                                                                                                                        [ 21%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_boolean_precedence PASSED                                                                                                               [ 21%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_equals PASSED                                                                                                                     [ 22%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_field_inequality PASSED                                                                                                                 [ 23%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_list_of_values PASSED                                                                                                                   [ 23%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_lone_value PASSED                                                                                                                       [ 24%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_nested_query PASSED                                                                                                                     [ 25%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_not_query PASSED                                                                                                                        [ 26%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_or_query PASSED                                                                                                                         [ 26%]
tests/kuery/test_kql2eql.py::TestKql2Eql::test_schema PASSED                                                                                                                           [ 27%]
tests/kuery/test_lint.py::LintTests::test_and_not PASSED                                                                                                                               [ 28%]
tests/kuery/test_lint.py::LintTests::test_compound PASSED                                                                                                                              [ 28%]
tests/kuery/test_lint.py::LintTests::test_double_negate PASSED                                                                                                                         [ 29%]
tests/kuery/test_lint.py::LintTests::test_extract_not PASSED                                                                                                                           [ 30%]
tests/kuery/test_lint.py::LintTests::test_ip PASSED                                                                                                                                    [ 30%]
tests/kuery/test_lint.py::LintTests::test_lint_field PASSED                                                                                                                            [ 31%]
tests/kuery/test_lint.py::LintTests::test_lint_precedence PASSED                                                                                                                       [ 32%]
tests/kuery/test_lint.py::LintTests::test_merge_fields PASSED                                                                                                                          [ 33%]
tests/kuery/test_lint.py::LintTests::test_mixed_demorgans PASSED                                                                                                                       [ 33%]
tests/kuery/test_lint.py::LintTests::test_not_demorgans PASSED                                                                                                                         [ 34%]
tests/kuery/test_lint.py::LintTests::test_not_or PASSED                                                                                                                                [ 35%]
tests/kuery/test_parser.py::ParserTests::test_conversion PASSED                                                                                                                        [ 35%]
tests/kuery/test_parser.py::ParserTests::test_date PASSED                                                                                                                              [ 36%]
tests/kuery/test_parser.py::ParserTests::test_keyword PASSED                                                                                                                           [ 37%]
tests/kuery/test_parser.py::ParserTests::test_list_equals PASSED                                                                                                                       [ 38%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_fail PASSED                                                                                                               [ 38%]
tests/kuery/test_parser.py::ParserTests::test_multiple_types_success PASSED                                                                                                            [ 39%]
tests/kuery/test_parser.py::ParserTests::test_number_exists PASSED                                                                                                                     [ 40%]
tests/kuery/test_parser.py::ParserTests::test_number_wildcard_fail PASSED                                                                                                              [ 40%]
tests/kuery/test_parser.py::ParserTests::test_type_family_fail PASSED                                                                                                                  [ 41%]
tests/kuery/test_parser.py::ParserTests::test_type_family_success PASSED                                                                                                               [ 42%]
tests/test_all_rules.py::TestAlertSuppression::test_group_field_in_schemas PASSED                                                                                                      [ 42%]
tests/test_all_rules.py::TestBuildTimeFields::test_build_fields_min_stack PASSED                                                                                                       [ 43%]
tests/test_all_rules.py::TestIncompatibleFields::test_rule_backports_for_restricted_fields PASSED                                                                                      [ 44%]
tests/test_all_rules.py::TestIntegrationRules::test_all_min_stack_rules_have_comment PASSED                                                                                            [ 45%]
tests/test_all_rules.py::TestIntegrationRules::test_integration_guide SKIPPED (8.3+ Stacks Have Related Integrations Feature)                                                          [ 45%]
tests/test_all_rules.py::TestIntegrationRules::test_ml_integration_jobs_exist PASSED                                                                                                   [ 46%]
tests/test_all_rules.py::TestIntegrationRules::test_rule_demotions PASSED                                                                                                              [ 47%]
tests/test_all_rules.py::TestLicense::test_elastic_license_only_v2 PASSED                                                                                                              [ 47%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_if_plugins_explicitly_defined PASSED                                                                                            [ 48%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_note_has_osquery_warning PASSED                                                                                                 [ 49%]
tests/test_all_rules.py::TestNoteMarkdownPlugins::test_plugin_placeholders_match_entries PASSED                                                                                        [ 50%]
tests/test_all_rules.py::TestRiskScoreMismatch::test_rule_risk_score_severity_mismatch PASSED                                                                                          [ 50%]
tests/test_all_rules.py::TestRuleFiles::test_bbr_in_correct_dir PASSED                                                                                                                 [ 51%]
tests/test_all_rules.py::TestRuleFiles::test_non_bbr_in_correct_dir PASSED                                                                                                             [ 52%]
tests/test_all_rules.py::TestRuleFiles::test_rule_file_name_tactic PASSED                                                                                                              [ 52%]
tests/test_all_rules.py::TestRuleMetadata::test_deprecated_rules PASSED                                                                                                                [ 53%]
tests/test_all_rules.py::TestRuleMetadata::test_event_dataset PASSED                                                                                                                   [ 54%]
tests/test_all_rules.py::TestRuleMetadata::test_integration_tag PASSED                                                                                                                 [ 54%]
tests/test_all_rules.py::TestRuleMetadata::test_invalid_queries PASSED                                                                                                                 [ 55%]
tests/test_all_rules.py::TestRuleMetadata::test_updated_date_newer_than_creation PASSED                                                                                                [ 56%]
tests/test_all_rules.py::TestRuleTags::test_casing_and_spacing PASSED                                                                                                                  [ 57%]
tests/test_all_rules.py::TestRuleTags::test_investigation_guide_tag SKIPPED (Skipping until all Investigation Guides follow the proper format.)                                        [ 57%]
tests/test_all_rules.py::TestRuleTags::test_ml_rule_type_tags PASSED                                                                                                                   [ 58%]
tests/test_all_rules.py::TestRuleTags::test_no_duplicate_tags PASSED                                                                                                                   [ 59%]
tests/test_all_rules.py::TestRuleTags::test_os_tags PASSED                                                                                                                             [ 59%]
tests/test_all_rules.py::TestRuleTags::test_primary_tactic_as_tag PASSED                                                                                                               [ 60%]
tests/test_all_rules.py::TestRuleTags::test_required_tags PASSED                                                                                                                       [ 61%]
tests/test_all_rules.py::TestRuleTags::test_tag_prefix PASSED                                                                                                                          [ 61%]
tests/test_all_rules.py::TestRuleTimelines::test_timeline_has_title PASSED                                                                                                             [ 62%]
tests/test_all_rules.py::TestRuleTiming::test_eql_interval_to_maxspan PASSED                                                                                                           [ 63%]
tests/test_all_rules.py::TestRuleTiming::test_eql_lookback PASSED                                                                                                                      [ 64%]
tests/test_all_rules.py::TestRuleTiming::test_event_override PASSED                                                                                                                    [ 64%]
tests/test_all_rules.py::TestRuleTiming::test_required_lookback PASSED                                                                                                                 [ 65%]
tests/test_all_rules.py::TestThreatMappings::test_duplicated_tactics PASSED                                                                                                            [ 66%]
tests/test_all_rules.py::TestThreatMappings::test_tactic_to_technique_correlations PASSED                                                                                              [ 66%]
tests/test_all_rules.py::TestThreatMappings::test_technique_deprecations PASSED                                                                                                        [ 67%]
tests/test_all_rules.py::TestValidRules::test_all_rule_queries_optimized PASSED                                                                                                        [ 68%]
tests/test_all_rules.py::TestValidRules::test_bbr_validation PASSED                                                                                                                    [ 69%]
tests/test_all_rules.py::TestValidRules::test_duplicate_file_names PASSED                                                                                                              [ 69%]
tests/test_all_rules.py::TestValidRules::test_file_names PASSED                                                                                                                        [ 70%]
tests/test_all_rules.py::TestValidRules::test_max_signals_note PASSED                                                                                                                  [ 71%]
tests/test_all_rules.py::TestValidRules::test_production_rules_have_rta PASSED                                                                                                         [ 71%]
tests/test_all_rules.py::TestValidRules::test_rule_type_changes PASSED                                                                                                                 [ 72%]
tests/test_all_rules.py::TestValidRules::test_schema_and_dupes PASSED                                                                                                                  [ 73%]
tests/test_hunt_data.py::TestHunt::test_load_toml_files PASSED                                                                                                                         [ 73%]
tests/test_hunt_data.py::TestHunt::test_markdown_existence PASSED                                                                                                                      [ 74%]
tests/test_hunt_data.py::TestHunt::test_toml_loading PASSED                                                                                                                            [ 75%]
tests/test_mappings.py::TestMappings::test_false_positives PASSED                                                                                                                      [ 76%]
tests/test_mappings.py::TestMappings::test_true_positives PASSED                                                                                                                       [ 76%]
tests/test_mappings.py::TestRTAs::test_rtas_with_triggered_rules_have_uuid PASSED                                                                                                      [ 77%]
tests/test_packages.py::TestPackages::test_package_loader_default_configs PASSED                                                                                                       [ 78%]
tests/test_packages.py::TestPackages::test_package_loader_production_config PASSED                                                                                                     [ 78%]
tests/test_packages.py::TestPackages::test_package_summary PASSED                                                                                                                      [ 79%]
tests/test_packages.py::TestPackages::test_rule_versioning PASSED                                                                                                                      [ 80%]
tests/test_packages.py::TestRegistryPackage::test_registry_package_config PASSED                                                                                                       [ 80%]
tests/test_python_library.py::TestEQLInSet::test_eql_in_set PASSED                                                                                                                     [ 81%]
tests/test_schemas.py::TestSchemas::test_eql_validation PASSED                                                                                                                         [ 82%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_7_x PASSED                                                                                                                    [ 83%]
tests/test_schemas.py::TestSchemas::test_query_downgrade_8_x PASSED                                                                                                                    [ 83%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_7_x PASSED                                                                                                                [ 84%]
tests/test_schemas.py::TestSchemas::test_threshold_downgrade_8_x PASSED                                                                                                                [ 85%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_7_x PASSED                                                                                                                [ 85%]
tests/test_schemas.py::TestSchemas::test_versioned_downgrade_8_x PASSED                                                                                                                [ 86%]
tests/test_schemas.py::TestVersionLockSchema::test_version_lock_no_previous PASSED                                                                                                     [ 87%]
tests/test_schemas.py::TestVersions::test_stack_schema_map PASSED                                                                                                                      [ 88%]
tests/test_specific_rules.py::TestESQLRules::test_esql_queries PASSED                                                                                                                  [ 88%]
tests/test_specific_rules.py::TestEndpointQuery::test_os_and_platform_in_query PASSED                                                                                                  [ 89%]
tests/test_specific_rules.py::TestNewTerms::test_history_window_start PASSED                                                                                                           [ 90%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_field_exists PASSED                                                                                                         [ 90%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_fields PASSED                                                                                                               [ 91%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_fields_unique PASSED                                                                                                        [ 92%]
tests/test_specific_rules.py::TestNewTerms::test_new_terms_max_limit PASSED                                                                                                            [ 92%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_deep PASSED                                                                                                        [ 93%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_formatter_rule PASSED                                                                                                        [ 94%]
tests/test_toml_formatter.py::TestRuleTomlFormatter::test_normalization PASSED                                                                                                         [ 95%]
tests/test_transform_fields.py::TestGuideMarkdownPlugins::test_plugin_conversion PASSED                                                                                                [ 95%]
tests/test_transform_fields.py::TestGuideMarkdownPlugins::test_transform_guide_markdown_plugins PASSED                                                                                 [ 96%]
tests/test_utils.py::TestTimeUtils::test_caching PASSED                                                                                                                                [ 97%]
tests/test_utils.py::TestTimeUtils::test_event_class_normalization PASSED                                                                                                              [ 97%]
tests/test_utils.py::TestTimeUtils::test_schema_multifields PASSED                                                                                                                     [ 98%]
tests/test_utils.py::TestTimeUtils::test_time_normalize PASSED                                                                                                                         [ 99%]
tests/test_version_locking.py::TestVersionLock::test_previous_entries_gte_current_min_stack PASSED                                                                                     [100%]

====================================================================================== warnings summary ======================================================================================
env/detection-rules-build/lib/python3.12/site-packages/_pytest/config/__init__.py:1285
  /home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/_pytest/config/__init__.py:1285: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: typeguard
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================= 140 passed, 2 skipped, 1 warning in 5.77s ==========================================================================

@eric-forte-elastic eric-forte-elastic added enhancement New feature or request python Internal python for the repository Area: DED Team: TRADE detections-as-code labels May 16, 2024
@eric-forte-elastic eric-forte-elastic requested a review from traut May 16, 2024 00:45
@eric-forte-elastic eric-forte-elastic self-assigned this May 16, 2024
@eric-forte-elastic eric-forte-elastic linked an issue May 16, 2024 that may be closed by this pull request
[FR][DAC] Unit Test Locked Version Support for Custom Config #3682
Closed
@botelastic botelastic bot added the schema label May 16, 2024
@eric-forte-elastic eric-forte-elastic changed the base branch from main to DAC-feature May 16, 2024 00:45
Mikaayenson
Mikaayenson approved these changes May 16, 2024
View reviewed changes
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Can you double check that we don't want to add any other tests?

For example:

#  - tests.test_all_rules.TestValidRules.test_schema_and_dupes
#  - tests.test_packages.TestRegistryPackage.test_registry_package_config

I don't recall exactly why those stood out, but I think they also fail with just the base setup-config run.

detection_rules/custom_rules.py
Comment on lines +55 to +56
"# To run all tests, set bypass to empty or leave this file commented out.",
"",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this just be a \n ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the return '\n'.join(lines), the "" is the equivalent to a a \n. This was just to clean up the text to potentially make it easier to add or subtract lines later.

@eric-forte-elastic
Copy link
Contributor Author

LGTM. Can you double check that we don't want to add any other tests?

For example:

#  - tests.test_all_rules.TestValidRules.test_schema_and_dupes
#  - tests.test_packages.TestRegistryPackage.test_registry_package_config

I don't recall exactly why those stood out, but I think they also fail with just the base setup-config run.

Good catch, we should add some other tests to bypass, moving back to in progress to add them now 👍

@eric-forte-elastic eric-forte-elastic mentioned this pull request May 16, 2024
Merged
Mikaayenson
Mikaayenson approved these changes May 16, 2024
View reviewed changes
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to update the summary.

@eric-forte-elastic
Copy link
Contributor Author

LGTM. Can you double check that we don't want to add any other tests?
For example:

#  - tests.test_all_rules.TestValidRules.test_schema_and_dupes
#  - tests.test_packages.TestRegistryPackage.test_registry_package_config

I don't recall exactly why those stood out, but I think they also fail with just the base setup-config run.

Good catch, we should add some other tests to bypass, moving back to in progress to add them now 👍

Done

@eric-forte-elastic eric-forte-elastic merged commit 944fe30 into DAC-feature May 16, 2024
15 checks passed
@eric-forte-elastic eric-forte-elastic deleted the 3682-frdac-unit-test-locked-version-support-for-custom-config branch May 16, 2024 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@traut traut traut approved these changes

@Mikaayenson Mikaayenson Mikaayenson approved these changes

@brokensound77 brokensound77 Awaiting requested review from brokensound77

Assignees

@eric-forte-elastic eric-forte-elastic

Labels
backport: auto detections-as-code enhancement New feature or request python Internal python for the repository schema Team: TRADE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FR][DAC] Unit Test Locked Version Support for Custom Config
3 participants
@eric-forte-elastic @traut @Mikaayenson