Move Rule into a dataclass

[rw-access]

Issues

No issues, just trying to clean some things up

Summary

I moved rules and the schemas into a dataclass. I also changed the way hashes are built, so that you never modify the original rule file. I commented out some tests that are no longer relevant.

Also bumped to Python 3.8 so I could use typing.Literal and a few other additions.

Since this changes a lot of stuff under the hood, I'm hoping that we can get it merged sooner. I'm okay with temporarily suppressing testing if it means that we can avoid broken builds.

There's still a good bit to go.

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[brokensound77]

This is great, thanks for starting this!

As we talked about the other day, we should jump on a call and discuss this to see what options make the most sense and make sure we are aligned.

I def like that we are doing this in stages and think disabling some tests is fine, but I don't think we should merge it to main in a broken state. There is a lot that still needs to be verified/tested concerning packaging, the CLI, and the remaining tests. I think we should most likely merge this to an intermediate base branch so that we can keep to the scope of the intermediate PRs.

[brokensound77]

some other changes for rule_loader that seem in scope and make sense to update with this

 import functools
-import glob
-import io
 import os
 import re
 from collections import OrderedDict
@@ -47,7 +45,7 @@ def reset():
 
 
 @cached
-def load_rule_files(verbose=True, paths=None):
+def load_rule_files(verbose=True, paths=None) -> Dict[Path, dict]:
     """Load the rule YAML files, but without parsing the EQL query portion."""
     file_lookup = {}  # type: dict[str, dict]
 
@@ -55,15 +53,14 @@ def load_rule_files(verbose=True, paths=None):
         print("Loading rules from {}".format(RULES_DIR))
 
     if paths is None:
-        paths = sorted(glob.glob(os.path.join(RULES_DIR, '**', '*.toml'), recursive=True))
+        paths: List[Path] = sorted(Path(get_path(RULES_DIR)).rglob('*.toml'))
 
     for rule_file in paths:
         try:
             # use pytoml instead of toml because of annoying bugs
             # https://github.com/uiri/toml/issues/152
             # might also be worth looking at https://github.com/sdispater/tomlkit
-            with io.open(rule_file, "r", encoding="utf-8") as f:
-                file_lookup[rule_file] = pytoml.load(f)
+            file_lookup[rule_file] = pytoml.loads(rule_file.read_text())
         except Exception:
             print(u"Error loading {}".format(rule_file))
             raise
@@ -118,7 +115,10 @@ def load_rules(file_lookup=None, verbose=True, error=True):
 
         except Exception as e:
             failed = True
-            err_msg = "Invalid rule file in {}\n{}".format(rule_file, click.style(str(e), fg='red'))
+            relative_rule_path = rule_file.relative_to(RULES_DIR)
+            rule_id = rule_contents.get('rule', {}).get('rule_id', '')
+            colored_error = click.style(str(e), fg='red')
+            err_msg = f"Invalid rule file in: {relative_rule_path} - {rule_id}\n{colored_error}"
             errors.append(err_msg)
             if error:
                 if verbose:

[rw-access]

those are good changes, and the loader is next.
i'll punt until a follow up PR

[brokensound77]

:soclose:

I added comments on changes from new commits

[brokensound77]

from marshmallow import Schema

Suggested change

	def __schema(cls):
	def __schema(cls: ClassT) -> Type[Schema][ClassT]:

[rw-access]

40d9e58.
i changed it slightly, because i return the instantiated class

[brokensound77]

I originally was going to suggest that, but it has this error: Class 'SchemaMeta' does not define '__getitem__', so the '[]' operator cannot be used on its instances. It doesn't look like my suggestion is correctly hinting either, but it is an instantiated type

[threat-punter]

I saw this in the tests. Just making sure it's on your radar.

================================================================================================================================= warnings summary ==================================================================================================================================
env/detection-rules-build/lib/python3.8/site-packages/marshmallow/fields.py:195: 35 warnings
tests/test_packages.py: 3 warnings
  /Users/threatpunter/Documents/GitHub/detection-rules/env/detection-rules-build/lib/python3.8/site-packages/marshmallow/fields.py:195: RemovedInMarshmallow4Warning: Passing field metadata as a keyword arg is deprecated. Use the explicit `metadata=...` argument instead.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================================================= 93 passed, 38 warnings in 29.71s ==========================================================================================================================
(.venv) $ 

[rw-access]

@threat-punter looks like that'll be fixed when lovasoa/marshmallow_dataclass#119 is updated and we pull in the dependencies.

[brokensound77]

1. pull in updates from main and make release
2. from main, make release

If those two package hashes match, I think that means we are good and I ✔️

Also, left a few other final comments as well

Thanks for enduring through this 🎈

[brokensound77]

you can make this test work again by loading an actual rule vs a test rule

[brokensound77]

from detection_rules import rule_loader
from detection_rules.rule import EQLRuleData

Suggested change

	#
	# def test_format_of_all_rules(self):
	# """Test all rules."""
	# rules = rule_loader.load_rules().values()
	#
	# for rule in rules:
	# is_eql_rule = isinstance(rule.contents.data, EQLRuleData)
	# self.compare_formatted(
	# rule.rule_format(formatted_query=False), callback=nested_normalize, kwargs={'eql_rule': is_eql_rule})
	def test_format_of_all_rules(self):
	"""Test all rules."""
	rules = rule_loader.load_rules().values()
	for rule in rules:
	is_eql_rule = isinstance(rule.contents.data, EQLRuleData)
	self.compare_formatted(
	rule.contents.to_dict(), callback=nested_normalize, kwargs={'eql_rule': is_eql_rule})

[rw-access]

i changed this a different way, so i think we're good now 🤞

[rw-access]

Package hash unchanged and no version bumps detected from master.
Current package hash was 826f795f49ef0731a15ddd945d5977adc789ecb3926e655578fbd0d570a19561