[FR] git synchronize

[SHolzhauer]

Is your feature request related to a problem? Please describe.
Within organizations there is a desire to "version control" the rules/code in use.
SOC teams want to organize and make documented, incremental changes to their custom
detection-rules and have a "back-up" of their code in case of data loss.

This is an extension of #17 as discussed in the issue.

Describe the solution you'd like
We need a way to

• Update existing, custom, rules in Kibana,
• Update existing, elastic, rules in Kibana between stack releases,
• Create new, custom, rules in Kibana
• Create new, elastic, rules in Kibana between stack releases
• Do above with multiple rules at once

This process should

• be possible to automate (CI/CD)
• validate rules (python -m detection_rules test functionality ?)
• only update/create valid rules

This FR should not

• interfere with/change existing commands

Possible command could be: python -m detection_rules kibana git-sync

python -m detection_rules kibana

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Usage: detection_rules kibana [OPTIONS] COMMAND [ARGS]...

  Commands for integrating with Kibana.

Options:
  -k, --kibana-url TEXT
  --cloud-id TEXT
  -u, --kibana-user TEXT
  -p, --kibana-password TEXT
  --space TEXT
  -h, --help                  Show this message and exit.

Commands:
  upload-rule  Upload a list of rule .toml files to Kibana.
  git-sync     Sync the rules defined in Kibana with the rules in this repository

Additional context
#17

[SHolzhauer]

I think this could use the

1. find api To list the current existing rules in Kibana
2. bulk_create api to create new rules
3. bulk_update api to update existing rules and deactivate rules no longer present in the repository.

[rw-access]

Hey @SHolzhauer I've been talking with @brokensound77 and had some discussions in slack and I think we have an option to sync between this repo to and from Kibana. Here's my proposal for the high level workflow and design:

• Add a kibana push command. Point this command to a folder and it will update or create all rules in Kibana to match. I'm thinking that we can add a --dry-run flag so you can get a preview of all the changes
• Add a kibana pull command. Point this to a folder and it will retrieve all the Kibana rules and create new files or update existing. This would also have --dry-run
• Add a kibana prune command. This would remove rules that aren't in the other location. prune --local will remove lcoal copies that aren't in Kibana. --remote does the opposite. --dry-run makes sense here too
• Elastic provided rules will be out of scope. There are other efforts I'm involved with to make rules provided by Elastic more natively updated in Kibana. For the above commands, we can identify these rules and make them hard or soft errors.
• We won't provide a mechanism for conflict resolution if a rule is updated in two places. You might have to do that yourself with git branches. Conflicts will be hard errors.
• I think we can do a kibana sync [--dry-run] to do a two way sync. The commands will automatically store a JSON file at the root of the specified folder to track the local version of a rule, and what the version of the rule is in Kibana when it was last synced. We can autosync rules that aren't in conflict.

How's that plan sound to you?

[SHolzhauer]

@rw-access LGTM

Couple of notes:

• The main use-case for this functionality is CI-CD for (custom) detection rules. Storing a json file at the root could produce issues when the files are "touched" in version control. This is a general thing with version controlled projects ofcourse but maybe something to think about?
• I don't object to excluding OOTB rules, however I would say let's create the functionality in such a way this behaviour can be overridden. It's up to the user how to use it. It also reduces (i suspect) the complexity of the code.
• prune command, what would happen and/or should it be supported to use both flags? --local and --remote. This could be a cleaning step for example.

On the --dry-run flag. Should it maybe be moved up 1 level? So the kibana command gets it. Just a thought but as it will be shared across multiple commands might be a better place for it.

[rw-access]

Great point about the state file. CI/CD solutions generally don't modify files in version control so that's problematic. I think a Kibana saved object might be a more appropriate place to store that state, since it's persistent and outside of VCS.

For Elastic provided rules, we don't have many options because the rules themselves are actually immutable. So we can't change them via API even if we wanted to. We're also working on a more native upgrade mechanism in Kibana, hopefully as simple as an "upgrade rules" button that can happen at any time, without needing a stack update. You can always layer exceptions on top of the rules when you want deviations in the logic but want to remain subscribed to the rule.

[SHolzhauer]

Thinkin about it, wouldn't you be able to use the updated_date field? Shouldn't that one be comparable to the date in kibana?

[rw-access]

Hey, @SHolzhauer haven't updated you for a while.
This is still on my radar.

We are planning some refactors to the repository internals, and I think that we'll unlock some nice new functionality to make the rules/ directory completely configurable. So you can have your rules live outside the repo, or in a fork or whatever you decide. So then you just point the repository to your configuration file and it'll find everything, can run tests on that, etc.

I think our refactor will span a month or so, and hopefully this becomes a natural solution that emerges easily from there. 🤞. I think this is a really cool use case, and something that I think will be a huge value add for organizations that are very CI/CD leaning for rules management and "detections as code" thinking.

[SHolzhauer]

@rw-access thank you for the update, looking forward to it!

[SHolzhauer]

Hey @rw-access, I was curious if you had any update regarding this issue?

We are starting to notice more and more bugs with our current work-around/implementation and can really use this feature 😃

[rw-access]

hey @SHolzhauer, yeah here's some updates for you and your team:

• we've been refactoring things internally (PR Move Rule into a dataclass #1029, PR Add a RuleCollection object instead of a "loader" module #1063, PR Port historical schemas to jsonschema #1084) to enable
• there's an effort i'm leading that will let us "push" Elastic prebuilt rules to Kibana via a Fleet Integration (PR [Security] Adds pre-packaged rule updates through the "Prebuilt Security Detection Rules" Fleet integration kibana#96698) so that you can update your prebuilt rules between stack releases
• we're working on a new branching strategy that's more cherry-pick based instead of Kibana. this will make it so main, 7.13, 7.14, ... branches can all have the same tooling and validation in place, just different rules

given all of that, my hope is that you can specify where your rules/ directory lives with an environment variable. like DETECTION_RULES_DIR=../sholzhauer-rules. and then everything in the repo would use that directory instead of our rules/ directory.

within that directory, you'd put all of the configuration. it would have version.lock.json, config.toml, and whatever subfolders you want for your rules. in the config file, you could specify what stack version you want to validate your rules against. that way we can make sure that you're not trying to use rules that require 7.14 features but you only have a 7.13 stack (for example).

and if you want, you could put that directory inside your own git repo or VCS it however you like. then you don't have to worry about forking this repo and dealing with merge conflicts.

[SHolzhauer]

Sounds promising, looking forward.

And thank you for the blazing fast response!

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.

[jonny-mcc]

Just wanted to check if this functionality is still planned?

[Mikaayenson]

Update November 30

Based on recent discussions, as we plan to move forward, we will want to support custom rules only (not prebuilt rules for now).

[eric-forte-elastic]

This is now supported for custom rules via #3889. Please feel free to re-open this issue or create a new one if you run into any trouble. Thanks!