[Bug] Using the CLI to export esql (ES|QL) rules from Kibana results in ValidationError if using metadata according to documentation

[frederikb96]

Describe the Bug

The CLI python -m detection_rules kibana export-rules doesnt work with a simple esql rule, where metadata is set according to official documentation. It always leads to:

marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'_schema': ["Rule: test_fberg_esql contains a non-aggregate query without metadata fields '_id', '_version', and '_index' -> Add 'metadata _id, _version, _index' to the from command or add an aggregate function."]}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}

To Reproduce

1. Create a simple esql rule:

2. Try to export it with the CLI and kibana export-rules
3. Leads to ValidationError

Expected Behavior

No ValidationError since esql metadata is set according to documentation.

Edit: I fixed this via PR where we validate the order and allow any order of metadata

Screenshots

No response

Desktop - OS

None

Desktop - Version

No response

Additional Context

No response

[frederikb96]

Can be closed via #4579 once its merged

[eric-forte-elastic]

Just adding for context for reviewers on the ^ PR. This issue can be more easily tested via DaC commands (loading just a specific rule to the rule loader), but the fundamental issue is with the ES|QL validation for rules passing schema validation rather than any DaC command.

Another testing example illustrating issue: