Skip to content

[Bug] Collect Events Outdated Sort Argument #3530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eric-forte-elastic opened this issue Mar 22, 2024 · 0 comments · Fixed by #3531
Closed

[Bug] Collect Events Outdated Sort Argument #3530

eric-forte-elastic opened this issue Mar 22, 2024 · 0 comments · Fixed by #3531
Assignees
Labels
bug Something isn't working python Internal python for the repository

Comments

@eric-forte-elastic
Copy link
Contributor

Describe the bug
When running collect events to capture data, the current search call specified the sort order via field:order which is not supported.
Current:

self.search(dsl, language='dsl', index=indexes, start_time=start_time, end_time='now', size=5000,
                              sort='@timestamp:asc')

Required:

results = self.search(dsl, language='dsl', index=indexes, start_time=start_time, end_time='now', size=5000,
                              sort=[{'@timestamp': {'order': 'asc'}}])

To Reproduce
Steps to reproduce the behavior:

  1. Use the collect-events command to attempt to collect data
endpoint-rules on  2356-fr-add-host-family-to-data-path [?] via  v3.8.10 (venv) on  eric.forte took 20s 
❯ python -m endpoint_rules es --cloud-id <cloud_id> collect-events --rta-name bitsadmin_execution <host_id>

███████╗███╗   ██╗██████╗ ██████╗  ██████╗ ██╗███╗   ██╗████████╗
██╔════╝████╗  ██║██╔══██╗██╔══██╗██╔═══██╗██║████╗  ██║╚══██╔══╝
█████╗  ██╔██╗ ██║██║  ██║██████╔╝██║   ██║██║██╔██╗ ██║   ██║
██╔══╝  ██║╚██╗██║██║  ██║██╔═══╝ ██║   ██║██║██║╚██╗██║   ██║
███████╗██║ ╚████║██████╔╝██║     ╚██████╔╝██║██║ ╚████║   ██║
╚══════╝╚═╝  ╚═══╝╚═════╝ ╚═╝      ╚═════╝ ╚═╝╚═╝  ╚═══╝   ╚═╝
        ██████╗ ██╗   ██╗██╗     ███████╗███████╗
        ██╔══██╗██║   ██║██║     ██╔════╝██╔════╝
        ██████╔╝██║   ██║██║     █████╗  ███████╗
        ██╔══██╗██║   ██║██║     ██╔══╝  ╚════██║
        ██║  ██║╚██████╔╝███████╗███████╗███████║
        ╚═╝  ╚═╝ ╚═════╝ ╚══════╝╚══════╝╚══════╝

es_user: eric.forte
es_password: 
Press any key once detonation is complete ...
Press any key once detonation is complete ...
CLI Error (AssertionError): No events collected! Verify events are streaming and that the agent-hostname is correct

Expected behavior
When running collect-events this is an example of correct output (where the events are collected)

endpoint-rules on  2356-fr-add-host-family-to-data-path [?] via  v3.8.10 (venv) on  eric.forte took 20s 
❯ python -m endpoint_rules es --cloud-id <cloud_id> collect-events --rta-name bitsadmin_execution <host_id>

███████╗███╗   ██╗██████╗ ██████╗  ██████╗ ██╗███╗   ██╗████████╗
██╔════╝████╗  ██║██╔══██╗██╔══██╗██╔═══██╗██║████╗  ██║╚══██╔══╝
█████╗  ██╔██╗ ██║██║  ██║██████╔╝██║   ██║██║██╔██╗ ██║   ██║
██╔══╝  ██║╚██╗██║██║  ██║██╔═══╝ ██║   ██║██║██║╚██╗██║   ██║
███████╗██║ ╚████║██████╔╝██║     ╚██████╔╝██║██║ ╚████║   ██║
╚══════╝╚═╝  ╚═══╝╚═════╝ ╚═╝      ╚═════╝ ╚═╝╚═╝  ╚═══╝   ╚═╝
        ██████╗ ██╗   ██╗██╗     ███████╗███████╗
        ██╔══██╗██║   ██║██║     ██╔════╝██╔════╝
        ██████╔╝██║   ██║██║     █████╗  ███████╗
        ██╔══██╗██║   ██║██║     ██╔══╝  ╚════██║
        ██║  ██║╚██████╔╝███████╗███████╗███████║
        ╚═╝  ╚═╝ ╚═════╝ ╚══════╝╚══════╝╚══════╝

es_user: eric.forte
es_password: 
Press any key once detonation is complete ...
149 events saved to: /tmp/init_tmp/endpoint-rules/unit_tests/data/true_positives/bitsadmin_execution/windows/endpoint.ndjson
32 events saved to: /tmp/init_tmp/endpoint-rules/unit_tests/data/true_positives/bitsadmin_execution/windows/metricbeat.ndjson
2 events saved to: /tmp/init_tmp/endpoint-rules/unit_tests/data/true_positives/bitsadmin_execution/windows/filebeat.ndjson

Additional context
This is the result of a bug that was present in the 8.11 elasticsearch library that was patched in 8.12, which enforces the correct order parameter instead of the prior behavior. In 8.11, one could make the call as currently exists in detection-rules; however, the library would strip the faulty asc string parameter and replace it with the default ascending sort order when performing the query. Since the result is the same, the bug was unnoticed.

@eric-forte-elastic eric-forte-elastic added bug Something isn't working python Internal python for the repository Area: DED labels Mar 22, 2024
@eric-forte-elastic eric-forte-elastic self-assigned this Mar 22, 2024
@eric-forte-elastic eric-forte-elastic linked a pull request Mar 22, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Internal python for the repository
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant