[Bug] EQL Sequence Multi-Data Source Schema Validation

[terrancedejesus]

Describe the Bug

Summary

There appears to be a bug in our EQL validation for sequences where each node has a separate data source.
Reference rule: https://github.com/elastic/detection-rules/pull/4688/files#diff-3c9bd16592d9ff2afc43398d439ca6f1f645e37e05c4811f49e1fbda3cbd8739

This is a blocker to this PR, but in a big picture, potentially multi-domain based queries and sequences.

To Reproduce

1. Checkout the branch azure-oauth-phish
2. Run python -m detection_rules view-rule detection-rules/rules/integrations/azure/persistence_identity_protect_alert_followed_by_device_reg.toml
3. Notice we have an error ValueError: Error in both stack and integrations checks. This means that a field failed validation in both ECS, Non-ECS, Beats and Integration schemas.
4. Quick bug verification - Replace the queries with either or below and you will notice it passes, so the fields are valid.

sequence with maxspan=30m
[any where event.dataset == "azure.identity_protection"] by azure.identityprotection.properties.user_principal_name
[any where event.dataset == "azure.identity_protection"] by azure.identityprotection.properties.user_principal_name

or

sequence with maxspan=30m
[any where event.dataset == "azure.auditlogs" and event.action == "Register device"] by azure.auditlogs.properties.initiated_by.user.userPrincipalName
[any where event.dataset == "azure.auditlogs" and event.action == "Register device"] by azure.auditlogs.properties.initiated_by.user.userPrincipalName

Starting breakpoints - Here are some breakpoints to start at for the validation in rule_validations.py...

• L351 - Validation checks for the stack
• L353 - If clause for integration validation
• L411 - Validate query with schema (EQL, Beats, Min-stack, etc. params)
• L544 - Error handling for unknown fields
• L469 - Casts integration schema (azure) from KQL to EQL???
• L476 - Validates query with integration schema (EQL Schema) --> Note here you can check if azure.identityprotection.properties.user_principal_name is in eql_schema which returns true
• L485 - Integration validation error handling if unknown field

Expected Behavior

When running view-rule it should validate and output a JSON object (Kibana format of the rule). It should also have the related_integrations for Azure package versions.

Screenshots

No response

Desktop - OS

None

Desktop - Version

macOS - Sequoia 15.3.2

Additional Context

• I also updated the integration schemas and manifest but the field should exist prior to.

python -m detection_rules dev integrations build-schemas -i azure   

python -m detection_rules dev integrations build-manifests -o -i azure

No response