[otel-data] Hide 10m and 60m aggregated metrics generated for the APM UI

[gregkalapos]

Solves #113551

Aligning otel-data with apm-data to hide 10m and 60m aggregated metrics generated by collector components.

Tested with the e2e setup as well:

[gregkalapos]

Originally I wanted to define constant_keyword under resource.attributes for metricset.interval and under attributes for metricset.name, since that's where these are stored. But that'd again run into the usual issue of wiping out existing mapping for resource.attributes.

But as it seems, this works as well - this is also covered by tests.

Just calling out to make sure there is no unwanted implication I may missed.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gregkalapos]

Should we do something with these? We repeat them everywhere.

If I remember correctly, the idea was to implement these here in the template, so people can't overwrite these.

First idea would be to move them to [email protected], but that won't cover what we want.

[felixbarny]

Yes, I think it would be better to have a separate component template for these.

If I remember correctly, the idea was to implement these here in the template, so people can't overwrite these.

I think it was the opposite. Users should have the ability to customize the dynamic mappings. Therefore, when extracting this to a component template, we need to import that template after the @Custom component template.

[gregkalapos]

Yes, so what I mean: users should be able to change dynamic templates currently defined in [email protected], but not the ones above (ecs_ip and all_strings_to_keywords), because changing or removing those could easily break the data stream since it effects dimensions. Right?

So I'd move these two dynamic templates into a separate components (maybe metrics-otel-fixed@mappings) and metrics templates would be something like:

composed_of:
  - metrics@tsdb-settings
  - otel@mappings
  - metrics-otel@mappings
  - semconv-resource-to-ecs@mappings
  - metrics@custom
  - metrics-otel@custom
  - metrics-otel-fixed@mappings #<- the new one

(plus adding metrics-[x]m.otel@custom according to #114042 (comment))

Does that sound ok?

Happy to take suggestion on the naming metrics-otel-fixed@mappings :)

[gregkalapos]

Strange failing tests... so far what I see:

distribution/bwc/minor/build/bwc/checkout-8.x/x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/services/openai/OpenAiService.java:318: error: cannot find symbol
 [8.16.0] > Task :x-pack:plugin:vector-tile:compileJava
 [8.16.0] > Task :x-pack:plugin:vector-tile:classes
 [8.16.0] > Task :x-pack:plugin:vector-tile:jar
 [8.16.0]     public Set<TaskType> supportedStreamingTasks() {
 [8.16.0]            ^
 [8.16.0]   symbol:   class Set
 [8.16.0]   location: class OpenAiService
 [8.16.0] 
 [8.16.0] > Task :x-pack:plugin:inference:compileJava

Doesn't seem to be related to this PR.

Update: yeah, other PRs fail as well on the same step. I'll rebase once this is fixed.

[gregkalapos]

@elasticmachine update branch

[gregkalapos]

@elasticmachine update branch

[felixbarny]

It's the reverse. Users should be able to customize the dynamic mappings. That's why this component template needs to be imported last. This is because for dynamic templates, the first one that matches wins. Therefore, users need to be able to put a dynamic template before this in order to be able to customize how attributes are mapped dynamically. We have a test for this scenario:

elasticsearch/x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_metrics_tests.yml

Lines 112 to 148 in 3b9150d

	"Custom dynamic template":
	- do:
	cluster.put_component_template:
	name: metrics-otel@custom
	body:
	template:
	settings:
	index:
	routing_path: [unit, attributes.*, resource.attributes.*]
	mode: time_series
	time_series:
	start_time: 2024-07-01T13:03:08.138Z
	mappings:
	dynamic_templates:
	- no_ip_fields:
	mapping:
	type: keyword
	match_mapping_type: string
	- do:
	bulk:
	index: metrics-generic.otel-default
	refresh: true
	body:
	- create: {}
	- '{"@timestamp":"2024-07-18T14:48:33.467654000Z","data_stream":{"dataset":"generic.otel","namespace":"default"},"attributes":{"host.ip":"127.0.0.1","foo":"bar"}}'
	- is_false: errors
	- do:
	indices.get_data_stream:
	name: metrics-generic.otel-default
	- set: { data_streams.0.indices.0.index_name: idx0name }
	- do:
	indices.get_mapping:
	index: $idx0name
	expand_wildcards: hidden
	- match: { .$idx0name.mappings.properties.attributes.properties.host\.ip.type: 'keyword' }
	- match: { .$idx0name.mappings.properties.attributes.properties.foo.type: "keyword" }

So I think all is correctly implemented, but we'll need to change the description, and probably the name of the dynamic template. Suggestion: ecs-tsdb@mappings (since the ip path match is coming from ECS).

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x