Make xpack.otel_data.registry.enabled default to true

[gregkalapos]

Solves #113368

Enabling otel-data by default.

Testing

I removed setting("xpack.otel_data.registry.enabled", "true") from tests, so naturally all test will assume default config and test that the plugin is turned on.

[gregkalapos]

@elasticmachine update branch

[gregkalapos]

@elasticmachine update branch

[gregkalapos]

This fails with:

---
No features should be used just by starting up with default configuration:
- do:
    license.get_feature_usage: {}
    headers:
      Content-Type: "application/vnd.elasticsearch+json;compatible-with=8"
      Accept: "application/vnd.elasticsearch+json;compatible-with=8"
- length:
    features: 0

[2024-09-27T21:43:42,054][INFO ][o.e.l.XPackCoreClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "request_headers" : {
      "Accept" : "application/vnd.elasticsearch+json;compatible-with=8",
      "Content-Type" : "application/vnd.elasticsearch+json;compatible-with=8"
    },
    "body" : {
      "features" : [
        {
          "family" : "mappings",
          "name" : "synthetic-source",
          "context" : null,
          "last_used" : "2024-09-27T16:43:40.454Z",
          "license_level" : "enterprise"
        }
      ]
    }
  }
}]

That makes me think, synthetic-source requires some license, but since we use that by default now (otel-data would be on by default and that would need synthetic source), this test fails.

We set _source to synthetic in traces-otel@mappings - strangely when I remove that, this test still fails.

[felixbarny]

This isn't really about the usage of synthetic _source itself but is related to this fallback check for LogsDB

elasticsearch/x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/SyntheticSourceLicenseService.java

Lines 46 to 55 in 4318ebb

	/**
	* @return whether synthetic source mode should fallback to stored source.
	*/
	public boolean fallbackToStoredSource() {
	if (syntheticSourceFallback) {
	return true;
	}
	return SYNTHETIC_SOURCE_FEATURE.check(licenseState) == false;
	}

By calling check, the synthetic-source feature usage gets tracked.
@martijnvg I wonder if it makes sense do adjust the way the fallback is implemented to not consider executing the fallback check usage of synthetic _source. I'd argue that actual usage only happens when an index is created with synthetic _source being used (which is not checked at all at the moment).

By enabling the OTel logs and traces data streams which are using LogsDB by default, ES always executes this fallback check.

This trips the following test that assert that no features should be used by starting up with default configuration:

• https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/logsdb/src/javaRestTest/java/org/elasticsearch/xpack/logsdb/LogsdbRestIT.java
• https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/yamlRestTest/resources/rest-api-spec/test/license/10_feature_usage.yml

[martijnvg]

So synthetic source usage will be checked once this PR gets merged: #113759

But I think for index template we should use: SYNTHETIC_SOURCE_FEATURE.checkWithoutTracking(...) instead of SYNTHETIC_SOURCE_FEATURE.check(...). The later should only be used when an actual index gets created.

I can work on a PR for that.

[felixbarny]

@gregkalapos: there are also a bunch of tests failing in native-multi-node-tests. See https://gradle-enterprise.elastic.co/s/55fntou6anahm.

@axw has disabled the apm_data plugin in these tests in the PR that enables the apm_data plugin by default (#108860)

elasticsearch/x-pack/plugin/ml/qa/native-multi-node-tests/build.gradle

Line 61 in 61d5363

setting 'xpack.apm_data.enabled', 'false'

So you'll probably need to make a similar change for this, too.

[gregkalapos]

@elasticmachine update branch

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x