oss-distro-docs build doesn't test docs for OSS

[nik9000]

@lcawl opened #51570 which I thought should fail the oss-docs tests. It didn't. It turns out that the build runs:

./gradlew --parallel --scan -Dorg.elasticsearch.build.cache.url=https://gradle-enterprise.elastic.co/cache/ -S --parallel --max-workers=16 -p docs -Dtests.distribution=default -Dignore.tests.seed --build-cache check

The important bit here is:

-Dtests.distribution=default

Which is backwards. This should be:

-Dtests.distribution=oss

When I run:

./gradlew -p docs/ check -Dtests.distribution=oss

The node doesn't even come up.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[rjernst]

Looks like it's a settings problem:

»    ↓ errors and warnings from /Users/rjernst/code/elastic/elasticsearch/docs/build/testclusters/integTest-0/logs/es.stdout.log ↓
» WARN ][o.e.c.s.SettingsModule   ] [node-0]
»  *************************************************************************************
»  Found index level settings on node level configuration.
»
»  Since elasticsearch 5.x index level settings can NOT be set on the nodes
»  configuration like the elasticsearch.yaml, in system properties or command line
»  arguments.In order to upgrade all indices the settings must be updated via the
»  /${index}/_settings API. Unless all settings are dynamic all indices must be closed
»  in order to apply the upgradeIndices created in the future should use index templates
»  to set default values.
»
»  Please ensure all required values are updated on all indices by executing:
»
»  curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
»    "index.lifecycle.history_index_enabled" : "false"
»  }'
»  *************************************************************************************
»

[mark-vieira]

This looks to be the bit it doesn't like. My question is, based on the message above, why doesn't this break with the default distribution as well? Also that looks to be a warning. Is that what is actually stopping the node from starting?

[rjernst]

The node not starting was fixed in #51711. I agree it seems like the node should have failed with the default distribution as well; I don't have an explanation there.

Aside from SLM docs files needing annotations (#51711), there are 2 failures left to investigate:

Tests with failures:
 - org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/setup/restart-cluster/line_50}
 - org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/ml/anomaly-detection/categories/line_131}

[lcawl]

I'll open a PR to add the missing testenv attribute to categories.asciidoc

[lcawl]

With respect to the restart-cluster failure, I suspect it's because one of the optional steps in https://www.elastic.co/guide/en/elasticsearch/reference/master/restart-cluster.html involves a machine learning API. Is it possible to skip this page from the OSS-only tests without marking the entire page as "role=xpack" (since I think that would be misleading)?

[rjernst]

@lcawl I don't think that is possible within a single file. However, could the ML part be split out to a separate file that is included, and the other file has the xpack role?

[mark-vieira]

Sorry I'm coming back to this issue late. I'd like to understand what we are trying to test here. I assume, the purpose of this is to ensure that all the snippets in /docs work against an OSS distribution. Well... that's simply not the case. Running locally we get a number of failures, primarily related to CCR, which I assume is because this is an x-pack only feature.

https://gradle-enterprise.elastic.co/s/knsgu5iwoh3yi/tests/failed

So then my question is this? What's the "error" here? Is it that those docs should actually be under /x-pack/docs or should we be ignoring these tests, or some other misconfiguration? Did the CCR docs simply "sneak" in because we weren't actually properly testing against an OSS distro?

[nik9000]

Docs for pages that require xpack should be marked with some attribute at the top of the file. That attribute skips the tests if xpack isn't available and adds a little bug at the top of the page saying the feature is part of xpack. The goal is to make sure all pages have the bug that should and that pages that don't have it do indeed work with the Apache licensed distro.

…

On Wed, Apr 15, 2020, 18:12 Mark Vieira ***@***.***> wrote: Sorry I'm coming back to this issue late. I'd like to understand what we are trying to test here. I assume, the purpose of this is to ensure that all the snippets in /docs work against an OSS distribution. Well... that's simply not the case. Running locally we get a number of failures, primarily related to CCR, which I assume is because this is an x-pack only feature. https://gradle-enterprise.elastic.co/s/knsgu5iwoh3yi/tests/failed So then my question is this? What's the "error" here? Is it that those docs should actually be under /x-pack or should we be ignoring these tests, or some other misconfiguration? Did the CCR docs simply "sneak" in because we weren't actually properly testing against an OSS distro? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51678 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUXIQIQ32AJV4HAWDYUKTRMYWNBANCNFSM4KNWYUBQ> .

[mark-vieira]

OSS is going away so.... 👋