Merge Remote Tracking Branch #3
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds support for reading and writing sets as generic values in stream input and output. closes #54708
Unmutes weighted avg rest test
This reverts commit dfddd8b
This is a first cut at giving NodeInfo the ability to carry a flexible list of heterogeneous info responses. The trick is to be able to serialize and deserialize an arbitrary list of blocks of information. It is convenient to be able to deserialize into usable Java objects so that we can aggregate nodes stats for the cluster stats endpoint. In order to provide a little bit of clarity about which objects can and can't be used as info blocks, I've introduced a new interface called "ReportingService." I have removed the hard-coded getters (e.g., getOs()) in favor of a flexible method that can return heterogeneous kinds of info blocks (e.g., getInfo(OsInfo.class)). Taking a class as an argument removes the need to cast in the client code.
…rds (#55133) * [DOCS] Removed obsolete warning about no way to securely store passwords. * Update x-pack/docs/en/watcher/actions/email.asciidoc Co-Authored-By: James Rodewig <[email protected]>
Fixes a couple of related failures in SearchableSnapshotsIntegTests. Firstly, we were not correctly accounting for the case where the cache was so small that some/all files were read directly; fixed this by only asserting that the cache is definitely used if the corresponding node has a cache that's large enough to hold the whole index. Secondly, we were not permitting shards to be completely empty, which might be the case (rarely) if there were not many documents indexed and the distribution of IDs was a bit unlucky; fixed this by asserting that we get stats for at least one file for the whole index, rather than for each shard separately. Closes #55126
Test failures are tracked in #55106
Provides basic repository-level stats that will allow us to get some insight into how many requests are actually being made by the underlying SDK. Currently only tracks GET and LIST calls for S3 repositories. Most of the code is unfortunately boiler plate to add a new endpoint that will help us better understand some of the low-level dynamics of searchable snapshots.
…#55141) This commit refactors the `AuditTrail` to use the `TransportRequest` as a parameter for all its audit methods, instead of the current `TransportMessage` super class. The goal is to gain access to the `TransportRequest#parentTaskId` member, so that it can be audited. The `parentTaskId` is used internally when spawning tasks that handle transport requests; in this way tasks across nodes are related by the same parent task. Relates #52314
This change ensures that internal client requests spawned by the transform persistent task executor and that use the end user security credentials, have the parent task id assigned. The objective here is to permit auditing (as well as tracking for debugging purposes) of all the end-user requests executed on its behalf by persistent tasks. Because transform tasks already implements graceful shutdown of the child tasks, this change does not interfere with that by opting out of the persistent task cancellation of child tasks. Relates #55046 #54943 #52314 Closes #54957
Just like in `AbstractCoordinatorTestCase` we can't just assume the cluster is stable once all the cluster states align since stray follower/leader check tasks could still hit us after a disconnect, causing future test operations to fail. => fixed by running all tasks in the possible time span of running into these checks before validating that cluster states align on all nodes to prevent this like we do in the coordinator tests. Closes #55103
Updates the supported upgrade path table in [Upgrade Elasticsearch][0] to include a new row for maintenance releases. For example, this row covers upgrading from 7.6.0 to 7.6.2. The new table row only displays for releases greater than n.x.0. For example, the new row will display for the 7.7.1 release but not the 7.7.0 release. [0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-upgrade.html
Currently there is a clear mechanism to stub sending a request through the transport. However, this is limited to testing exceptions on the sender side. This commit reworks our transport related testing infrastructure to allow stubbing request handling on the receiving side.
`master` doesn't need to talk to Elasticsearch versions before 7.8.0 so `PipelineAggregator` doesn't need to be writable *at all* in master. New pipeline aggregations don't need to worry about serializing `PipelineAggregator` at all so this drops all of it. For the most part we don't need to worry about serialization of `PipelineAggregator` at all any more. When backporting a change to an aggregator that is serialized to previous versions of Elasticsearch it *should* be fairly simple to pick a value to serialize. And the compiler *should* tell you that you need to do it. In many cases you this'll be a noop. *Hopefully* all cases. Closes #53742
This removes deprecated endpoints in hot threads action. Closes #52640
Trivial change that removes all remaining mentions of data science plugin that was renamed into analytics plugin. Follow up for #54847
Apply the :distribution:archives naming convention to some of the Docker sub-projects, so that we have a more consistent naming scheme. Also, we've seen some examples of Docker packaging tests failing sporadically when they try to clean up the temp directory, citing a not-empty directory. Ensure that any running container is removed before cleaning up the temp dir, in an effort to avoid this problem.
Prior to the change in #51631 indices were moved to the `TerminalPolicyStep` when their ILM actions had completed. Once we switched ILM to stop in the last policy configured, these steps because inaccessible from the policy's perspective. This meant that indices upgraded from ES prior to 7.7.0 could see the following error spammed in their logs every 10 minutes (by default) for every index in this state: ``` [2020-04-14T15:52:23,764][ERROR][o.e.x.i.IndexLifecycleRunner] [midgar] current step [{"phase":"completed","action":"completed","name":"completed"}] for index [foo] with policy [full] is not recognized ``` This changes the runner to ignore these steps, which is what is desired anyway since the index is already in the terminal phase.
…54969) This change removes the extendable AStoreable node from the Painless user tree. AStoreable was used to determine if a node was allowed to store a value. However, this was fragile and inflexible as the instanceof check was only 1-level deep so nodes still could throw errors under certain circumstances. This has been replaced by an expression input "write" to tell an expression node whether or not it is meant to store a value. This allows expressions nodes to determine the best course of action when asked to store a value and also gives better error messaging. This also makes isDefOptimized an output of an expression as opposed to a mutable value on an expression. This is used to determine if a def method returning a value can change the returned value as part of the method signature as opposed to an extra cast.
This commit updates the Java version used within vagrant tests by gradle inside the VM to use JDK14. closes #55044
If we start unbanning when the last child task completed and that child task executed with a specific user, then unban requests are denied because internal requests can't run with a user. We need to remove bans with the current thread context.
While the current version of forbidden apis still does not support java 14, the warning message has become very noisy as we now require java 14 for the elasticsearch build. This commit replaces the warn log message with a comment in the code.
While communicating with an earlier version of elasticsearch, VectorsFeatureSetUsage is using writeInt instead of writeVInt that 7.x version is expecting. This commit fixes the issue. Fixes #55378
Security features in the license state currently do a dynamic check on whether security is enabled. This is because the license level can change the default security enabled state. This commit splits out the check on security being enabled, so that the combo method of security enabled plus license allowed is no longer necessary.
Some aggregations, such as the Terms* family, will use an alternate class to represent unmapped shard results (while the rest of the aggs use the same object but with some form of "empty" or "nullish" values to represent unmapped). This was problematic with AbstractWireSerializingTestCase because it expects the instanceReader to always match the original class. Instead, we need to use the NamedWriteable version so that the registry can be consulted for the proper deserialization reader.
We have some Dockerfiles that reference Ubuntu 19.04, which is not an LTS version and has now appears to have been retired from the Ubuntu repositories. Switch to 18.04, which is the current long-term support version. This also requires a switch from OpenJDK 12 to 11. Also change a usage of 16.04 to 18.04, for consistency.
* Fix security manager bug writing large blobs to GCS
This commit addresses a security manager permissions issue writing large
blobs (on the resumable upload path) to GCS. The underlying issue here
is that we need to wrap the close and write calls on the channel. It is
not enough to do this:
SocketAccess.doPrivilegedVoidIOException(
() -> Streams.copy(
inputStream,
Channels.newOutputStream(client().writer(blobInfo, writeOptions))));
This reason that this is not enough is because Streams#copy will be in
the stacktrace and it is not granted the security manager permissions
needed to close or write this channel. We only grant those permissions
to classes loaded in the plugin classloader, and Streams#copy is from
the parent classloader. This is why we must wrap the close and write
calls as privileged, to truncate the Streams#copy call out of the
stacktrace.
The reason that this issue is not caught in testing is because the size
of data that we use in testing is too small to trigger the large blob
resumable upload path. Therefore, we address this by adding a system
property to control the threshold, which we can then set in tests to
exercise this code path. Prior to rewriting the writeBlobResumable
method to wrap the close and write calls as privileged, with this
additional test, we are able to reproduce the security manager
permissions issue. After adding the wrapping, this test now passes.
* Fix forbidden APIs issue
* Remove leftover debugging
This change adds an assertion making sure that we won't accidentally open directory readers using transport threads.
This validation is not needed, as we have discovered the source of the serialization error that was leading to some usage instances appearing to not have a name.
This test tries to set the number of allocated processors to seven. That is fine, unless the test is running on a machine where Runtime#availableProcessors is less than seven (typically when there are fewer than seven logical cores, but it can happen in various other ways related to CPU quotas in Docker containers with Docker-aware JVMs, etc.). This commit addresses this by randomizing the number of allocated processors, to a value that is less than or equal to the hard constraint on the number of allocated processors.
isHidden was a `Boolean` in order to treat a special case identified with V1 templates where if the create index request didn't specify if the index should be hidden or not (ie. isHidden was `null`) but the index matched a template that specified the `index.hidden` setting we needed to remove the global templates from the templates we'll apply to the new index (note: this is important with V1 templates as inheritance is supported). With V2 templates we match only one template with an index so the equivalent check did not need to exist (we added a sanity check in #55015 where we make sure we don't apply an invalid global template - one that specifes the `index.hidden` setting, but this is a check we make irrespective of the user specifying or not if the index should be hidden) This commit makes `isHidden` when matching V2 templates a boolean primitive, eliminating the need for the `null` state to exist. Note that some methods which use the matching V2 templates still work with a `Boolean` object `isHidden` attribute as they are also matching the V1 templates. These methods will pass in `false` instead of `null` when finding the V2 templates.
In #55298 we saw a failure of `CoordinationStateTests#testSafety` in which a single master-eligible node is bootstrapped, then rebooted as a master-ineligible node (losing its persistent state) and then rebooted as a master-eligible node and bootstrapped again. This happens because this test loses too much of the persistent state; in fact once bootstrapped the node would not allow itself to be bootstrapped again. This commit adjusts the test logic to reflect this. Closes #55298
Adds ranged read support for GCS repositories in order to enable searchable snapshot support for GCS. As part of this PR, I've extracted some of the test infrastructure to make sure that GoogleCloudStorageBlobContainerRetriesTests and S3BlobContainerRetriesTests are covering similar test (as I saw those diverging in what they cover)
* Fix Path Style Access Setting Priority Fixing obvious bug in handling path style access if it's the only setting overridden by the repository settings. Closes #55407
This is long outdated and can be removed in `8` now.
This allows us to perform some dummy indexing including updates/deletes.
Removing the deprecated "xpack.monitoring.enabled" setting introduced log spam and potentially some failures in ML tests. It's possible to use a different, non-deprecated setting to disable monitoring, so we do that here.
Co-authored-by: Igor Motov <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
We don't really need `LinkedHashSet` here. We can assume that all the entries are unique and just use a list and use the list utilities to create the cheapest possible version of the list. Also, this fixes a bug in `addSnapshot` which would mutate the existing linked hash set on the current instance (fortunately this never caused a real world bug) and brings the collection in line with the java docs on its getter that claim immutability.
This TODO became fixable with #54765
This commit adds a new querystring parameter on the following APIs: - Index - Update - Bulk - Create Index - Rollover These APIs now support a `?prefer_v2_templates=true|false` flag. This flag changes the preference creation to use either V2 index templates or V1 templates. This flag defaults to `false` and will be changed to `true` for 8.0+ in subsequent work. Additionally, setting this flag internally sets the `index.prefer_v2_templates` index-level setting. This setting is used so that actions that automatically create a new index (things like rollover initiated by ILM) will inherit the preference from the original index. This setting is dynamic so that a transition from v1 to v2 templates can occur for long-running indices grouped by an alias performing periodic rollover. This also adds support for sending this parameter to the High Level Rest Client. Relates to #53101
This adds a dependency to ingest common. This removes the log spam resulting from basic plugins being enabled that require the common ingest processors.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.