Skip to content

Share common readFrom/writeTo code in AcknowledgeResponse #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 88 commits into from

Conversation

JoeNguyen93
Copy link

@JoeNguyen93 JoeNguyen93 commented Jun 5, 2018

This is example PR to demonstrate Softagram Impact reports, Original PR here: elastic#30983

sohaibiftikhar and others added 30 commits May 29, 2018 15:45
…28939)

The stored scripts API today accepts malformed requests instead of throwing an exception.
This PR deprecates accepting malformed put stored script requests (requests not using the official script format).

Relates to elastic#27612
)

This commit removes some log traces in AzureStorageServiceImpl and also
fixes the AzureStorageServiceTests so that is uses the real
implementation to create Azure clients.
We failed to register "aliases" and "version" into the list of keywords
in the IndexTemplateMetaData; then fail to parse the following index
template.

```
{
    "aliases": {"log": {}},
    "index_patterns": ["pattern-1"]
}
```
This commit registers that missing keywords.
Limits the scope of the runtime dependency on
BouncyCastle so that it can be eventually removed.

* Splits functionality related to reading and generating certificates
and keys in two utility classes so that reading certificates and
keys doesn't require BouncyCastle.
* Implements a class for parsing PEM Encoded key material (which also
adds support for reading PKCS8 encoded encrypted private keys).
* Removes BouncyCastle dependency for all of our test suites(except
for the tests that explicitly test certificate generation) by using
pre-generated keys/certificates/keystores.
This snapshot includes LUCENE-8328 which is needed to stabilize CCR builds.
Currently AbstractHttpServerTransport is in a netty4 module. This is the
incorrect location. This commit moves it out of netty4 module.
Additionally, it moves unit tests that test AbstractHttpServerTransport
logic to server.
When subprocesses are started with ProcessBuilder, they're forked by the
java process directly rather than from a shell, which can be surprising
for our use case here in the packaging tests which is similar to
scripting.

This commit changes the tests to run their subprocess commands in a
shell, using the bash -c <script> syntax for commands on linux and using
the powershell.exe -Command <script> syntax for commands on windows.
This syntax on windows is essentially what the tests were already doing.
Applies default file and directory permissions to zip distributions
similar to how they're set for the tar distributions. Previously zip
distributions would retain permissions they had on the build host's
working tree, which could vary depending on its umask

For elastic#30799
* Make sure all instance variables are final.
* Make generateKey a private static method, instead of protected.
* Rename formatter -> format for consistency.
* Serialize bucket keys as strings as opposed to optional strings.
* Pull the stream serialization logic for buckets into the Bucket class.
ML has dedicated APIs for datafeeds and jobs yet base test classes and
some tests were relying on the cluster state for this state. This commit
removes this usage in favor of using the dedicated endpoints.
This commit renames methods in the PersistentTasksService, to 
make obvious that the methods send requests in order to change 
the state of persistent tasks. 

Relates to elastic#29608.
This commit also adds index_prefixes tests to TextFieldMapperTests to ensure that cloning and wire-serialization work correctly
This change adds a new option to the composite aggregation named `missing_bucket`.
This option can be set by source and dictates whether documents without a value for the
source should be ignored. When set to true, documents without a value for a field emits
an explicit `null` value which is then added in the composite bucket.
The `missing` option that allows to set an explicit value (instead of `null`) is deprecated in this change and will be removed in a follow up (only in 7.x).
This commit also changes how the big arrays are allocated, instead of reserving
the provided `size` for all sources they are created with a small intial size and they grow
depending on the number of buckets created by the aggregation:
Closes elastic#29380
With multiple data paths, we write the state files for index metadata to all data paths. We only properly fsync on the first location, though. For other locations, we possibly expose the file before its contents is properly fsynced. This can lead to situations where, after a crash, and where the first data path is not available anymore, ES will see a partially-written state file, preventing the node to start up.
AliasMetaData should be parsed more leniently so that the high-level REST client can support forward compatibility on it. This commit addresses this issue that was found as part of elastic#28799 and adds dedicated XContent tests as well.
…ic#30926)

When we are connecting to a remote cluster we should never select
dedicated master nodes as gateway nodes, or we will end up loading them
with requests that should rather go to other type of nodes e.g. data
nodes or coord_only nodes.

This commit adds the selection based on the node role, to the existing
selection based on version and potential node attributes.

Closes elastic#30687
Adds the synced flush API to the high level REST client.

Relates to elastic#27205.
They had some copy and paste errors that failed the docs build.
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing 
ScriptException to return 400 (bad request) as status code.

Closes elastic#12315
Fix serialization after backport

Relates elastic#29465
ywelsch and others added 26 commits June 1, 2018 10:02
With the default distribution changing in 6.3, clusters might now contain custom metadata that a
pure OSS transport client cannot deserialize. As this can break transport clients when accessing
the cluster state or reroute APIs, we've decided to exclude any custom metadata that the transport
client might not be able to deserialize. This will ensure compatibility between a < 6.3 transport
client and a 6.3 default distribution cluster. Note that this PR only covers interoperability with older
clients, another follow-up PR will cover full interoperability for >= 6.3 transport clients where we will
make it possible again to get the custom metadata from the cluster state.

Relates to elastic#30731
Conflicts:
	server/src/main/java/org/elasticsearch/action/admin/cluster/reroute/ClusterRerouteResponse.java
)

Otherwise we could end up with persistent tasks metadata in the cluster that some of the nodes
might not understand in case where the cluster is during rolling upgrade from the default 6.2 to the
default 6.3 distribution.

Follow-up to elastic#30743
 Conflicts:
	server/src/main/java/org/elasticsearch/action/ingest/PutPipelineResponse.java
	server/src/main/java/org/elasticsearch/action/ingest/WritePipelineResponse.java
This commit introduces the ability for a client to communicate to the
server features that it can support and for these features to be used in
influencing the decisions that the server makes when communicating with
the client. To this end we carry the features from the client to the
underlying stream as we carry the version of the client today. This
enables us to enhance the logic where we make protocol decisions on the
basis of the version on the stream to also make protocol decisions on
the basis of the features on the stream. With such functionality, the
client can communicate to the server if it is a transport client, or if
it has, for example, X-Pack installed. This enables us to support
rolling upgrades from the OSS distribution to the default distribution
without breaking client connectivity as we can now elect to serialize
customs in the cluster state depending on whether or not the client
reports to us using the feature capabilities that it can under these
customs. This means that we would avoid sending a client pieces of the
cluster state that it can not understand. However, we want to take care
and always send the full cluster state during node-to-node communication
as otherwise we would end up with different understanding of what is in
the cluster state across nodes depending on which features they reported
to have. This is why when deciding whether or not to write out a custom
we always send the custom if the client is not a transport client and
otherwise do not send the custom if the client is transport client that
does not report to have the feature required by the custom.

Co-authored-by: Yannick Welsch <[email protected]>
…c#31034)

If you invoke elasticsearch-plugin (or any other CLI script on Windows)
with a path that has a percent-encoded space (or any other
percent-encoded character) because the CLI scripts now shell into a
common shell script (elasticsearch-cli) the percent-encoded space ends
up being interpreted as a parameter. For example passing install --batch
file:/c:/encoded%20%space/analysis-icu-7.0.0.zip to elasticsearch-plugin
leads to the %20 being interpreted as %2 followed by a zero. Here, the
%2 is interpreted as the second parameter (--batch) and the
InstallPluginCommand class ends up seeing
file:/c/encoded--batch0space/analysis-icu-7.0.0.zip as the path which
will not exist. This commit addresses this by escaping the %* that is
used to pass the parameters to the common CLI script so that the common
script sees the correct parameters without the %2 being substituted.
Reduces shape size and precision in geo shape mapper examples to reduce
amount of memory required to check docs.

Fixes elastic#23836
This commit adjusts the BWC version on client features in master to
6.4.0 after the functionality was backported to the 6.x branch.
This is related to elastic#31017. That issue identified that these three http
methods were treated like GET requests. This commit adds them to
RestRequest. This means that these methods will be handled properly and
generate 405s.
This commit adjusts the BWC version on client features in master to
6.3.0 after the functionality was backported to the 6.3 branch.
We compute a random version and later try to compute the version prior
that random version. If the random version is the earliest version in
our list of versions then it, by definition, does not have a previous
version. Yet trying to find its previous is someting we do and so the
test fails. This commit adds a version check to the randomization so
that we do not select the earliest version in our list.
We now serialize a feature array, which takes an extra byte when empty.
…tic#31045)

With elastic#31020 we introduced the ability for transport clients to indicate what features they support
in order to make sure we don't serialize object to them they don't support. This PR adapts the
serialization logic of persistent tasks to be aware of those features and not serialize tasks that
aren't supported. 

Also, a version check is added for the future where we may add new tasks implementations and
need to be able to indicate they shouldn't be serialized both to nodes and clients.

As the implementation relies on the interface of `PersistentTaskParams`, these are no longer
optional. That's acceptable as all current implementation have them and we plan to make
`PersistentTaskParams` more central in the future.

Relates to elastic#30731
This commit fixes an issue with
PersistentTasksCustomMetaDataTests#testMinVersionSerialization. There
were two problems here:
 - some versions do not have future compatible version (e.g., betas)
 - the feature logic was incorrect
This commit removes some leftover debugging statements.
Specifying `index_phrases: true` on a text field mapping will add a subsidiary
[field]._index_phrase field, indexing two-term shingles from the parent field.
The parent analysis chain is re-used, wrapped with a FixedShingleFilter.

At query time, if a phrase match query is executed, the mapping will redirect it
to run against the subsidiary field.

This should trade faster phrase querying for a larger index and longer indexing
times.

Relates to elastic#27049
* [DOCS] Rewords _field_names documentation

Corrects the language around when we write to `_field_names` and when you might want to disable it given that n recent versions it does not carry the indexing overhead it once did.

Relates to elastic#30862

* Update wording following review
The documentation for the account circuit breaker listed the settings for it's limit and overhead to be `network.breaker.accounting.limit` and `network.breaker.accounting.overhead` when in `HieratchyCircuitBreakerService` it seems the settings are actually `indices.breaker.accounting.limit` and `indices.breaker.accounting.overhead`.
@JoeNguyen93 JoeNguyen93 changed the title Pull 30983 Share common readFrom/writeTo code in AcknowledgeResponse Jun 5, 2018
@JoeNguyen93 JoeNguyen93 closed this Jun 5, 2018
@JoeNguyen93 JoeNguyen93 deleted the pull-30983 branch June 5, 2018 08:57
JoeNguyen93 pushed a commit that referenced this pull request Jun 18, 2018
The `requires_replica` yaml test feature hasn't worked for years. This
is what happens if you try to use it:
```
   > Throwable #1: java.lang.NullPointerException
   >    at __randomizedtesting.SeedInfo.seed([E6602FB306244B12:6E341069A8D826EA]:0)
   >    at org.elasticsearch.test.rest.yaml.Features.areAllSupported(Features.java:58)
   >    at org.elasticsearch.test.rest.yaml.section.SkipSection.skip(SkipSection.java:144)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:321)
```

None of our tests use it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.