Skip to content

Commit 10a8554

Browse files
jasontedorjasontedor
committed
Merge branch '6.7' into retention-lease-bwc
* 6.7: (39 commits) Remove beta label from CCR (elastic#39722) Rename retention lease setting (elastic#39719) Add Docker build type (elastic#39378) Use any index specified by .watches for Watcher (elastic#39541) (elastic#39706) Add documentation on remote recovery (elastic#39483) fix typo in synonym graph filter docs Removed incorrect ML YAML tests (elastic#39400) Improved Terms Aggregation documentation (elastic#38892) Fix Fuzziness#asDistance(String) (elastic#39643) Revert "unmute EvilLoggerTests#testDeprecatedSettings (elastic#38743)" Mute TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration (elastic#39690) Fix security index auto-create and state recovery race (elastic#39582) [DOCS] Sorts security APIs Check for .watches that wasn't upgraded properly (elastic#39609) Assert recovery done in testDoNotWaitForPendingSeqNo (elastic#39595) [DOCS] Updates API in Watcher transform context (elastic#39540) Fixing the custom object serialization bug in diffable utils. (elastic#39544) mute test SQL: Don't allow inexact fields for MIN/MAX (elastic#39563) Update release notes for 6.7.0 ...
2 parents c2d036f + 8acf59b commit 10a8554

File tree

141 files changed

+2659
-934
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

141 files changed

+2659
-934
lines changed

‎buildSrc/src/main/groovy/org/elasticsearch/gradle/test/ClusterConfiguration.groovy

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ class ClusterConfiguration {
110110
}
111111
if (ant.properties.containsKey("failed.${seedNode.transportPortsFile.path}".toString())) {
112112
throw new GradleException("Failed to locate seed node transport file [${seedNode.transportPortsFile}]: " +
113-
"timed out waiting for it to be created after ${waitSeconds} seconds")
113+
"timed out waiting for it to be created after 40 seconds")
114114
}
115115
return seedNode.transportUri()
116116
}

‎distribution/docker/src/docker/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ WORKDIR /usr/share/elasticsearch
3333
${source_elasticsearch}
3434

3535
RUN tar zxf /opt/${elasticsearch} --strip-components=1
36+
RUN grep ES_DISTRIBUTION_TYPE=tar /usr/share/elasticsearch/bin/elasticsearch-env \
37+
&& sed -ie 's/ES_DISTRIBUTION_TYPE=tar/ES_DISTRIBUTION_TYPE=docker/' /usr/share/elasticsearch/bin/elasticsearch-env
3638
RUN mkdir -p config data logs
3739
RUN chmod 0775 config data logs
3840
COPY config/elasticsearch.yml config/log4j2.properties config/

‎docs/painless/painless-contexts/painless-watcher-context-example.asciidoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ The following example shows the use of metadata and transforming dates into a re
104104

105105
[source,Painless]
106106
----
107-
POST _xpack/watcher/watch/_execute
107+
POST _watcher/watch/_execute
108108
{
109109
"watch" : {
110110
"metadata" : { "min_hits": 10000 },

‎docs/reference/aggregations/bucket/terms-aggregation.asciidoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,13 +61,15 @@ GET /_search
6161
{
6262
"aggs" : {
6363
"genres" : {
64-
"terms" : { "field" : "genre" }
64+
"terms" : { "field" : "genre" } <1>
6565
}
6666
}
6767
}
6868
--------------------------------------------------
6969
// CONSOLE
7070
// TEST[s/_search/_search\?filter_path=aggregations/]
71+
<1> `terms` aggregation should be a field of type `keyword` or any other data type suitable for bucket aggregations. In order to use it with `text` you will need to enable
72+
<<fielddata, fielddata>>.
7173

7274
Response:
7375

‎docs/reference/analysis/tokenfilters/synonym-graph-tokenfilter.asciidoc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,8 @@ PUT /test_index
174174
Using `synonyms_path` to define WordNet synonyms in a file is supported
175175
as well.
176176

177-
=== Parsing synonym files
177+
[float]
178+
==== Parsing synonym files
178179

179180
Elasticsearch will use the token filters preceding the synonym filter
180181
in a tokenizer chain to parse the entries in a synonym file. So, for example, if a
@@ -186,7 +187,7 @@ parsing synonyms, e.g. `asciifolding` will only produce the folded version of th
186187
token. Others, e.g. `multiplexer`, `word_delimiter_graph` or `ngram` will throw an
187188
error.
188189

189-
WARNING:The synonym rules should not contain words that are removed by
190+
WARNING: The synonym rules should not contain words that are removed by
190191
a filter that appears after in the chain (a `stop` filter for instance).
191192
Removing a term from a synonym rule breaks the matching at query time.
192193

‎docs/reference/ccr/apis/auto-follow/delete-auto-follow-pattern.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Delete auto-follow pattern</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Delete auto-follow patterns.
1210

1311
==== Description

‎docs/reference/ccr/apis/auto-follow/get-auto-follow-pattern.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Get auto-follow pattern</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Get auto-follow patterns.
1210

1311
==== Description

‎docs/reference/ccr/apis/auto-follow/put-auto-follow-pattern.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Create auto-follow pattern</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Creates an auto-follow pattern.
1210

1311
==== Description

‎docs/reference/ccr/apis/ccr-apis.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@
33
[[ccr-apis]]
44
== Cross-cluster replication APIs
55

6-
beta[]
7-
86
You can use the following APIs to perform {ccr} operations.
97

108
[float]

‎docs/reference/ccr/apis/follow/get-follow-info.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Get follower info</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Retrieves information about all follower indices.
1210

1311
==== Description

‎docs/reference/ccr/apis/follow/get-follow-stats.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Get follower stats</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Get follower stats.
1210

1311
==== Description

‎docs/reference/ccr/apis/follow/post-pause-follow.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Pause follower</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Pauses a follower index.
1210

1311
==== Description

‎docs/reference/ccr/apis/follow/post-resume-follow.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Resume follower</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Resumes a follower index.
1210

1311
==== Description

‎docs/reference/ccr/apis/follow/post-unfollow.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Unfollow</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Converts a follower index to a regular index.
1210

1311
==== Description

‎docs/reference/ccr/apis/follow/put-follow.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Create follower</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Creates a follower index.
1210

1311
==== Description

‎docs/reference/ccr/apis/get-ccr-stats.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@
66
<titleabbrev>Get CCR stats</titleabbrev>
77
++++
88

9-
beta[]
10-
119
Get {ccr} stats.
1210

1311
==== Description

‎docs/reference/ccr/auto-follow.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@
33
[[ccr-auto-follow]]
44
=== Automatically following indices
55

6-
beta[]
7-
86
In time series use cases where you want to follow new indices that are
97
periodically created (such as daily Beats indices), manually configuring follower
108
indices for each new leader index can be an operational burden. The auto-follow

‎docs/reference/ccr/getting-started.asciidoc

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@
33
[[ccr-getting-started]]
44
== Getting started with {ccr}
55

6-
beta[]
7-
86
This getting-started guide for {ccr} shows you how to:
97

108
* <<ccr-getting-started-remote-cluster,Connect a local cluster to a remote
@@ -261,6 +259,11 @@ PUT /server-metrics-copy/_ccr/follow?wait_for_active_shards=1
261259
262260
//////////////////////////
263261

262+
The follower index is initialized using the <<remote-recovery, remote recovery>>
263+
process. The remote recovery process transfers the existing Lucene segment files
264+
from the leader to the follower. When the remote recovery process is complete,
265+
the index following begins.
266+
264267
Now when you index documents into your leader index, you will see these
265268
documents replicated in the follower index. You can
266269
inspect the status of replication using the

‎docs/reference/ccr/index.asciidoc

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,7 @@
66
[partintro]
77
--
88

9-
beta[]
10-
11-
WARNING: {ccr} is currently not supported by indices being managed by
12-
{ref}/index-lifecycle-management.html[Index Lifecycle Management].
13-
14-
The {ccr} (CCR) feature enables replication of indices in remote clusters to a
9+
The {ccr} (CCR) feature enables replication of indices in remote clusters to a
1510
local cluster. This functionality can be used in some common production use
1611
cases:
1712

@@ -32,3 +27,4 @@ include::overview.asciidoc[]
3227
include::requirements.asciidoc[]
3328
include::auto-follow.asciidoc[]
3429
include::getting-started.asciidoc[]
30+
include::remote-recovery.asciidoc[]

‎docs/reference/ccr/overview.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@
33
[[ccr-overview]]
44
== Overview
55

6-
beta[]
7-
86
Cross-cluster replication is done on an index-by-index basis. Replication is
97
configured at the index level. For each configured replication there is a
108
replication source index called the _leader index_ and a replication target

‎docs/reference/ccr/remote-recovery.asciidoc

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
[role="xpack"]
2+
[testenv="platinum"]
3+
[[remote-recovery]]
4+
== Remote recovery
5+
6+
When you create a follower index, you cannot use it until it is fully initialized.
7+
The _remote recovery_ process builds a new copy of a shard on a follower node by
8+
copying data from the primary shard in the leader cluster. {es} uses this remote
9+
recovery process to bootstrap a follower index using the data from the leader index.
10+
This process provides the follower with a copy of the current state of the leader index,
11+
even if a complete history of changes is not available on the leader due to Lucene
12+
segment merging.
13+
14+
Remote recovery is a network intensive process that transfers all of the Lucene
15+
segment files from the leader cluster to the follower cluster. The follower
16+
requests that a recovery session be initiated on the primary shard in the leader
17+
cluster. The follower then requests file chunks concurrently from the leader. By
18+
default, the process concurrently requests `5` large `1mb` file chunks. This default
19+
behavior is designed to support leader and follower clusters with high network latency
20+
between them.
21+
22+
There are dynamic settings that you can use to rate-limit the transmitted data
23+
and manage the resources consumed by remote recoveries. See
24+
{ref}/ccr-settings.html[{ccr-cap} settings].
25+
26+
You can obtain information about an in-progress remote recovery by using the
27+
{ref}/cat-recovery.html[recovery API] on the follower cluster. Remote recoveries
28+
are implemented using the {ref}/modules-snapshots.html[snapshot and restore] infrastructure. This means that on-going remote recoveries are labelled as type
29+
`snapshot` in the recovery API.

‎docs/reference/ccr/requirements.asciidoc

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@
33
[[ccr-requirements]]
44
=== Requirements for leader indices
55

6-
beta[]
7-
86
Cross-cluster replication works by replaying the history of individual write
97
operations that were performed on the shards of the leader index. This means that the
108
history of these operations needs to be retained on the leader shards so that

‎docs/reference/commands/setup-passwords.asciidoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
[[setup-passwords]]
44
== elasticsearch-setup-passwords
55

6-
The `elasticsearch-setup-passwords` command sets the passwords for the built-in
7-
`elastic`, `kibana`, `logstash_system`, `beats_system`, and `apm_system` users.
6+
The `elasticsearch-setup-passwords` command sets the passwords for the
7+
{stack-ov}/built-in-users.html[built-in users].
88

99
[float]
1010
=== Synopsis

‎docs/reference/release-notes/6.7.asciidoc

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,9 @@ Authorization::
101101
CCR::
102102
* Add ccr follow info api {pull}37408[#37408] (issue: {issue}37127[#37127])
103103

104+
CRUD::
105+
* Make `_doc` work as an alias of the actual type of an index. {pull}39505[#39505] (issue: {issue}39469[#39469])
106+
104107
Features/ILM::
105108
* [ILM] Add unfollow action {pull}36970[#36970] (issue: {issue}34648[#34648])
106109

@@ -260,6 +263,7 @@ Infra/Scripting::
260263
* Add getZone to JodaCompatibleZonedDateTime {pull}37084[#37084]
261264

262265
Infra/Settings::
266+
* Provide a clearer error message on keystore add {pull}39327[#39327] (issue: {issue}39324[#39324])
263267
* Separate out validation of groups of settings {pull}34184[#34184]
264268

265269
License::
@@ -298,6 +302,7 @@ Rollup::
298302
* Replace the TreeMap in the composite aggregation {pull}36675[#36675]
299303

300304
SQL::
305+
* SQL: Enhance checks for inexact fields {pull}39427[#39427] (issue: {issue}38501[#38501])
301306
* SQL: change the default precision for CURRENT_TIMESTAMP function {pull}39391[#39391] (issue: {issue}39288[#39288])
302307
* SQL: add "validate.properties" property to JDBC's allowed list of settings {pull}39050[#39050] (issue: {issue}38068[#38068])
303308
* SQL: Allow look-ahead resolution of aliases for WHERE clause {pull}38450[#38450] (issue: {issue}29983[#29983])
@@ -456,6 +461,7 @@ Geo::
456461
* Geo: Do not normalize the longitude with value -180 for Lucene shapes {pull}37299[#37299] (issue: {issue}37297[#37297])
457462

458463
Infra/Core::
464+
* Correct name of basic_date_time_no_millis {pull}39367[#39367]
459465
* Fix DateFormatters.parseMillis when no timezone is given {pull}39100[#39100] (issue: {issue}39067[#39067])
460466
* Prefix java formatter patterns with '8' {pull}38712[#38712] (issue: {issue}38567[#38567])
461467
* Bubble-up exceptions from scheduler {pull}38317[#38317] (issue: {issue}38014[#38014])
@@ -508,6 +514,10 @@ Recovery::
508514
* RecoveryMonitor#lastSeenAccessTime should be volatile {pull}36781[#36781]
509515

510516
SQL::
517+
* SQL: Fix merging of incompatible multi-fields {pull}39560[#39560] (issue: {issue}39547[#39547])
518+
* SQL: fix COUNT DISTINCT column name {pull}39537[#39537] (issue: {issue}39511[#39511])
519+
* SQL: ignore UNSUPPORTED fields for JDBC and ODBC modes in 'SYS COLUMNS' {pull}39518[#39518] (issue: {issue}39471[#39471])
520+
* SQL: Use underlying exact field for LIKE/RLIKE {pull}39443[#39443] (issue: {issue}39442[#39442])
511521
* SQL: enforce JDBC driver - ES server version parity {pull}38972[#38972] (issue: {issue}38775[#38775])
512522
* SQL: fall back to using the field name for column label {pull}38842[#38842] (issue: {issue}38831[#38831])
513523
* SQL: Prevent grouping over grouping functions {pull}38649[#38649] (issue: {issue}38308[#38308])
@@ -531,6 +541,7 @@ SQL::
531541
* SQL: Fix issue with always false filter involving functions {pull}36830[#36830] (issue: {issue}35980[#35980])
532542
* SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests {pull}36800[#36800] (issue: {issue}36756[#36756])
533543
* SQL: Enhance Verifier to prevent aggregate or grouping functions from {pull}36799[#36799] (issue: {issue}36798[#36798])
544+
* SQL: normalized keywords shouldn't be allowed for groupings and sorting [ISSUE] {pull}35203[#35203]
534545

535546
Search::
536547
* Fix simple query string serialization conditional {pull}38960[#38960] (issues: {issue}21504[#21504], {issue}38889[#38889])
@@ -546,7 +557,9 @@ Security::
546557
* Fix potential NPE in UsersTool {pull}37660[#37660]
547558

548559
Snapshot/Restore::
560+
* Fix Concurrent Snapshot Ending And Stabilize Snapshot Finalization {pull}38368[#38368] (issue: {issue}38226[#38226])
549561
* Fix Two Races that Lead to Stuck Snapshots {pull}37686[#37686] (issues: {issue}32265[#32265], {issue}32348[#32348])
562+
* Fix Race in Concurrent Snapshot Delete and Create {pull}37612[#37612] (issue: {issue}37581[#37581])
550563
* Streamline S3 Repository- and Client-Settings {pull}37393[#37393]
551564
* SNAPSHOTS: Upgrade GCS Dependencies to 1.55.0 {pull}36634[#36634] (issues: {issue}35229[#35229], {issue}35459[#35459])
552565

‎docs/reference/settings/ccr-settings.asciidoc

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
[role="xpack"]
2+
[[ccr-settings]]
3+
=== {ccr-cap} settings
4+
5+
These {ccr} settings can be dynamically updated on a live cluster with the
6+
<<cluster-update-settings,cluster update settings API>>.
7+
8+
[float]
9+
[[ccr-recovery-settings]]
10+
==== Remote recovery settings
11+
12+
The following setting can be used to rate-limit the data transmitted during
13+
{stack-ov}/remote-recovery.html[remote recoveries]:
14+
15+
`ccr.indices.recovery.max_bytes_per_sec` (<<cluster-update-settings,Dynamic>>)::
16+
Limits the total inbound and outbound remote recovery traffic on each node.
17+
Since this limit applies on each node, but there may be many nodes performing
18+
remote recoveries concurrently, the total amount of remote recovery bytes may be
19+
much higher than this limit. If you set this limit too high then there is a risk
20+
that ongoing remote recoveries will consume an excess of bandwidth (or other
21+
resources) which could destabilize the cluster. This setting is used by both the
22+
leader and follower clusters. For example if it is set to `20mb` on a leader,
23+
the leader will only send `20mb/s` to the follower even if the follower is
24+
requesting and can accept `60mb/s`. Defaults to `40mb`.
25+
26+
[float]
27+
[[ccr-advanced-recovery-settings]]
28+
==== Advanced remote recovery settings
29+
30+
The following _expert_ settings can be set to manage the resources consumed by
31+
remote recoveries:
32+
33+
`ccr.indices.recovery.max_concurrent_file_chunks` (<<cluster-update-settings,Dynamic>>)::
34+
Controls the number of file chunk requests that can be sent in parallel per
35+
recovery. As multiple remote recoveries might already running in parallel,
36+
increasing this expert-level setting might only help in situations where remote
37+
recovery of a single shard is not reaching the total inbound and outbound remote recovery traffic as configured by `ccr.indices.recovery.max_bytes_per_sec`.
38+
Defaults to `5`. The maximum allowed value is `10`.
39+
40+
`ccr.indices.recovery.chunk_size`(<<cluster-update-settings,Dynamic>>)::
41+
Controls the chunk size requested by the follower during file transfer. Defaults to
42+
`1mb`.
43+
44+
`ccr.indices.recovery.recovery_activity_timeout`(<<cluster-update-settings,Dynamic>>)::
45+
Controls the timeout for recovery activity. This timeout primarily applies on
46+
the leader cluster. The leader cluster must open resources in-memory to supply
47+
data to the follower during the recovery process. If the leader does not receive recovery requests from the follower for this period of time, it will close the resources. Defaults to 60 seconds.
48+
49+
`ccr.indices.recovery.internal_action_timeout` (<<cluster-update-settings,Dynamic>>)::
50+
Controls the timeout for individual network requests during the remote recovery
51+
process. An individual action timing out can fail the recovery. Defaults to
52+
60 seconds.

‎docs/reference/settings/configuring-xes.asciidoc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66
++++
77

88
include::{asciidoc-dir}/../../shared/settings.asciidoc[]
9+
include::ccr-settings.asciidoc[]
910
include::license-settings.asciidoc[]
1011
include::ml-settings.asciidoc[]
11-
include::notification-settings.asciidoc[]
1212
include::sql-settings.asciidoc[]
13+
include::notification-settings.asciidoc[]

0 commit comments

Comments
 (0)