Generate release notes for 8.4 release (#89475)

Author: mark-vieira

build-tools-internal/src/main/resources/templates/breaking-changes.asciidoc

@@ -72,7 +72,7 @@ if (deprecationsByNotabilityByArea.isEmpty() == false) { %>
 The following functionality has been deprecated in {es} ${majorDotMinor}
 and will be removed in a future version.
 While this won't have an immediate impact on your applications,
-we strongly encourage you take the described steps to update your code
+we strongly encourage you to take the described steps to update your code
 after upgrading to ${majorDotMinor}.
 
 To find out if you are using any deprecated functionality,

build-tools-internal/src/test/resources/org/elasticsearch/gradle/internal/release/BreakingChangesGeneratorTest.generateMigrationFile.asciidoc

@@ -89,7 +89,7 @@ Breaking change impact description 3
 The following functionality has been deprecated in {es} 8.4
 and will be removed in a future version.
 While this won't have an immediate impact on your applications,
-we strongly encourage you take the described steps to update your code
+we strongly encourage you to take the described steps to update your code
 after upgrading to 8.4.
 
 To find out if you are using any deprecated functionality,

docs/reference/migration/migrate_8_4.asciidoc

@@ -20,3 +20,33 @@ coming::[8.4.0]
 There are no breaking changes in {es} 8.4.
 // end::notable-breaking-changes[]
 
+
+[discrete]
+[[deprecated-8.4]]
+=== Deprecations
+
+The following functionality has been deprecated in {es} 8.4
+and will be removed in a future version.
+While this won't have an immediate impact on your applications,
+we strongly encourage you to take the described steps to update your code
+after upgrading to 8.4.
+
+To find out if you are using any deprecated functionality,
+enable <<deprecation-logging, deprecation logging>>.
+
+
+[discrete]
+[[deprecations_84_rest_api]]
+==== REST API deprecations
+
+[[deprecate_knn_search_endpoint]]
+.Deprecate the `_knn_search` endpoint
+[%collapsible]
+====
+*Details* +
+-| The kNN search API is deprecated in favor of the new 'knn' option inside the search API. The 'knn' option is now the recommended way of running ANN search.
+
+*Impact* +
+Users should switch from `_knn_search` to the search `knn` option.
+====
+

docs/reference/release-notes/8.4.0.asciidoc

@@ -10,10 +10,16 @@ Also see <<breaking-changes-8.4,Breaking changes in 8.4>>.
 === Bug fixes
 
 Aggregations::
+* Fix multi-value handling in composite agg {es-pull}88638[#88638]
+* Fix: extract matrix stats using `bucket_selector` `buckets_path` {es-pull}88271[#88271] (issue: {es-issue}87454[#87454])
 * Make the metric in the `buckets_path` parameter optional {es-pull}87220[#87220] (issue: {es-issue}72983[#72983])
+* Propagate alias filters to significance aggs filters {es-pull}88221[#88221] (issue: {es-issue}81585[#81585])
 
 Allocation::
 * Clamp auto-expand replicas to the closest value {es-pull}87505[#87505] (issue: {es-issue}84788[#84788])
+* Prevent re-balancing using outdated node weights in some cases {es-pull}88385[#88385] (issue: {es-issue}88384[#88384])
+* Remove any existing `read_only_allow_delete` index blocks when `cluster.routing.allocation.disk.threshold_enabled` is set to `false` {es-pull}87841[#87841] (issue: {es-issue}86383[#86383])
+* Replace health request with a state observer {es-pull}88641[#88641]
 
 Authentication::
 * Fix unique realm name check to cover default realms {es-pull}87999[#87999]
@@ -22,12 +28,30 @@ Authorization::
 * Add rollover permissions for `remote_monitoring_agent` {es-pull}87717[#87717] (issue: {es-issue}84161[#84161])
 
 Autoscaling::
+* Autoscaling during shrink {es-pull}88292[#88292] (issue: {es-issue}85480[#85480])
 * Do not include desired nodes in snapshots {es-pull}87695[#87695]
 
+Cluster Coordination::
+* Improve rejection of ambiguous voting config name {es-pull}89239[#89239]
+
+Data streams::
+* Fix renaming data streams with CCR replication {es-pull}88875[#88875] (issue: {es-issue}81751[#81751])
+
+Distributed::
+* Fixed NullPointerException on bulk request {es-pull}88385[#88385]
+
 EQL::
 * Avoid attempting PIT close on PIT open failure {es-pull}87498[#87498]
+* Improve EQL Sequence circuit breaker precision {es-pull}88538[#88538] (issue: {es-issue}88300[#88300])
+
+Geo::
+* Geo_line aggregation returns a geojson point when the resulting line has only one point {es-pull}89199[#89199] (issue: {es-issue}85748[#85748])
+* Sort ranges in `geo_distance` aggregation {es-pull}89154[#89154] (issue: {es-issue}89147[#89147])
 
 Health::
+* Fix NPE when checking if the last snapshot was success {es-pull}88811[#88811]
+* Fixing a version check for master stability functionality {es-pull}89322[#89322]
+* Fixing internal action names {es-pull}89182[#89182]
 * Using the correct connection to fetch remote master history {es-pull}87299[#87299]
 
 Highlighting::
@@ -36,84 +60,152 @@ Highlighting::
 ILM+SLM::
 * Batch ILM move to retry step task update {es-pull}86759[#86759]
 
+Infra/CLI::
+* Quote paths with whitespace in Windows service CLIs {es-pull}89072[#89072] (issue: {es-issue}89043[#89043])
+
 Infra/Core::
+* Always close directory streams {es-pull}88560[#88560]
+* Delete invalid settings for system indices {es-pull}88903[#88903] (issue: {es-issue}88324[#88324])
 * Disallow three-digit minor and revision versions {es-pull}87338[#87338]
+* Handle snapshot restore in file settings {es-pull}89321[#89321] (issue: {es-issue}89183[#89183])
+* System indices ignore all user templates {es-pull}87260[#87260] (issues: {es-issue}42508[#42508], {es-issue}74271[#74271])
 
-Ingest::
-* Don't ignore pipeline for upserts in bulk api {es-pull}87719[#87719] (issue: {es-issue}87131[#87131])
-* Geoip processor should respect the `ignore_missing` in case of missing database {es-pull}87793[#87793] (issue: {es-issue}87345[#87345])
+Infra/Node Lifecycle::
+* Fix message for stalled shutdown {es-pull}89254[#89254]
+
+Infra/Plugins::
+* Disable URL connection caching in SPIClassIterator {es-pull}88586[#88586] (issue: {es-issue}88275[#88275])
+
+Infra/Scripting::
+* Script: `UpdateByQuery` can read doc version if requested {es-pull}88740[#88740]
 
 Machine Learning::
-* Improve trained model stats API performance {es-pull}87978[#87978]
+* Address potential bug where trained models get stuck in starting after being allocated to node {es-pull}88945[#88945]
+* Fix BERT and MPNet tokenization bug when handling unicode accents {es-pull}88907[#88907] (issue: {es-issue}88900[#88900])
+* Fix NLP `question_answering` task when best answer is only one token {es-pull}88347[#88347]
+* Include start params in `_stats` for non-started model deployments {es-pull}89091[#89091]
+* fix minor tokenization bug when using fill_mask task with roberta tokenizer {es-pull}88825[#88825]
+
+Mapping::
+* Assign the right path to objects merged when parsing mappings {es-pull}89389[#89389] (issue: {es-issue}88573[#88573])
+* Don't modify source map when parsing composite runtime field {es-pull}89114[#89114]
+
+Network::
+* Ensure that the extended socket options TCP_KEEPXXX are available {es-pull}88935[#88935] (issue: {es-issue}88897[#88897])
 
 SQL::
+* Fix `SqlSearchIT` `testAllTypesWithRequestToOldNodes` {es-pull}88883[#88883] (issue: {es-issue}88866[#88866])
 * Fix date range checks {es-pull}87151[#87151] (issue: {es-issue}77179[#77179])
+* fix object equals {es-pull}87887[#87887]
+
+Search::
+* Fix: use status code 500 for aggregation reduce phase errors if no shard failed {es-pull}88551[#88551] (issue: {es-issue}20004[#20004])
+* Override bulk visit methods of exitable point visitor {es-pull}82120[#82120]
+
+Security::
+* Ensure `secureString` remain open when reloading secure settings {es-pull}88922[#88922]
 
 Snapshot/Restore::
-* Use the provided SAS token without SDK sanitation that can produce invalid signatures {es-pull}88155[#88155] (issue: {es-issue}88140[#88140])
+* Fix queued snapshot assignments after partial snapshot fails due to delete {es-pull}88470[#88470] (issue: {es-issue}86724[#86724])
 
 Transform::
-* Execute `_refresh` separately from DBQ, with system permissions {es-pull}88005[#88005] (issue: {es-issue}88001[#88001])
+* Handle update error correctly {es-pull}88619[#88619]
+
+[[deprecation-8.4.0]]
+[float]
+=== Deprecations
+
+Vector Search::
+* Deprecate the `_knn_search` endpoint {es-pull}88828[#88828]
 
 [[enhancement-8.4.0]]
 [float]
 === Enhancements
 
 Aggregations::
+* Adding cardinality support for `random_sampler` agg {es-pull}86838[#86838]
 * Minor `RangeAgg` optimization {es-pull}86935[#86935] (issue: {es-issue}84262[#84262])
 * Speed counting filters/range/date_histogram aggs {es-pull}81322[#81322]
 * Update bucket metric pipeline agg paths to allow intermediate single bucket and bucket qualified multi-bucket aggs {es-pull}85729[#85729]
 
 Allocation::
 * Add debug information to `ReactiveReason` about assigned and unassigned shards {es-pull}86132[#86132] (issue: {es-issue}85243[#85243])
+* Optimize log cluster health performance. {es-pull}87723[#87723]
 * Use desired nodes during data tier allocation decisions {es-pull}87735[#87735]
 
 Audit::
+* Audit API key ID when create or grant API keys {es-pull}88456[#88456]
+* Include API key metadata in audit log when an API key is created, granted, or updated {es-pull}88642[#88642]
+* Updatable API keys - logging audit trail event {es-pull}88276[#88276]
 * User Profile - audit support for security domain {es-pull}87097[#87097]
 
+Authentication::
+* If signature validation fails, reload JWKs and retry if new JWKs are found {es-pull}88023[#88023]
+
 Authorization::
 * App permissions with action patterns do not retrieve privileges {es-pull}85455[#85455]
 * Cancellable Profile Has Privilege check {es-pull}87224[#87224]
 * Return action denied error when user with insufficient privileges (`manage_own_api_key`) attempts a grant API key request {es-pull}87461[#87461] (issue: {es-issue}87438[#87438])
+* Update indices permissions to Enterprise Search service account {es-pull}88703[#88703]
 
 Autoscaling::
 * Add processors to autoscaling capacity response {es-pull}87895[#87895]
 * Keep track of desired nodes status in cluster state {es-pull}87474[#87474]
 
 Cluster Coordination::
+* Deduplicate mappings in persisted cluster state {es-pull}88479[#88479]
 * Expose segment details in PCSS debug log {es-pull}87412[#87412]
+* Periodic warning for 1-node cluster w/ seed hosts {es-pull}88013[#88013] (issue: {es-issue}85222[#85222])
 * Report overall mapping size in cluster stats {es-pull}87556[#87556]
 
 Data streams::
 * Give doc-value-only mappings to numeric fields on metrics templates {es-pull}87100[#87100]
 
 Distributed::
+* Adding the ability to register a `PeerFinderListener` to Coordinator {es-pull}88626[#88626]
 * Make Desired Nodes API operator-only {es-pull}87778[#87778] (issue: {es-issue}87777[#87777])
+* Support "dry run" mode for updating Desired Nodes {es-pull}88305[#88305]
 
 FIPS::
 * Log warning when hash function used by cache is not recommended in FIPS mode {es-pull}86740[#86740]
 * Log warning when hashers for stored API keys or service tokens are not compliant with FIPS {es-pull}87363[#87363]
 
 Geo::
 * Optimize geogrid aggregations for singleton points {es-pull}87439[#87439]
+* Support cartesian shape with doc values {es-pull}88487[#88487]
 * Use a faster but less accurate log algorithm for computing Geotile Y coordinate {es-pull}87515[#87515]
+* Use faster maths to project WGS84 to mercator {es-pull}88231[#88231]
 
 Health::
+* Add health user action for unhealthy SLM policy failure counts {es-pull}88523[#88523]
 * Adding a transport action to get cluster formation info {es-pull}87306[#87306]
 * Adding additional capability to the `master_is_stable` health indicator service {es-pull}87482[#87482]
 * Creating a transport action for the `CoordinationDiagnosticsService` {es-pull}87984[#87984]
 * Move the master stability logic into its own service separate from the `HealthIndicatorService` {es-pull}87672[#87672]
+* Polling cluster formation state for master-is-stable health indicator {es-pull}88397[#88397]
 * Remove cluster block preflight check from health api {es-pull}87520[#87520] (issue: {es-issue}87464[#87464])
 
+ILM+SLM::
+* Add min_* conditions to rollover {es-pull}83345[#83345]
+* Track the count of failed invocations since last successful policy snapshot {es-pull}88398[#88398]
+
 Infra/Core::
 * Improve console exception messages {es-pull}87942[#87942]
+* Print full exception when console is non-interactive {es-pull}88297[#88297]
 * Stop making index read-only when executing force merge index lifecycle management action {es-pull}81162[#81162] (issue: {es-issue}81162[#81162])
 * Stream input and output support for optional collections {es-pull}88127[#88127]
 * Update version of internal http client {es-pull}87491[#87491]
 
 Infra/Logging::
 * Catch an exception when formatting a string fails {es-pull}87132[#87132]
 
+Infra/Scripting::
+* Script: Add Metadata to ingest context {es-pull}87309[#87309]
+* Script: Metadata for update context {es-pull}88333[#88333]
+
+Infra/Settings::
+* Convert disk watermarks to RelativeByteSizeValues {es-pull}88719[#88719]
+
 Ingest::
 * Allow pipeline processor to ignore missing pipelines {es-pull}87354[#87354]
 * Move the ingest attachment processor to the default distribution {es-pull}87989[#87989]
@@ -122,20 +214,24 @@ Ingest::
 
 Machine Learning::
 * Add authorization info to ML config listings {es-pull}87884[#87884]
+* Add deployed native models to `inference_stats` in trained model stats response {es-pull}88187[#88187]
+* Add inference cache hit count to inference node stats {es-pull}88807[#88807]
+* Add new `cache_size` parameter to `trained_model` deployments API {es-pull}88450[#88450]
 * Expand allowed NER labels to be any I-O-B tagged labels {es-pull}87091[#87091]
 * Improve scalability of NLP models {es-pull}87366[#87366]
+* Indicate overall deployment failure if all node routes are failed {es-pull}88378[#88378]
+* New `frequent_items` aggregation {es-pull}83055[#83055]
 
 Mapping::
+* Enable synthetic source support on constant keyword fields {es-pull}88603[#88603]
 * Speed up `NumberFieldMapper` {es-pull}85688[#85688]
 
 Monitoring::
 * JvmService use SingleObjectCache {es-pull}87236[#87236]
 
 Network::
 * Allow start cluster with unreachable remote clusters {es-pull}87298[#87298]
-
-Performance::
-* Warn about impact of large readahead on search {es-pull}88007[#88007]
+* Increase `http.max_header_size` default to 16kb {es-pull}88725[#88725] (issue: {es-issue}88501[#88501])
 
 Query Languages::
 * Add support for VERSION field type in SQL and EQL {es-pull}87590[#87590] (issue: {es-issue}83375[#83375])
@@ -145,18 +241,33 @@ Rollup::
 
 SQL::
 * Implement support for partial search results in SQL CLI {es-pull}86982[#86982] (issue: {es-issue}86082[#86082])
+* Update Tableau connector to use connection dialog v2 {es-pull}88462[#88462]
 
 Search::
 * Add mapping stats for indexed `dense_vectors` {es-pull}86859[#86859]
+* Improve error when sorting on incompatible types {es-pull}88399[#88399] (issue: {es-issue}73146[#73146])
+* Support kNN vectors in disk usage action {es-pull}88785[#88785] (issue: {es-issue}84801[#84801])
 
 Security::
-* Automatically close idle connections in OIDC back-channel {es-pull}87773[#87773]
+* Add setting for `tcp_keepalive` for oidc back-channel {es-pull}87868[#87868]
+* Support `run_as` another user when granting API keys {es-pull}88335[#88335]
 * Support exists query for API key query {es-pull}87229[#87229]
+* Updatable API keys - REST API spec and tests {es-pull}88270[#88270]
+* Updatable API keys - noop check {es-pull}88346[#88346]
 
 Snapshot/Restore::
+* INFO logging of snapshot restore and completion {es-pull}88257[#88257] (issue: {es-issue}86610[#86610])
 * Make snapshot deletes not block the repository during data blob deletes {es-pull}86514[#86514]
+* Retry after all S3 get failures that made progress {es-pull}88015[#88015] (issue: {es-issue}87243[#87243])
+* Speed up creating new `IndexMetaDataGenerations` without removed snapshots {es-pull}88344[#88344]
 * Update HDFS Repository to HDFS 3.3.3 {es-pull}88039[#88039]
 
+Stats::
+* Sort ingest pipeline stats by use {es-pull}88035[#88035]
+
+TLS::
+* Add issuer to GET _ssl/certificates {es-pull}88445[#88445]
+
 Transform::
 * Add authorization info to transform config listings {es-pull}87570[#87570]
 * Implement per-transform num_failure_retries setting {es-pull}87361[#87361]
@@ -165,16 +276,38 @@ Transform::
 [float]
 === New features
 
+Authentication::
+* Support updates of API key attributes (single operation route) {es-pull}88186[#88186]
+
 Health::
 * Master stability health indicator part 1 (when a master has been seen recently) {es-pull}86524[#86524]
+* Remove help_url,rename summary to symptom, and `user_actions` to diagnosis {es-pull}88553[#88553] (issue: {es-issue}88474[#88474])
+
+Infra/Core::
+* File Settings Service {es-pull}88329[#88329]
 
 Infra/Logging::
 * Stable logging API - the basic use case {es-pull}86612[#86612]
 
+Machine Learning::
+* Make composite aggs in datafeeds Generally Available {es-pull}88589[#88589]
+
+Search::
+* Add 'mode' option to `_source` field mapper {es-pull}88211[#88211]
+
+TSDB::
+* TSDB: Implement downsampling ILM Action for time-series indices {es-pull}87269[#87269] (issue: {es-issue}68609[#68609])
+
+Vector Search::
+* Integrate ANN into `_search` endpoint {es-pull}88694[#88694] (issue: {es-issue}87625[#87625])
+
 [[upgrade-8.4.0]]
 [float]
 === Upgrades
 
+Infra/Core::
+* Upgrade to Log4J 2.18.0 {es-pull}88237[#88237]
+
 Network::
 * Upgrade to Netty 4.1.77 {es-pull}86630[#86630]