Skip to content

Autoscaling APIs not experimental #67202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
3fbf3b2
Autoscaling APIs not experimental
henningandersen Jan 8, 2021
1d314b9
Update docs/reference/autoscaling/index.asciidoc
henningandersen Jan 11, 2021
5e17d2a
Update docs/reference/autoscaling/index.asciidoc
henningandersen Jan 11, 2021
aa7412a
Update docs/reference/autoscaling/deciders/reactive-storage-decider.a…
henningandersen Jan 11, 2021
8759aa4
Update docs/reference/autoscaling/deciders/reactive-storage-decider.a…
henningandersen Jan 11, 2021
0448818
Update docs/reference/autoscaling/deciders/proactive-storage-decider.…
henningandersen Jan 11, 2021
e9a0f26
Update docs/reference/autoscaling/deciders/proactive-storage-decider.…
henningandersen Jan 11, 2021
2d790c6
Update docs/reference/autoscaling/deciders/machine-learning-decider.a…
henningandersen Jan 11, 2021
6fd7d52
Apply suggestions from code review
henningandersen Jan 11, 2021
9bcf5bb
Add outer policies
henningandersen Jan 11, 2021
bf00912
Apply suggestions from code review
henningandersen Jan 11, 2021
94a0a21
Max bytes of storage/memory of a node.
henningandersen Jan 11, 2021
734fa7c
Merge branch 'docs_autoscaling_not_experimental' of github.com:hennin…
henningandersen Jan 11, 2021
b8c7350
Designed for note.
henningandersen Jan 11, 2021
7ab0ebf
Add short intro to policy, roles and deciders.
henningandersen Jan 11, 2021
d0429bd
Add configuration settings sections.
henningandersen Jan 11, 2021
4695138
Apply suggestions from code review
henningandersen Jan 12, 2021
f5628d0
Update docs/reference/autoscaling/autoscaling-designed-for-note.asciidoc
henningandersen Jan 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/reference/autoscaling/apis/autoscaling-apis.asciidoc
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[role="xpack"]
[testenv="platinum"]
[testenv="enterprise"]
[[autoscaling-apis]]
== Autoscaling APIs

experimental[]
include::../autoscaling-designed-for-note.asciidoc[]

You can use the following APIs to perform autoscaling operations.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[role="xpack"]
[testenv="platinum"]
[testenv="enterprise"]
[[autoscaling-delete-autoscaling-policy]]
=== Delete autoscaling policy API
++++
<titleabbrev>Delete autoscaling policy</titleabbrev>
++++

experimental[]
include::../autoscaling-designed-for-note.asciidoc[]

Delete autoscaling policy.

Expand Down
194 changes: 192 additions & 2 deletions docs/reference/autoscaling/apis/get-autoscaling-capacity.asciidoc
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[role="xpack"]
[testenv="platinum"]
[testenv="enterprise"]
[[autoscaling-get-autoscaling-capacity]]
=== Get autoscaling capacity API
++++
<titleabbrev>Get autoscaling capacity</titleabbrev>
++++

experimental[]
include::../autoscaling-designed-for-note.asciidoc[]

Get autoscaling capacity.

Expand All @@ -33,6 +33,196 @@ This API gets the current autoscaling capacity based on the configured
autoscaling policy. This API will return information to size the cluster
appropriately to the current workload.

The `required_capacity` is calculated as the max of the `required_capacity`
result of all individual deciders that are enabled for the policy.

The operator should verify that the `current_nodes` match
the operator's knowledge of the cluster to avoid making autoscaling decisions
based on stale or incomplete information.

The response contains decider-specific information you can use to diagnose how
and why autoscaling determined a certain capacity was required. This information
is provided for diagnosis only. Do not use this information to make autoscaling
decisions.

[role="child_attributes"]
[[autoscaling-get-autoscaling-capacity-api-response-body]]
==== {api-response-body-title}

`policies`::
(object)
Contains the map of policy name to capacity result
+
.Properties of `policies`
[%collapsible%open]
====
`<policy_name>`::
(object)
Contains capacity information for the policy
+
.Properties of `<policy_name>`
[%collapsible%open]
=====
`required_capacity`::
(object)
Contains the required capacity for the policy.
+
.Properties of `required_capacity`
[%collapsible%open]
======
`node`::
(object)
Contains the minimum node sizes required per node, ensuring that individual
shards or ML jobs can fit into a single node.
+
.Properties of `node`
[%collapsible%open]
=======
`storage`::
(integer)
Bytes of storage required per node.

`memory`::
(integer)
Bytes of memory required per node.

=======
`total`::
(object)
Contains the total size required for the policy.
+
.Properties of `total`
[%collapsible%open]
=======
`storage`::
(integer)
Total bytes of storage required for the policy.

`memory`::
(integer)
Total bytes of memory required for the policy.
=======
======
`current_capacity`::
(object)
Contains the current capacity for nodes governed by the policy, i.e. the nodes
that {es} bases its calculation on.
+
.Properties of `current_capacity`
[%collapsible%open]
======
`node`::
(object)
Contains the maximum sizes of nodes governed by the policy.
+
.Properties of `node`
[%collapsible%open]
=======
`storage`::
(integer)
Maximum bytes of storage of a node.

`memory`::
(integer)
Maximum bytes of memory of a node.

=======
`total`::
(object)
Contains the current total storage and memory sizes for nodes governed by the policy.
+
.Properties of `total`
[%collapsible%open]
=======
`storage`::
(integer)
Current bytes of storage available for the policy.

`memory`::
(integer)
Current bytes of memory available for the policy.
=======
======
`current_nodes`::
(array of objects)
List of nodes used for capacity calculation.
+
.Properties of elements in `current_nodes`
[%collapsible%open]
======
`name`::
(string)
Name of the node.
======
`deciders`::
(object)
The capacity results from individual deciders, allowing insight into how the
outer level `required_capacity` was calculated.
+
.Properties of `deciders`
[%collapsible%open]
=======
`<decider_name>`::
(object)
The capacity result for a specific decider enabled for the policy.
+
.Properties of `<decider_name>`
[%collapsible%open]
========
`required_capacity`::
(object)
Required capacity determined by the decider.
+
.Properties of `required_capacity`
[%collapsible%open]
=========
`node`::
(object)
Contains the minimum node sizes required per node, ensuring that individual
shards or {ml} jobs can fit into a single node.
+
.Properties of `node`
[%collapsible%open]
==========
`storage`::
(integer)
Bytes of storage required per node.

`memory`::
(integer)
Bytes of memory required per node.

==========
`total`::
(object)
Contains the total size required for the policy.
+
.Properties of `total`
[%collapsible%open]
==========
`storage`::
(integer)
Total bytes of storage required for the policy.

`memory`::
(integer)
Total bytes of memory required for the policy.
==========
=========
`reason_summary`::
(string)
Description of the basis for the decider's result.

`reason_details`::
(object)
A per-decider structure containing details about the basis for the deciders' result.
The contents should not be relied on for application purposes and are not subject
to backwards compatibility guarantees.
========
=======
=====
====

[[autoscaling-get-autoscaling-capacity-examples]]
==== {api-examples-title}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[role="xpack"]
[testenv="platinum"]
[testenv="enterprise"]
[[autoscaling-get-autoscaling-policy]]
=== Get autoscaling policy API
++++
<titleabbrev>Get autoscaling policy</titleabbrev>
++++

experimental[]
include::../autoscaling-designed-for-note.asciidoc[]

Get autoscaling policy.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[role="xpack"]
[testenv="platinum"]
[testenv="enterprise"]
[[autoscaling-put-autoscaling-policy]]
=== Put autoscaling policy API
++++
<titleabbrev>Put autoscaling policy</titleabbrev>
++++

experimental[]
include::../autoscaling-designed-for-note.asciidoc[]

Put autoscaling policy.

Expand Down Expand Up @@ -47,6 +47,7 @@ DELETE /_autoscaling/policy/name
==== {api-description-title}

This API puts an autoscaling policy with the provided name.
See <<autoscaling-deciders,Autoscaling Deciders>> for available deciders.

[[autoscaling-put-autoscaling-policy-examples]]
==== {api-examples-title}
Expand Down
24 changes: 24 additions & 0 deletions docs/reference/autoscaling/autoscaling-deciders.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[role="xpack"]
[testenv="enterprise"]
[[autoscaling-deciders]]
== Autoscaling deciders

<<autoscaling-reactive-storage-decider,Reactive storage decider>>::
Estimates required storage capacity of current data set. Available for policies
governing data nodes.

<<autoscaling-proactive-storage-decider,Proactive storage decider>>::
Estimates required storage capacity based on current ingestion into hot nodes.
Available for policies governing hot data nodes.

<<autoscaling-machine-learning-decider,Machine learning decider>>::
Estimates required memory capacity based on machine learning jobs.
Available for policies governing machine learning nodes.

<<autoscaling-fixed-decider,Fixed decider>>::
Responds with a fixed required capacity. This decider is intended for testing only.

include::deciders/reactive-storage-decider.asciidoc[]
include::deciders/proactive-storage-decider.asciidoc[]
include::deciders/machine-learning-decider.asciidoc[]
include::deciders/fixed-decider.asciidoc[]
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[NOTE]
Autoscaling is designed for indirect use by {ess-trial}[{ess}], {ece-ref}[{ece}], and {eck-ref}[Elastic Cloud on Kubernetes]. Direct use is not supported.
67 changes: 67 additions & 0 deletions docs/reference/autoscaling/deciders/fixed-decider.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
[role="xpack"]
[testenv="enterprise"]
[[autoscaling-fixed-decider]]
=== Fixed decider

experimental[]
[WARNING]
The fixed decider is intended for testing only. Do not use this decider in production.

The `fixed` decider responds with a fixed required capacity. It is not enabled
by default but can be enabled for any policy by explicitly configuring it.

==== Configuration settings

`storage`::
(Optional, <<byte-units,byte value>>)
Required amount of node-level storage. Defaults to `-1` (disabled).

`memory`::
(Optional, <<byte-units,byte value>>)
Required amount of node-level memory. Defaults to `-1` (disabled).

`nodes`::
(Optional, integer)
Number of nodes to use when calculating capacity. Defaults to `1`.

[[autoscaling-fixed-decider-examples]]
==== {api-examples-title}

This example puts an autoscaling policy named `my_autoscaling_policy`, enabling
and configuring the fixed decider.

[source,console]
--------------------------------------------------
PUT /_autoscaling/policy/my_autoscaling_policy
{
"roles" : [ "data_hot" ],
"deciders": {
"fixed": {
"storage": "1tb",
"memory": "32gb",
"nodes": 8
}
}
}
--------------------------------------------------
// TEST

The API returns the following result:

[source,console-result]
--------------------------------------------------
{
"acknowledged": true
}
--------------------------------------------------

//////////////////////////

[source,console]
--------------------------------------------------
DELETE /_autoscaling/policy/my_autoscaling_policy
--------------------------------------------------
// TEST[continued]

//////////////////////////

Loading