@@ -30,20 +30,20 @@ in an ingest pipeline or directly in the <<infer-trained-model>> API.
3030Scaling inference performance can be achieved by setting the parameters
3131`number_of_allocations` and `threads_per_allocation`.
3232
33- Increasing `threads_per_allocation` means more threads are used when
34- an inference request is processed on a node. This can improve inference speed
35- for certain models. It may also result in improvement to throughput.
33+ Increasing `threads_per_allocation` means more threads are used when an
34+ inference request is processed on a node. This can improve inference speed for
35+ certain models. It may also result in improvement to throughput.
3636
37- Increasing `number_of_allocations` means more threads are used to
38- process multiple inference requests in parallel resulting in throughput
39- improvement. Each model allocation uses a number of threads defined by
37+ Increasing `number_of_allocations` means more threads are used to process
38+ multiple inference requests in parallel resulting in throughput improvement.
39+ Each model allocation uses a number of threads defined by
4040`threads_per_allocation`.
4141
42- Model allocations are distributed across {ml} nodes. All allocations assigned
43- to a node share the same copy of the model in memory. To avoid
44- thread oversubscription which is detrimental to performance, model allocations
45- are distributed in such a way that the total number of used threads does not
46- surpass the node's allocated processors.
42+ Model allocations are distributed across {ml} nodes. All allocations assigned to
43+ a node share the same copy of the model in memory. To avoid thread
44+ oversubscription which is detrimental to performance, model allocations are
45+ distributed in such a way that the total number of used threads does not surpass
46+ the node's allocated processors.
4747
4848[[start-trained-model-deployment-path-params]]
4949== {api-path-parms-title}
@@ -57,33 +57,36 @@ include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=model-id]
5757
5858`cache_size`::
5959(Optional, <<byte-units,byte value>>)
60- The inference cache size (in memory outside the JVM heap) per node for the model.
61- The default value is the same size as the `model_size_bytes`. To disable the cache, `0b` can be provided.
60+ The inference cache size (in memory outside the JVM heap) per node for the
61+ model. The default value is the size of the model as reported by the
62+ `model_size_bytes` field in the <<get-trained-models-stats>>. To disable the
63+ cache, `0b` can be provided.
6264
6365`number_of_allocations`::
6466(Optional, integer)
6567The total number of allocations this model is assigned across {ml} nodes.
66- Increasing this value generally increases the throughput.
67- Defaults to 1.
68+ Increasing this value generally increases the throughput. Defaults to 1.
6869
6970`queue_capacity`::
7071(Optional, integer)
7172Controls how many inference requests are allowed in the queue at a time.
7273Every machine learning node in the cluster where the model can be allocated
7374has a queue of this size; when the number of requests exceeds the total value,
74- new requests are rejected with a 429 error. Defaults to 1024. Max allowed value is 1000000.
75+ new requests are rejected with a 429 error. Defaults to 1024. Max allowed value
76+ is 1000000.
7577
7678`threads_per_allocation`::
7779(Optional, integer)
78- Sets the number of threads used by each model allocation during inference. This generally increases
79- the speed per inference request. The inference process is a compute-bound process;
80- `threads_per_allocations` must not exceed the number of available allocated processors per node.
81- Defaults to 1. Must be a power of 2. Max allowed value is 32.
80+ Sets the number of threads used by each model allocation during inference. This
81+ generally increases the speed per inference request. The inference process is a
82+ compute-bound process; `threads_per_allocations` must not exceed the number of
83+ available allocated processors per node. Defaults to 1. Must be a power of 2.
84+ Max allowed value is 32.
8285
8386`timeout`::
8487(Optional, time)
85- Controls the amount of time to wait for the model to deploy. Defaults
86- to 20 seconds.
88+ Controls the amount of time to wait for the model to deploy. Defaults to 20
89+ seconds.
8790
8891`wait_for`::
8992(Optional, string)
0 commit comments