Skip to content

Commit a41f5e4

Browse files
authored
Reword frozen indices introduction (elastic#40160)
Clarifies that we don't move anything extra to persistent storage when freezing an index, and gives a bit more context for less experienced users.
1 parent 0e0357c commit a41f5e4

File tree

1 file changed

+35
-14
lines changed

1 file changed

+35
-14
lines changed

docs/reference/frozen-indices.asciidoc

+35-14
Original file line numberDiff line numberDiff line change
@@ -5,20 +5,41 @@
55

66
[partintro]
77
--
8-
Elasticsearch indices can require a significant amount of memory available in order to be open and searchable. Yet, not all indices need
9-
to be writable at the same time and have different access patterns over time. For example, indices in the time series or logging use cases
10-
are unlikely to be queried once they age out but still need to be kept around for retention policy purposes.
11-
12-
In order to keep indices available and queryable for a longer period but at the same time reduce their hardware requirements they can be transitioned
13-
into a frozen state. Once an index is frozen, all of its transient shard memory (aside from mappings and analyzers)
14-
is moved to persistent storage. This allows for a much higher disk to heap storage ratio on individual nodes. Once an index is
15-
frozen, it is made read-only and drops its transient data structures from memory. These data structures will need to be reloaded on demand (and subsequently dropped) for each search request that targets the frozen index. A search request that hits
16-
one or more frozen shards will be executed on a throttled threadpool that ensures that we never search more than
17-
`N` (`1` by default) searches concurrently (see <<search-throttled>>). This protects nodes from exceeding the available memory due to incoming search requests.
18-
19-
In contrast to ordinary open indices, frozen indices are expected to execute slowly and are not designed for high query load. Parallelism is
20-
gained only on a per-node level and loading data-structures on demand is expected to be one or more orders of a magnitude slower than query
21-
execution on a per shard level. Depending on the data in an index, a frozen index may execute searches in the seconds to minutes range, when the same index in an unfrozen state may execute the same search request in milliseconds.
8+
{es} indices keep some data structures in memory to allow you to search them
9+
efficiently and to index into them. If you have a lot of indices then the
10+
memory required for these data structures can add up to a significant amount.
11+
For indices that are searched frequently it is better to keep these structures
12+
in memory because it takes time to rebuild them. However, you might access some
13+
of your indices so rarely that you would prefer to release the corresponding
14+
memory and rebuild these data structures on each search.
15+
16+
For example, if you are using time-based indices to store log messages or time
17+
series data then it is likely that older indices are searched much less often
18+
than the more recent ones. Older indices also receive no indexing requests.
19+
Furthermore, it is usually the case that searches of older indices are for
20+
performing longer-term analyses for which a slower response is acceptable.
21+
22+
If you have such indices then they are good candidates for becoming _frozen
23+
indices_. {es} builds the transient data structures of each shard of a frozen
24+
index each time that shard is searched, and discards these data structures as
25+
soon as the search is complete. Because {es} does not maintain these transient
26+
data structures in memory, frozen indices consume much less heap than normal
27+
indices. This allows for a much higher disk-to-heap ratio than would otherwise
28+
be possible.
29+
30+
Searches performed on frozen indices use the small, dedicated,
31+
<<search-throttled,`search_throttled` threadpool>> to control the number of
32+
concurrent searches that hit frozen shards on each node. This limits the amount
33+
of extra memory required for the transient data structures corresponding to
34+
frozen shards, which consequently protects nodes against excessive memory
35+
consumption.
36+
37+
Frozen indices are read-only: you cannot index into them.
38+
39+
Searches on frozen indices are expected to execute slowly. Frozen indices are
40+
not intended for high search load. It is possible that a search of a frozen
41+
index may take seconds or minutes to complete, even if the same searches
42+
completed in milliseconds when the indices were not frozen.
2243
--
2344

2445
== Best Practices

0 commit comments

Comments
 (0)