Skip to content

Commit ce1aaec

Browse files
jrodewigabprash
andauthored
[DOCS] Remove approximate document counts example from term agg docs (elastic#55442) (elastic#56043)
Removes an example from the "Document counts are approximate" section of the terms agg documentation. As elastic#52377 details, the example was no longer accurate in 7.x or 6.8. Document counts were more precise than the example presented. We've opened issue elastic#56025 to discuss re-adding an example later. Co-authored-by: James Rodewig <[email protected]> Co-authored-by: AB Prashanth <[email protected]>
1 parent 5acb7b3 commit ce1aaec

File tree

1 file changed

+13
-147
lines changed

1 file changed

+13
-147
lines changed

docs/reference/aggregations/bucket/terms-aggregation.asciidoc

Lines changed: 13 additions & 147 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,8 @@ GET /_search
6868
--------------------------------------------------
6969
// CONSOLE
7070
// TEST[s/_search/_search\?filter_path=aggregations/]
71-
<1> `terms` aggregation should be a field of type `keyword` or any other data type suitable for bucket aggregations. In order to use it with `text` you will need to enable
71+
72+
<1> `terms` aggregation should be a field of type `keyword` or any other data type suitable for bucket aggregations. In order to use it with `text` you will need to enable
7273
<<fielddata, fielddata>>.
7374

7475
Response:
@@ -125,85 +126,10 @@ NOTE: If you want to retrieve **all** terms or all combinations of terms in a ne
125126
[[search-aggregations-bucket-terms-aggregation-approximate-counts]]
126127
==== Document counts are approximate
127128

128-
As described above, the document counts (and the results of any sub aggregations) in the terms aggregation are not always
129-
accurate. This is because each shard provides its own view of what the ordered list of terms should be and these are
130-
combined to give a final view. Consider the following scenario:
131-
132-
A request is made to obtain the top 5 terms in the field product, ordered by descending document count from an index with
133-
3 shards. In this case each shard is asked to give its top 5 terms.
134-
135-
[source,js]
136-
--------------------------------------------------
137-
GET /_search
138-
{
139-
"aggs" : {
140-
"products" : {
141-
"terms" : {
142-
"field" : "product",
143-
"size" : 5
144-
}
145-
}
146-
}
147-
}
148-
--------------------------------------------------
149-
// CONSOLE
150-
// TEST[s/_search/_search\?filter_path=aggregations/]
151-
152-
The terms for each of the three shards are shown below with their
153-
respective document counts in brackets:
154-
155-
[width="100%",cols="^2,^2,^2,^2",options="header"]
156-
|=========================================================
157-
| | Shard A | Shard B | Shard C
158-
159-
| 1 | Product A (25) | Product A (30) | Product A (45)
160-
| 2 | Product B (18) | Product B (25) | Product C (44)
161-
| 3 | Product C (6) | Product F (17) | Product Z (36)
162-
| 4 | Product D (3) | Product Z (16) | Product G (30)
163-
| 5 | Product E (2) | Product G (15) | Product E (29)
164-
| 6 | Product F (2) | Product H (14) | Product H (28)
165-
| 7 | Product G (2) | Product I (10) | Product Q (2)
166-
| 8 | Product H (2) | Product Q (6) | Product D (1)
167-
| 9 | Product I (1) | Product J (6) |
168-
| 10 | Product J (1) | Product C (4) |
169-
170-
|=========================================================
171-
172-
The shards will return their top 5 terms so the results from the shards will be:
173-
174-
[width="100%",cols="^2,^2,^2,^2",options="header"]
175-
|=========================================================
176-
| | Shard A | Shard B | Shard C
177-
178-
| 1 | Product A (25) | Product A (30) | Product A (45)
179-
| 2 | Product B (18) | Product B (25) | Product C (44)
180-
| 3 | Product C (6) | Product F (17) | Product Z (36)
181-
| 4 | Product D (3) | Product Z (16) | Product G (30)
182-
| 5 | Product E (2) | Product G (15) | Product E (29)
183-
184-
|=========================================================
185-
186-
Taking the top 5 results from each of the shards (as requested) and combining them to make a final top 5 list produces
187-
the following:
188-
189-
[width="40%",cols="^2,^2"]
190-
|=========================================================
191-
192-
| 1 | Product A (100)
193-
| 2 | Product Z (52)
194-
| 3 | Product C (50)
195-
| 4 | Product G (45)
196-
| 5 | Product B (43)
197-
198-
|=========================================================
199-
200-
Because Product A was returned from all shards we know that its document count value is accurate. Product C was only
201-
returned by shards A and C so its document count is shown as 50 but this is not an accurate count. Product C exists on
202-
shard B, but its count of 4 was not high enough to put Product C into the top 5 list for that shard. Product Z was also
203-
returned only by 2 shards but the third shard does not contain the term. There is no way of knowing, at the point of
204-
combining the results to produce the final list of terms, that there is an error in the document count for Product C and
205-
not for Product Z. Product H has a document count of 44 across all 3 shards but was not included in the final list of
206-
terms because it did not make it into the top five terms on any of the shards.
129+
Document counts (and the results of any sub aggregations) in the terms
130+
aggregation are not always accurate. Each shard provides its own view of what
131+
the ordered list of terms should be. These views are combined to give a final
132+
view.
207133

208134
==== Shard Size
209135

@@ -228,35 +154,7 @@ The default `shard_size` is `(size * 1.5 + 10)`.
228154

229155
There are two error values which can be shown on the terms aggregation. The first gives a value for the aggregation as
230156
a whole which represents the maximum potential document count for a term which did not make it into the final list of
231-
terms. This is calculated as the sum of the document count from the last term returned from each shard. For the example
232-
given above the value would be 46 (2 + 15 + 29). This means that in the worst case scenario a term which was not returned
233-
could have the 4th highest document count.
234-
235-
[source,js]
236-
--------------------------------------------------
237-
{
238-
...
239-
"aggregations" : {
240-
"products" : {
241-
"doc_count_error_upper_bound" : 46,
242-
"sum_other_doc_count" : 79,
243-
"buckets" : [
244-
{
245-
"key" : "Product A",
246-
"doc_count" : 100
247-
},
248-
{
249-
"key" : "Product Z",
250-
"doc_count" : 52
251-
}
252-
...
253-
]
254-
}
255-
}
256-
}
257-
--------------------------------------------------
258-
// TESTRESPONSE[s/\.\.\.//]
259-
// TESTRESPONSE[s/: (\-)?[0-9]+/: $body.$_path/]
157+
terms. This is calculated as the sum of the document count from the last term returned from each shard.
260158

261159
==== Per bucket document count error
262160

@@ -283,39 +181,7 @@ GET /_search
283181

284182
This shows an error value for each term returned by the aggregation which represents the 'worst case' error in the document count
285183
and can be useful when deciding on a value for the `shard_size` parameter. This is calculated by summing the document counts for
286-
the last term returned by all shards which did not return the term. In the example above the error in the document count for Product C
287-
would be 15 as Shard B was the only shard not to return the term and the document count of the last term it did return was 15.
288-
The actual document count of Product C was 54 so the document count was only actually off by 4 even though the worst case was that
289-
it would be off by 15. Product A, however has an error of 0 for its document count, since every shard returned it we can be confident
290-
that the count returned is accurate.
291-
292-
[source,js]
293-
--------------------------------------------------
294-
{
295-
...
296-
"aggregations" : {
297-
"products" : {
298-
"doc_count_error_upper_bound" : 46,
299-
"sum_other_doc_count" : 79,
300-
"buckets" : [
301-
{
302-
"key" : "Product A",
303-
"doc_count" : 100,
304-
"doc_count_error_upper_bound" : 0
305-
},
306-
{
307-
"key" : "Product Z",
308-
"doc_count" : 52,
309-
"doc_count_error_upper_bound" : 2
310-
}
311-
...
312-
]
313-
}
314-
}
315-
}
316-
--------------------------------------------------
317-
// TESTRESPONSE[s/\.\.\.//]
318-
// TESTRESPONSE[s/: (\-)?[0-9]+/: $body.$_path/]
184+
the last term returned by all shards which did not return the term.
319185

320186
These errors can only be calculated in this way when the terms are ordered by descending document count. When the aggregation is
321187
ordered by the terms values themselves (either ascending or descending) there is no error in the document count since if a shard
@@ -689,7 +555,7 @@ GET /_search
689555

690556
===== Filtering Values with partitions
691557

692-
Sometimes there are too many unique terms to process in a single request/response pair so
558+
Sometimes there are too many unique terms to process in a single request/response pair so
693559
it can be useful to break the analysis up into multiple requests.
694560
This can be achieved by grouping the field's values into a number of partitions at query-time and processing
695561
only one partition in each request.
@@ -729,10 +595,10 @@ GET /_search
729595
This request is finding the last logged access date for a subset of customer accounts because we
730596
might want to expire some customer accounts who haven't been seen for a long while.
731597
The `num_partitions` setting has requested that the unique account_ids are organized evenly into twenty
732-
partitions (0 to 19). and the `partition` setting in this request filters to only consider account_ids falling
598+
partitions (0 to 19). and the `partition` setting in this request filters to only consider account_ids falling
733599
into partition 0. Subsequent requests should ask for partitions 1 then 2 etc to complete the expired-account analysis.
734600

735-
Note that the `size` setting for the number of results returned needs to be tuned with the `num_partitions`.
601+
Note that the `size` setting for the number of results returned needs to be tuned with the `num_partitions`.
736602
For this particular account-expiration example the process for balancing values for `size` and `num_partitions` would be as follows:
737603

738604
1. Use the `cardinality` aggregation to estimate the total number of unique account_id values
@@ -741,8 +607,8 @@ For this particular account-expiration example the process for balancing values
741607
4. Run a test request
742608

743609
If we have a circuit-breaker error we are trying to do too much in one request and must increase `num_partitions`.
744-
If the request was successful but the last account ID in the date-sorted test response was still an account we might want to
745-
expire then we may be missing accounts of interest and have set our numbers too low. We must either
610+
If the request was successful but the last account ID in the date-sorted test response was still an account we might want to
611+
expire then we may be missing accounts of interest and have set our numbers too low. We must either
746612

747613
* increase the `size` parameter to return more results per partition (could be heavy on memory) or
748614
* increase the `num_partitions` to consider less accounts per request (could increase overall processing time as we need to make more requests)

0 commit comments

Comments
 (0)