Small speed up of date_histogram with children #67012
Conversation
elasticmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
This allows us to run the optimization introduced in elastic#63643 when the `date_histogram` has children. It isn't a revolutionary performance improvement though because children tend to be a lot heavier than the `date_histogram`. It is faster, but only by a couple of percentage points.
| SubCollector collector = new SubCollector(filterOrd, filterLeafCollector); | ||
| scorer.score(collector, live); | ||
| incrementBucketDocCount(filterOrd, collector.total); | ||
| } |
There was a problem hiding this comment.
I think it's possible that this could actually be slower than the standard execution mechanism. I wonder if we need an escape hatch so folks can dodge this mechanism if it proves a bad idea.
Also: there is another possible implementation here that involves collecting a block of matches for each filter and then running all of the children in parallel. I'm not sure if it'll be faster or not. It kind of depends on the speed of iterating the doc values. It is a little more complex so I didn't do it.
There was a problem hiding this comment.
After taking a couple of days away I think the block matching mechanism is probably better here. Its much simpler to estimate the cost. Also - I'd love to know why the old way is so slow - the block based mechanism feels like it'd be fast and it reads quite similarly to the Compatible mechanism. I think the big difference is that we don't get to join the main query with the filter query. So it can't skip matches effectively. Maybe. I've got to play.
|
The test failure may be related. Fun |
|
The build failure found a bug that I'd like to land the fix for into 7.11: #67043. |
| TotalHitCountCollector collector = new TotalHitCountCollector(); | ||
| scorer.score(collector, live); | ||
| incrementBucketDocCount(filterOrd, collector.getTotalHits()); | ||
| if (sub == LeafBucketCollector.NO_OP_COLLECTOR) { |
There was a problem hiding this comment.
The equality check feels brittle to me here. I wonder if we should put a method on LeafBucketCollector to return a boolean if it's going to do any work, and check that. Might be premature abstraction on my part though, what do you think?
Sorry about that! When I first opened it I was more comfortable with it and then I kind of went backwards on that..... |
nik9000
requested review from
not-napoleon
and removed request for
polyfractal and
not-napoleon
|
@elasticmachine update branch |
|
The master branch is quite different now. I'm going to try and revive this against master and reopen with the new block algorithm. |
This allows us to run the optimization introduced in #63643 when the
date_histogramhas children. It isn't a revolutionary performanceimprovement though because children tend to be a lot heavier than the
date_histogram. It is faster, but only by a couple of percentagepoints.
Mostly the idea is that we can build on this to do other optimization.