Skip to content

Shortcut aggs for TSDB #90423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tmgordeeva opened this issue Sep 27, 2022 · 3 comments
Closed

Shortcut aggs for TSDB #90423

tmgordeeva opened this issue Sep 27, 2022 · 3 comments
Labels
>enhancement :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@tmgordeeva
Copy link
Contributor

Description

Our goal is to enable efficient pipeline aggregations on TSDB by taking
advantage of TSDB data distribution. Data nodes could process queries
themselves, but so far there hasn't been a reason to do this because of
the equal distribution of shards among nodes. With TSDB, we can take
advantage of the data distribution.

Initial steps would be to enable processing on data nodes to begin with:

  • Add ability for data nodes to propagate requests back up
    • Push down request to data node
    • Suppress response from coordinating node in favor of data nodes
  • Test data node propagation with a trivial agg (ie no merging logic required)

The point of pushing requests down to data nodes is if we have all the data in
a bucket like in TSDB, we can optimize by processing that bucket entirely on
that data node. So the next steps would be something like:

  • Enable processing whole buckets on data nodes and propagating the request up
  • Data nodes with incomplete buckets must return up to the coordinating node
    • The request will then have to be processed on multiple nodes as usual
@tmgordeeva tmgordeeva self-assigned this Sep 27, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 27, 2022
@weizijun
Copy link
Contributor

hi, @tmgordeeva , is the issue #85798 can solve the problem?

@martijnvg
Copy link
Member

At this time we're not going to focus on the time series support in aggregations.
Closing this for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Development

No branches or pull requests

4 participants