Skip to content
This repository was archived by the owner on Apr 14, 2022. It is now read-only.

BFS executor: reduce memory usage #236

Closed
Totktonada opened this issue Sep 17, 2018 · 1 comment
Closed

BFS executor: reduce memory usage #236

Totktonada opened this issue Sep 17, 2018 · 1 comment
Labels

Comments

@Totktonada
Copy link
Member

Totktonada commented Sep 17, 2018

We need to collect memory usage data on our benchmarking test suite and run at least one separate test with many unrelated data within the spaces (say, increase each space size from 100 to 10^6).

There are several ways to reduce memory usage. Both handle some part of the entire problem, so I think it worth to implement both.

Reorder execution

When connected collection filter is involved we just create cache only requests for the entire set of tuples we need to check against the filter. But we can fetch block of tuples from the connected collection, then invoke resolvers for the block of upper collection tuples and free resources.

Possible problem: when a filter is used and then the connection is in resulting fields of a query, we'll fetch tuples for resulting fields again after the resources will be freed.

Apply filters before save to cache

We need to transform object filters to tuple filters, add then to the cache key and apply before save a tuple.

The problem is that this will not help with tuples for the upper collection (and, more general, with all collections except deepmost). This should be handled using executor reordering.

Possible follow up: push down execution of filters to storages: #18.

@Totktonada
Copy link
Member Author

I'm going to archive the repository. I'll proceed as follows:

  • Mark all open pull requests with the archived label and close.
  • Mark all open issues with archived and wontfix labels and close.
  • Archive the repository.

Consider the following projects:

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant