Skip to content
This repository was archived by the owner on Jan 28, 2021. It is now read-only.

Index generation is slower than previous implementation #676

Closed
ajnavarro opened this issue Apr 16, 2019 · 5 comments · Fixed by #681
Closed

Index generation is slower than previous implementation #676

ajnavarro opened this issue Apr 16, 2019 · 5 comments · Fixed by #681
Assignees
Labels
bug Something isn't working

Comments

@ajnavarro
Copy link
Contributor

Log indexing a table with the actual version:

DEBU[0461] finished pilosa indexing                      duration=5m26.700323786s id=blobs_idx mapping=508129748 pilosa=384190894 rows=55764

Log with previous implementation, the same table:

DEBU[0022] finished pilosa indexing                      duration=5.323093051s id=blobs_idx mapping=429.571981ms pilosa=268.452932ms rows=55796

5 mins vs 5 secs.

@ajnavarro ajnavarro added the bug Something isn't working label Apr 16, 2019
@kuba-- kuba-- self-assigned this Apr 16, 2019
@kuba--
Copy link
Contributor

kuba-- commented Apr 16, 2019

Screen Shot 2019-04-16 at 14 39 50

@kuba--
Copy link
Contributor

kuba-- commented Apr 16, 2019

After creating mapping per partition we get better performance:
Screen Shot 2019-04-16 at 19 21 25

@kuba--
Copy link
Contributor

kuba-- commented Apr 16, 2019

you can see that bbolt.Commit dropped from 3.39s to 0.06s

@kuba--
Copy link
Contributor

kuba-- commented Apr 16, 2019

The fix will break backward compatibility, because we'll require mapping file per partition.

@erizocosmico
Copy link
Contributor

It's a 0.x, so I guess it's fine to break compat

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants