This repository was archived by the owner on Sep 11, 2020. It is now read-only.
plumbing: use seen map in tree walker
#564
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
strib
force-pushed
the
tree-walker-use-seen-cache
branch
from
498dc32 to
52b25c7
Compare
Codecov Report
@@ Coverage Diff @@
## master #564 +/- ##
==========================================
- Coverage 78% 77.38% -0.62%
==========================================
Files 129 129
Lines 9842 9845 +3
==========================================
- Hits 7677 7619 -58
- Misses 1327 1400 +73
+ Partials 838 826 -12
Continue to review full report at Codecov.
|
strib
commented
Aug 25, 2017
| @@ -297,9 +297,10 @@ func (iter *treeEntryIter) Next() (TreeEntry, error) { | |||
|
|
|||
| // TreeWalker provides a means of walking through all of the entries in a Tree. | |||
| type TreeWalker struct { | |||
| stack []treeEntryIter | |||
| stack []*treeEntryIter | |||
There was a problem hiding this comment.
This seems safer to me; I was worried things that accessed the stack might end up calling Next on a copy, rather than the real object. But the way the stack is used now, I don't think it's a real bug. If you want me to revert it, let me know.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 25, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 26, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
erizocosmico
suggested changes
Aug 27, 2017
This helps avoids iterating down the same trees for every commit. For a big-ish repo with 35K objects (17K commits), this reduced the time for calling `revlist.Objects` during a push (with 0 hashes to ignore) from more than ten minutes to less than a minute.
strib
force-pushed
the
tree-walker-use-seen-cache
branch
from
52b25c7 to
d68f45f
Compare
|
Thanks for looking @erizocosmico. Now there's a simple test. |
erizocosmico
approved these changes
Aug 28, 2017
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 28, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
strib
added a commit
to keybase/kbfs
that referenced
this pull request
Aug 28, 2017
Compared to the v4 branch we were using before, master branch seems to fix a few bugs and bring the time for pushing the KBFS repo from hours to about 25 minutes. With the following optimizations, I got that time down to less than 3 minutes. * src-d/go-git#564 * src-d/go-git#565 I suspect there are still lots more optimizations that can be done. I don't think there are many users of Push in this project.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This helps avoid iterating down the same trees for every commit. For a big-ish repo with 35K objects (17K commits), this reduced the time for calling
revlist.Objectsduring a push (with 0 hashes to ignore) from more than ten minutes to less than a minute.