Skip to content

Improve memory usage during map build #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 27, 2021

Conversation

duncanharris
Copy link
Contributor

Speed up build for large maps by allocating in chunks and then copying over to one slice at the end. This generates less garbage and involves less copying than the previous strategy of using append.

@duncanharris duncanharris self-assigned this Jul 25, 2021
Speed up build for large maps by allocating in chunks and
then copying over to one slice at the end. This generates
less garbage and involves less copying than the previous
strategy of using append.
@duncanharris duncanharris force-pushed the duncan/improve-build-memory-usage branch from 4c66c93 to a9b6916 Compare July 27, 2021 08:53
@duncanharris
Copy link
Contributor Author

duncanharris commented Jul 27, 2021

Using:

func BenchmarkNewUint32Store(b *testing.B) {
	b.ReportAllocs()
	m := typicalCodeStrings(200000)
	b.ResetTimer()
	for bi := 0; bi < b.N; bi++ {
		_ = faststringmap.NewUint32Store(m)
	}
}

I get:

Old:
76222743 ns/op	17807685 B/op	      33 allocs/op
New:
69234532 ns/op	 8749168 B/op	       6 allocs/op

So significantly less memory usage but not much of a performance improvement until we to get to very large maps.

Copy link
Contributor

@phynes-sensiblecode phynes-sensiblecode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM- there is a clear memory usage improvement.

@phynes-sensiblecode phynes-sensiblecode merged commit 99e4718 into main Jul 27, 2021
@phynes-sensiblecode phynes-sensiblecode deleted the duncan/improve-build-memory-usage branch July 27, 2021 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants