Skip to content

network: stateless vote compression #6276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 67 commits into from
May 7, 2025
Merged

Conversation

cce
Copy link
Contributor

@cce cce commented Mar 14, 2025

Summary

Similar to #4589 this provides single-message vote compression without any shared context between messages. It removes msgpack control characters and field names, reducing the size of votes down to just the values present, plus a 2-byte header. It provides 25% smaller messages on typical votes, compared to 6-7% smaller for zstd, and is >20x faster than zstd's fastest levels.

Test Plan

  • Adds options to protocol.RandomizeObject to use for testing encoding/decoding random and 10% empty values (used from vpack_test.go).
  • Also uses rapid to generate random votes, wiring them up to regular tests and Go fuzz tests.
  • Adds test hook to agreement to check all invocations of makeVote when run during tests
  • Adds features to algodump for benchmarking and testing against network data (will submit as separate PR)
  • Some testing based on plan from network: proposal payload compression #4589:
    • Unit test checking old + new combinations
    • Manual TwoNodes50Each test with master (m) + feature (f)
    • Manual ThreeNodesEvenDist test with [m]m+f, m+f[f] where [r] is a relay node
    • Manual ThreeNodesEvenDist test with relay running feature and other two nodes on master
    • Perf test

Copy link

codecov bot commented Mar 14, 2025

Codecov Report

Attention: Patch coverage is 94.03579% with 30 lines in your changes missing coverage. Please review.

Project coverage is 51.84%. Comparing base (039c933) to head (245cfec).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
protocol/codec_tester.go 26.31% 25 Missing and 3 partials ⚠️
network/vpack/vpack.go 98.83% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6276      +/-   ##
==========================================
+ Coverage   51.65%   51.84%   +0.19%     
==========================================
  Files         649      652       +3     
  Lines       86964    87433     +469     
==========================================
+ Hits        44918    45331     +413     
- Misses      39192    39236      +44     
- Partials     2854     2866      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cce cce force-pushed the vote-compression branch from 9cbd6a4 to 636d418 Compare March 14, 2025 19:11
Copy link
Contributor

@jannotti jannotti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple questions so far.

@gmalouf gmalouf self-requested a review March 14, 2025 21:15
@cce cce force-pushed the vote-compression branch from ec7e78c to 937f440 Compare March 17, 2025 15:10
@cce cce force-pushed the vote-compression branch from 6e4e24b to 79bb378 Compare March 18, 2025 02:32
@cce cce force-pushed the vote-compression branch from 79bb378 to 6e88ce1 Compare March 18, 2025 16:28
@cce cce marked this pull request as ready for review March 18, 2025 16:29
jannotti
jannotti previously approved these changes May 2, 2025
@cce cce dismissed stale reviews from jannotti and gmalouf via 9dfdc74 May 2, 2025 19:54
@cce cce force-pushed the vote-compression branch from 9dfdc74 to 58fa026 Compare May 2, 2025 20:02
jannotti
jannotti previously approved these changes May 6, 2025
gmalouf
gmalouf previously approved these changes May 6, 2025
algorandskiy
algorandskiy previously approved these changes May 6, 2025
@cce cce dismissed stale reviews from algorandskiy, gmalouf, and jannotti via 6bdbfde May 6, 2025 19:10
@algorandskiy algorandskiy requested review from jannotti and gmalouf May 6, 2025 20:39
@cce cce merged commit 8521387 into algorand:master May 7, 2025
19 checks passed
@cce cce deleted the vote-compression branch May 7, 2025 00:44
cce added a commit to cce/go-algorand that referenced this pull request May 29, 2025
@cce cce mentioned this pull request May 30, 2025
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants