Skip to content

Adding Benchmark test for Checksums #5701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Nov 29, 2024
Merged

Adding Benchmark test for Checksums #5701

merged 8 commits into from
Nov 29, 2024

Conversation

joviegas
Copy link
Contributor

@joviegas joviegas commented Nov 9, 2024 •
edited
Loading

Motivation and Context

  • Need to benchmark the different checksums

Modifications

  • Added new Test cases

Testing

updateEntireByteArrayChecksum (Java8)

@Warmup(iterations = 3, time = 15, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 10, timeUnit = TimeUnit.SECONDS)
@Fork(2)
Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 54.359 0.149
CRC32C (Crt) 128B 168.957 12.198
CRC64NVME 128B 238.858 3.05
MD5 128B 656.694 9.134
SHA1 128B 948.87 6.687
SHA256 128B 1407.35 15.987
CRC32 1MB 63937.4 311.145
CRC32C 1MB 142089 5188.44
CRC64NVME 1MB 779647 6751.18
MD5 1MB 3.23534e+06 1510.63
SHA1 1MB 4.81596e+06 6461.77
SHA256 1MB 7.17205e+06 60232.7
CRC32 4KB 327.245 37.799
CRC32C 4KB 553.963 25.966
CRC64NVME 4KB 3191.97 27.827
MD5 4KB 13246.6 10.639
SHA1 4KB 19684.2 26.322
SHA256 4KB 28974.7 124.396

updateIndividualByteChecksumOneByteATime (Java8)

Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 503.246 0.383
CRC32C (CRT) 128B 19749.9 4416.36
CRC64NVME 128B 17260.7 78.228
MD5 128B 2821.84 97.619
SHA1 128B 3187.29 140.14
SHA256 128B 3642.74 172.055
CRC32 1MB 3.8796e+06 1601.41
CRC32C 1MB 1.35245e+08 695891
CRC64NVME 1MB 1.35547e+08 373876
MD5 1MB 1.80819e+07 16197.2
SHA1 1MB 1.97299e+07 30513.9
SHA256 1MB 2.21067e+07 119922
CRC32 4KB 15508.9 21.904
CRC32C 4KB 539955 7495.23
CRC64NVME 4KB 545065 4185.17
MD5 4KB 83687 190.44
SHA1 4KB 90607.2 463.367
SHA256 4KB 97773.3 9369.09 
@Warmup(iterations = 3, time = 3, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 5, timeUnit = TimeUnit.SECONDS)
@Fork(1)

updateEntireByteArrayChecksum (Java8)

Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 108.077 0.748
CRC32C (CRT) 128B 161.075 0.321
CRC64NVME 128B 226.943 0.964
MD5 128B 659.158 2.358
SHA1 128B 954.494 18.984
SHA256 128B 1406 8.164
CRC32 1MB 64046.1 122.424
CRC32C 1MB 145312 799.353
CRC64NVME 1MB 776510 3767.16
MD5 1MB 3.234e+06 3911.2
SHA1 1MB 4.80487e+06 21984
SHA256 1MB 7.11034e+06 24152.7
CRC32 4KB 306.098 26.504
CRC32C 4KB 529.352 2.096
CRC64NVME 4KB 3189.51 60.239
MD5 4KB 13177.7 8.595
SHA1 4KB 19576.1 41.6
SHA256 4KB 28992.4 48.096

updateIndividualByteChecksumOneByteATime (Java8)

Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 503.304 0.966
CRC32C (CRT) 128B 16636.7 168.581
CRC64NVME 128B 16931.1 105.514
MD5 128B 2843.91 51.346
SHA1 128B 3255.72 234.994
SHA256 128B 3422.36 36.194
CRC32 1MB 3.88485e+06 3175.98
CRC32C 1MB 1.35303e+08 1.05541e+06
CRC64NVME 1MB 1.3671e+08 4.81374e+06
MD5 1MB 1.81059e+07 8601.73
SHA1 1MB 1.98254e+07 79692.5
SHA256 1MB 2.20228e+07 57341.4
CRC32 4KB 15571.7 8.456
CRC32C 4KB 521963 3788.89
CRC64NVME 4KB 539288 9694.32
MD5 4KB 84142.6 342.75
SHA1 4KB 83252.4 458.621
SHA256 4KB 93326.5 5752.15

Java 17

# JMH version: 1.37
# VM version: JDK 17.0.13, OpenJDK 64-Bit Server VM, 17.0.13+11-LTS

updateEntireByteArrayChecksum (Java17)

Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 78.845 44.233
CRC32C (Java9) 128B 21.478 3.463
CRC64NVME 128B 251.169 6.687
MD5 128B 391.578 1.532
SHA1 128B 832.668 4.879
SHA256 128B 833.296 4.806
CRC32 1MB 63977.6 167.208
CRC32C (Java9) 1MB 51711.2 158.02
CRC64NVME 1MB 773987 5784.79
MD5 1MB 1.81246e+06 495.827
SHA1 1MB 4.08432e+06 5031.87
SHA256 1MB 3.31528e+06 5846.02
CRC32 4KB 317.79 36.703
CRC32C (Java9) 4KB 225.776 0.307
CRC64NVME 4KB 3138.58 26.076
MD5 4KB 7427.89 5.941
SHA1 4KB 16643.6 32.817
SHA256 4KB 13791.9 35.488

updateIndividualByteChecksumOneByteATime (Java17)

Checksum Provider Size Score (ns/op) Error (ns/op)
CRC32 128B 485.281 0.289
CRC32C (Java9) 128B 376.345 0.49
CRC64NVME 128B 19788.5 164.965
MD5 128B 2276.97 15.291
SHA1 128B 3032.28 226.671
SHA256 128B 2769.88 43.214
CRC32 1MB 3.87965e+06 2500.93
CRC32C (Java9) 1MB 3.08291e+06 504.252
CRC64NVME 1MB 1.57607e+08 297691
MD5 1MB 1.78764e+07 179643
SHA1 1MB 2.09347e+07 34080.5
SHA256 1MB 1.99525e+07 196255
CRC32 4KB 15499 4.473
CRC32C (Java9) 4KB 12326 18.145
CRC64NVME 4KB 630964 9526.24
MD5 4KB 66340.3 1758.66
SHA1 4KB 78801.3 386.979
SHA256 4KB 76082.7 1249.44

License

  • I confirm that this pull request can be released under the Apache 2 license

@joviegas joviegas requested a review from a team as a code owner November 9, 2024 03:21
zoewangg
zoewangg reviewed Nov 18, 2024
View reviewed changes
...dk-benchmarks/src/main/java/software/amazon/awssdk/benchmark/checksum/CheckSumBenchmark.java Outdated

@Benchmark
public void updateEntireByteArrayChecksum(ChecksumState state, Blackhole blackhole) {
state.sdkChecksum.reset(); // Ensure we reset the checksum before each run
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do reset in tearDown methond @TearDown?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

...dk-benchmarks/src/main/java/software/amazon/awssdk/benchmark/checksum/CheckSumBenchmark.java Outdated
import software.amazon.awssdk.checksums.SdkChecksum;

@State(Scope.Benchmark)
@Warmup(iterations = 3, time = 3, timeUnit = TimeUnit.SECONDS)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is 3 seconds sufficient for warmup? Is the result consistent after warming up?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did test with both 3 and 15 , the relative Score is same .
However changed it to 15 to be consistent with other test

zoewangg
zoewangg reviewed Nov 25, 2024
View reviewed changes
...dk-benchmarks/src/main/java/software/amazon/awssdk/benchmark/checksum/ChecksumBenchmark.java Outdated

@Benchmark
public void updateIndividualByteChecksumOneByteATime(ChecksumState state, Blackhole blackhole) {
state.sdkChecksum.reset();// Ensure we reset the checksum before each run
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since it's done in tearDown, I don't think we need this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@joviegas joviegas enabled auto-merge November 29, 2024 19:05
@sonarqubecloud SonarQubeCloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@joviegas joviegas added this pull request to the merge queue Nov 29, 2024
Merged via the queue into master with commit 47fcc27 Nov 29, 2024
16 of 17 checks passed
@joviegas joviegas deleted the joviegas/benchmarks branch January 15, 2025 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zoewangg zoewangg zoewangg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@joviegas @zoewangg