Skip to content

cmd/dist: avoid CPU underutilization starting from GOMAXPROCS=2 runtime #65703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

qiulaidongfeng
Copy link
Member

@qiulaidongfeng qiulaidongfeng commented Feb 14, 2024

This CL is doing now is:
change maxbg to increase test parallelism.
adjust test sequence.

This CL speeds up the go tool dist test,
most of the speed up is due to the fact that the
three time-consuming tests
cmd/internal/testdir and API check and runtime/race
can be done in parallel with the GOMAXPROCS=2 runtime
on a machine with enough CPU cores.

In windows with an 8-core 16-thread CPU,
this CL can complete all other tests before
GOMAXPROCS=2 runtime -cpu=1,2,4 -quick completes.

Fixes #65164

@gopherbot
Copy link
Contributor

This PR (HEAD: 5624a70) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 1:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 1: Run-TryBot+1

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 1:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 1:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: cc5f615) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 1: TryBot-Result-1

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 2: Run-TryBot+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 2:

(2 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 2: TryBot-Result+1

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 2: -Run-TryBot

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 2:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 3:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 3:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: 94e7973) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 4: Commit-Queue+1

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 4:

Dry run: CV is trying the patch.

Bot data: {"action":"start","triggered_at":"2024-03-02T04:34:59Z","revision":"1d0ca531607d0036919c2c36687d154a1b899281"}


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 4: -Commit-Queue


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 4:

This CL has passed the run


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 4: LUCI-TryBot-Result+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 4:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 4:

(3 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: d1db60f) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 5: Commit-Queue+1

(3 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 5:

Dry run: CV is trying the patch.

Bot data: {"action":"start","triggered_at":"2024-03-13T09:37:06Z","revision":"ecdffe1031022dc6f7f389dfedefee5ab3f9b8fb"}


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 5: -Commit-Queue


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 5:

This CL has passed the run


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 5: LUCI-TryBot-Result+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 5:

(3 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: a9f5193) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 6: Commit-Queue+1

(3 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 6:

Dry run: CV is trying the patch.

Bot data: {"action":"start","triggered_at":"2024-03-14T13:30:00Z","revision":"02d314284b962c1c34dccaaa11ecbb371ab70fd0"}


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 6: -Commit-Queue


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 6:

This CL has passed the run


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 6: LUCI-TryBot-Result+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 6:

(2 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

Fixes golang#65164

Change-Id: Ia10952f50a3c2a7868e30dcdba333927947b3ac3
n
Change-Id: If8d2d4289ea3c0f951018115cb5652d6f0f7a43f
n
Change-Id: I8712b8b29108162ac7bb0f76b32b0c8f81259833
n
Change-Id: Ib83ccb39a0ea866a8baa82aee1741331b1a082e4
n
Change-Id: I15be25455bff2d364a6910a61beeb5131eebd067
@gopherbot
Copy link
Contributor

This PR (HEAD: 18cffb7) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/go/+/563916.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 7: Commit-Queue+1

(3 comments)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 7:

Dry run: CV is trying the patch.

Bot data: {"action":"start","triggered_at":"2024-03-16T03:44:29Z","revision":"ecc48dd43df6bfabe77f29e1be351ef27f161be0"}


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 7: -Commit-Queue


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 7:

This CL has passed the run


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Go LUCI:

Patch Set 7: LUCI-TryBot-Result+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 7:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 7: Code-Review+2

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Keith Randall:

Patch Set 7: Code-Review+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from David Chase:

Patch Set 7: Code-Review+1


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 7:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

@qiulaidongfeng qiulaidongfeng changed the title cmd/dist: increase test parallelism starting with GOMAXPROCS=2 runtime cmd/dist: avoid CPU underutilization starting from GOMAXPROCS=2 runtime Mar 23, 2024
@gopherbot
Copy link
Contributor

Message from qiulaidongfeng:

Patch Set 8:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/563916.
After addressing review feedback, remember to publish your drafts!

gopherbot pushed a commit that referenced this pull request Mar 23, 2024
This CL is doing now is:
change maxbg to increase test parallelism.
adjust test sequence.

This CL speeds up the go tool dist test,
most of the speed up is due to the fact that the
three time-consuming tests
cmd/internal/testdir and API check and runtime/race
can be done in parallel with the GOMAXPROCS=2 runtime
on a machine with enough CPU cores.

In windows with an 8-core 16-thread CPU,
this CL can complete all other tests before
GOMAXPROCS=2 runtime -cpu=1,2,4 -quick completes.

Fixes #65164

Change-Id: I56ed7031d58be3bece9f975bfc73e5c834d0a4fa
GitHub-Last-Rev: 18cffb7
GitHub-Pull-Request: #65703
Reviewed-on: https://go-review.googlesource.com/c/go/+/563916
Reviewed-by: Keith Randall <[email protected]>
Reviewed-by: Keith Randall <[email protected]>
Commit-Queue: Ian Lance Taylor <[email protected]>
Auto-Submit: Ian Lance Taylor <[email protected]>
Reviewed-by: David Chase <[email protected]>
LUCI-TryBot-Result: Go LUCI <[email protected]>
@gopherbot
Copy link
Contributor

This PR is being closed because golang.org/cl/563916 has been merged.

@gopherbot gopherbot closed this Mar 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cmd/dist: GOMAXPROCS=2 runtime -cpu=1,2,4 -quick makes poor use of CPU resources, increasing all.bat latency
2 participants