Skip to content

Increase time out on build to 45 #1598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 11, 2018
Merged

Increase time out on build to 45 #1598

merged 1 commit into from
Nov 11, 2018

Conversation

shauheen
Copy link
Contributor

I'm seeing builds getting cancelled at 40 minutes which might have completed otherwise. Increasing timeout to 45 minutes, in #1561 it was suggested we should bump this as well.

@Ivanidzo4ka
Copy link
Contributor

@shauheen do you have example of such build? If you look on build for this PR it's all done in 20 minutes.

@shauheen
Copy link
Contributor Author

@Ivanidzo4ka all of the builds on master have failed since yesterday because one leg is cancelled at 40 minutes.

@shauheen shauheen requested a review from abgoswam November 11, 2018 00:01
@@ -11,7 +11,7 @@ phases:
_phaseName: ${{ parameters.name }}
_arch: ${{ parameters.architecture }}
queue:
timeoutInMinutes: 40
timeoutInMinutes: 45
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why move from 40 to only 45? If it's dying at 40, do we think the extra 12.5% allowed runtime will be enough?

How about we first set a higher limit then see how much time it will take:

Suggested change
timeoutInMinutes: 45
timeoutInMinutes: 60

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to be 60 before, we can change it again if needed, but the ones I looked at yesterday were almost finished before cancelling.

Copy link
Contributor

@justinormont justinormont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know which steps are taking the most time? Do we have a plan to speed them up?

In @shauheen's link the overall time for the OSX Debug build/test was ~2x longer than the debug build on Linux/Windows.

I did note that we may want longer timeout to first find the right limit, then later we can reduce it.

@shauheen shauheen merged commit c31c2c1 into master Nov 11, 2018
@shauheen shauheen deleted the build-patch-1 branch November 11, 2018 02:16
@Ivanidzo4ka
Copy link
Contributor

@shauheen lets look in the log:

2018-11-10T19:59:31.5893620Z Finished test: Microsoft.ML.Tests.Scenarios.Api.CookbookSamples.CookbookSamples.Normalization
2018-11-10T20:29:08.6710070Z ##[error]The operation was canceled.

It didn't fail because it was working and didn't had time, it fail because it got stuck and got terminated during timeout.

As I said, in successful build you have about 20 minutes of execution, by increasing timeout, you only make situation worse, I would rather change timeout to 30 or maybe 25 minutes, than increasing it. We just block machines in the pool for longer period of time.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants