chore: Add retry to pipeline templates constructors to add retrier to each pipeline step #179
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fix build failures due to Sagemaker ThrottlingException when running pipeline integration tests
Fixes #(issue) - N/A
Why is the change necessary?
Recent build failures were due to Sagemaker ThrottlingException (Rate exceeded) during following tests:
Solution
Add an optional retry argument to the pipeline template constructors (InferencePipeline and TrainingPipeline) in order to add a retry strategy for each pipeline steps. The same retrier will be added for each step.
Caveat: This fix applies the retry strategy to all steps in the pipeline. The customer won't be able to customize the strategy for each step.
Alternate solution 1:
We could add the option for the client to customize retry strategies for each pipeline step by accepting a
dict
, in addition to accepting Retry object.Caveat: The retry strategy dict keys must correspond exactly to the step variable names - A validation step could be added to warn the customer of any unrecognized keys.
For example:
If a
dict
is received, only add retriers to steps with defined strategies in that dict.Alternate solution 2:
Only add retries to integration tests by updating the pipeline workflow with the added retries
Caveat: If the fix is only applied to the integration tests, customers who want to add retry strategies to the pipeline steps will have to do this each time they are creating a pipeline
Testing
Pull Request Checklist
Please check all boxes (including N/A items)
Testing
Documentation
Title and description
Fixes #xxx
- N/ABy submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.