Skip to content

Commit 22bc930

Browse files
kannon92kannon92
committed
add FairSharing, QueueLabelPolicy and GangScheduling
1 parent 9fabdde commit 22bc930

14 files changed

+630
-423
lines changed

‎config/schemapatch/kueue-operator.crd.yaml

+68-56
Original file line numberDiff line numberDiff line change
@@ -54,35 +54,6 @@ spec:
5454
required:
5555
- integrations
5656
properties:
57-
fairSharing:
58-
description: fairSharing TODO not done yet
59-
type: object
60-
properties:
61-
preemptionStrategies:
62-
description: |-
63-
preemptionStrategies indicates which constraints should a preemption satisfy.
64-
The preemption algorithm will only use the next strategy in the list if the
65-
incoming workload (preemptor) doesn't fit after using the previous strategies.
66-
Possible values are:
67-
- LessThanOrEqualToFinalShare: Only preempt a workload if the share of the preemptor CQ
68-
with the preemptor workload is less than or equal to the share of the preemptee CQ
69-
without the workload to be preempted.
70-
This strategy might favor preemption of smaller workloads in the preemptee CQ,
71-
regardless of priority or start time, in an effort to keep the share of the CQ
72-
as high as possible.
73-
- LessThanInitialShare: Only preempt a workload if the share of the preemptor CQ
74-
with the incoming workload is strictly less than the share of the preemptee CQ.
75-
This strategy doesn't depend on the share usage of the workload being preempted.
76-
As a result, the strategy chooses to preempt workloads with the lowest priority and
77-
newest start time first.
78-
The default strategy is ["LessThanOrEqualToFinalShare", "LessThanInitialShare"].
79-
type: array
80-
maxItems: 2
81-
items:
82-
type: string
83-
enum:
84-
- LessThanOrEqualToFinalShare
85-
- LessThanInitialShare
8657
integrations:
8758
description: |-
8859
integrations is a required field that configures the Kueue's workload integrations.
@@ -206,35 +177,76 @@ spec:
206177
x-kubernetes-list-type: atomic
207178
kueueGangSchedulingPolicy:
208179
description: |-
209-
kueueGangSchedulingPolicy controls how Kueue admits workloads
210-
Kueue provides the ability to admit workloads all in one (gang admission)
211-
and evicts workloads if they are not ready within a specific time.
212-
This is an optional field.
213-
The allowed values are BlockAdmission, EvictNotReadyWorkloads, and Disabled.
214-
The default value will be Disabled.
215-
BlockAdmission serializes the admission process for Kueue
216-
Workloads will be admitted one at a time and will wait
217-
for the workloads to be ready before going on to the next one in the list.
218-
EvictNotReadyWorkloads will not block admission but will evict Workloads that are not ready
219-
within a defaulted timelimit.
220-
type: string
221-
enum:
222-
- EvictNotReadyWorkloads
223-
- Disabled
180+
kueueGangSchedulingPolicy controls how Kueue admits workloads.
181+
Gang Scheduling is the act of all or nothing scheduling.
182+
Kueue provides this ability.
183+
This field is optional.
184+
type: object
185+
properties:
186+
byWorkload:
187+
description: |-
188+
byWorkload controls how admission is done.
189+
When admission is set to Sequential, only pods from the currently processing workload will be admitted.
190+
Once all pods from the current workload are admitted, and ready, Kueue will process the next workload.
191+
Sequential processing may slow down admission when the cluster has sufficient capacity for multiple workloads,
192+
but provides a higher guarantee of workloads scheduling all pods together successfully.
193+
When set to Parallel, pods from any workload will be admitted at any time.
194+
This may lead to a deadlock where workloads are in contention for cluster capacity and
195+
pods from another workload having successfully scheduled prevent pods from the current workload scheduling.
196+
type: string
197+
enum:
198+
- Parallel
199+
- Sequential
200+
x-kubernetes-validations:
201+
- rule: self.policy==ByWorkload
202+
message: byWorkload is only valid if policy equals ByWorkload
203+
policy:
204+
description: |-
205+
policy allows for changing the kinds of gang scheduling Kueue does.
206+
This is an optional field.
207+
The allowed values are ByWorkload and Disabled.
208+
The default value will be Disabled.
209+
ByWorkload allows for configuration how admission is performed
210+
for Kueue.
211+
type: string
212+
enum:
213+
- ByWorkload
214+
- Disabled
215+
preemption:
216+
description: |-
217+
preemption is the process of evicting one or more admitted Workloads to accommodate another Workload.
218+
Kueue has classical preemption and preemption via fair sharing.
219+
type: object
220+
properties:
221+
preemptionStrategy:
222+
description: |-
223+
preemptionStrategy are the types of preemption kueue allows.
224+
Kueue has two types of preemption: classical and fair sharing.
225+
type: string
226+
enum:
227+
- Classical
228+
- FairSharing
224229
queueLabelPolicy:
225230
description: |-
226-
queueLabelPolicy controls whether or not Kueue reconciles
227-
jobs that don't set the label kueue.x-k8s.io/queue-name.
228-
The allowed values are QueueNameRequired and QueueNameOptional.
229-
If set to QueueNameRequired, then those jobs will be suspended and never started unless
230-
they are assigned a queue and eventually admitted. This also applies to
231-
jobs created before starting the kueue controller.
232-
Defaults to QueueNameOptional; therefore, those jobs are not managed and if they are created
233-
unsuspended, they will start immediately.
234-
type: string
235-
enum:
236-
- QueueNameRequired
237-
- QueueNameOptional
231+
queueLabelPolicy controls how kueue manages workloads
232+
The default behavior of Kueue will manage workloads that have a queue-name label.
233+
This field is optional.
234+
type: object
235+
properties:
236+
queueLabelPolicy:
237+
description: |-
238+
queueLabelPolicy controls whether or not Kueue reconciles
239+
jobs that don't set the label kueue.x-k8s.io/queue-name.
240+
The allowed values are QueueNameRequired and QueueNameOptional.
241+
If set to QueueNameRequired, then those jobs will be suspended and never started unless
242+
they are assigned a queue and eventually admitted. This also applies to
243+
jobs created before starting the kueue controller.
244+
Defaults to QueueNameRequired; therefore, those jobs are not managed and if they are created
245+
unsuspended, they will start immediately.
246+
type: string
247+
enum:
248+
- QueueNameRequired
249+
- QueueNameOptional
238250
logLevel:
239251
description: |-
240252
logLevel is an intent based logging for an overall component. It does not give fine grained control, but it is a

‎config/schemapatch/operator.openshift.io_kueue.yaml

+68-56
Original file line numberDiff line numberDiff line change
@@ -54,35 +54,6 @@ spec:
5454
required:
5555
- integrations
5656
properties:
57-
fairSharing:
58-
description: fairSharing TODO not done yet
59-
type: object
60-
properties:
61-
preemptionStrategies:
62-
description: |-
63-
preemptionStrategies indicates which constraints should a preemption satisfy.
64-
The preemption algorithm will only use the next strategy in the list if the
65-
incoming workload (preemptor) doesn't fit after using the previous strategies.
66-
Possible values are:
67-
- LessThanOrEqualToFinalShare: Only preempt a workload if the share of the preemptor CQ
68-
with the preemptor workload is less than or equal to the share of the preemptee CQ
69-
without the workload to be preempted.
70-
This strategy might favor preemption of smaller workloads in the preemptee CQ,
71-
regardless of priority or start time, in an effort to keep the share of the CQ
72-
as high as possible.
73-
- LessThanInitialShare: Only preempt a workload if the share of the preemptor CQ
74-
with the incoming workload is strictly less than the share of the preemptee CQ.
75-
This strategy doesn't depend on the share usage of the workload being preempted.
76-
As a result, the strategy chooses to preempt workloads with the lowest priority and
77-
newest start time first.
78-
The default strategy is ["LessThanOrEqualToFinalShare", "LessThanInitialShare"].
79-
type: array
80-
maxItems: 2
81-
items:
82-
type: string
83-
enum:
84-
- LessThanOrEqualToFinalShare
85-
- LessThanInitialShare
8657
integrations:
8758
description: |-
8859
integrations is a required field that configures the Kueue's workload integrations.
@@ -209,35 +180,76 @@ spec:
209180
x-kubernetes-list-type: atomic
210181
kueueGangSchedulingPolicy:
211182
description: |-
212-
kueueGangSchedulingPolicy controls how Kueue admits workloads
213-
Kueue provides the ability to admit workloads all in one (gang admission)
214-
and evicts workloads if they are not ready within a specific time.
215-
This is an optional field.
216-
The allowed values are BlockAdmission, EvictNotReadyWorkloads, and Disabled.
217-
The default value will be Disabled.
218-
BlockAdmission serializes the admission process for Kueue
219-
Workloads will be admitted one at a time and will wait
220-
for the workloads to be ready before going on to the next one in the list.
221-
EvictNotReadyWorkloads will not block admission but will evict Workloads that are not ready
222-
within a defaulted timelimit.
223-
type: string
224-
enum:
225-
- EvictNotReadyWorkloads
226-
- Disabled
183+
kueueGangSchedulingPolicy controls how Kueue admits workloads.
184+
Gang Scheduling is the act of all or nothing scheduling.
185+
Kueue provides this ability.
186+
This field is optional.
187+
type: object
188+
properties:
189+
byWorkload:
190+
description: |-
191+
byWorkload controls how admission is done.
192+
When admission is set to Sequential, only pods from the currently processing workload will be admitted.
193+
Once all pods from the current workload are admitted, and ready, Kueue will process the next workload.
194+
Sequential processing may slow down admission when the cluster has sufficient capacity for multiple workloads,
195+
but provides a higher guarantee of workloads scheduling all pods together successfully.
196+
When set to Parallel, pods from any workload will be admitted at any time.
197+
This may lead to a deadlock where workloads are in contention for cluster capacity and
198+
pods from another workload having successfully scheduled prevent pods from the current workload scheduling.
199+
type: string
200+
enum:
201+
- Parallel
202+
- Sequential
203+
x-kubernetes-validations:
204+
- rule: self.policy==ByWorkload
205+
message: byWorkload is only valid if policy equals ByWorkload
206+
policy:
207+
description: |-
208+
policy allows for changing the kinds of gang scheduling Kueue does.
209+
This is an optional field.
210+
The allowed values are ByWorkload and Disabled.
211+
The default value will be Disabled.
212+
ByWorkload allows for configuration how admission is performed
213+
for Kueue.
214+
type: string
215+
enum:
216+
- ByWorkload
217+
- Disabled
218+
preemption:
219+
description: |-
220+
preemption is the process of evicting one or more admitted Workloads to accommodate another Workload.
221+
Kueue has classical preemption and preemption via fair sharing.
222+
type: object
223+
properties:
224+
preemptionStrategy:
225+
description: |-
226+
preemptionStrategy are the types of preemption kueue allows.
227+
Kueue has two types of preemption: classical and fair sharing.
228+
type: string
229+
enum:
230+
- Classical
231+
- FairSharing
227232
queueLabelPolicy:
228233
description: |-
229-
queueLabelPolicy controls whether or not Kueue reconciles
230-
jobs that don't set the label kueue.x-k8s.io/queue-name.
231-
The allowed values are QueueNameRequired and QueueNameOptional.
232-
If set to QueueNameRequired, then those jobs will be suspended and never started unless
233-
they are assigned a queue and eventually admitted. This also applies to
234-
jobs created before starting the kueue controller.
235-
Defaults to QueueNameOptional; therefore, those jobs are not managed and if they are created
236-
unsuspended, they will start immediately.
237-
type: string
238-
enum:
239-
- QueueNameRequired
240-
- QueueNameOptional
234+
queueLabelPolicy controls how kueue manages workloads
235+
The default behavior of Kueue will manage workloads that have a queue-name label.
236+
This field is optional.
237+
type: object
238+
properties:
239+
queueLabelPolicy:
240+
description: |-
241+
queueLabelPolicy controls whether or not Kueue reconciles
242+
jobs that don't set the label kueue.x-k8s.io/queue-name.
243+
The allowed values are QueueNameRequired and QueueNameOptional.
244+
If set to QueueNameRequired, then those jobs will be suspended and never started unless
245+
they are assigned a queue and eventually admitted. This also applies to
246+
jobs created before starting the kueue controller.
247+
Defaults to QueueNameRequired; therefore, those jobs are not managed and if they are created
248+
unsuspended, they will start immediately.
249+
type: string
250+
enum:
251+
- QueueNameRequired
252+
- QueueNameOptional
241253
logLevel:
242254
description: |-
243255
logLevel is an intent based logging for an overall component. It does not give fine grained control, but it is a

0 commit comments

Comments
 (0)