remove: DDPJobDefinition from SDK

[VanillaSpoon]

Issue link

https://issues.redhat.com/browse/RHOAIENG-5096

What changes have been made

This pr contains a removal of the DDPJobDefinition from the codeflare-sdk, as well as updated tests to reflect the change.

A follow up pr will be made to update the docs and demo notebooks upon being merged.

Verification steps

Run the sdk tests and ensure they are functioning correctly.
Run a demo notebook to ensure no breaks.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[Bobbins228]

/lgtm

[Srihari1192]

@VanillaSpoon Tried running tests mnist_raycluster_sdk_oauth_test.py and demo notebooks with cluster configuration defined , App wrapper creation is fine but Ray Cluster creation is Failing..

Tried running on installing Main branch aswell , same error
@ChristianZaccaria Does we missed any cluster configuration here?

tests/e2e/mnist_raycluster_sdk_oauth_test.py::TestRayClusterSDKOauth::test_mnist_ray_cluster_sdk_auth Insecure request warnings have been disabled Written to: /opt/app-root/src/.codeflare/appwrapper/mnist.yaml AppWrapper 'mnist' has been created in the namespace: 'test-ns-8iw5r' ╭──────────────────────╮ │ 🚀 Cluster Queue │ │ Status 🚀 │ │ +-------+----------+ │ │ | Name | Status | │ │ +=======+==========+ │ │ | mnist | queueing | │ │ | | | │ │ +-------+----------+ │ ╰──────────────────────╯ Waiting for requested resources to be set up...

cluster = Cluster(ClusterConfiguration( name='jobtest', namespace='test-dw', num_workers=2, head_cpus="500m", head_memory=2, min_cpus=1, max_cpus=1, min_memory=4, max_memory=4, num_gpus=0, image="quay.io/project-codeflare/ray:latest-py39-cu118", instascale=False, write_to_file=True, ))

[ChristianZaccaria]

@Srihari1192 Those clusterconfiguration look good to me, the openshift_oauth is removed as we are defaulting to it to be true. And all oauth resources are created by the RC Controller in Main. cc: @KPostOffice

[KPostOffice]

There are some examples in the demo notebooks that use these classes. Not sure what the best course of action is (remove notebooks?)

[KPostOffice]

You can remove torchx_config function in cluster.py and torchx_scheduler = "ray" as well as imports involving torchx at the top of unit_test.py and remove torchx from requirements.

[openshift-ci]

New changes are detected. LGTM label has been removed.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from bobbins228. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment