[RHOAIENG-9004] Adjust existing test and workflow for GPU testing

[sutaakar]

Issue link

RHOAIENG-9004

What changes have been made

Adjust GitHub e2e workflow to setup and install NVidia operator.
Convert Ray tests to use MNIST fashion test case on GPU. Adjust TestMNISTPyTorchAppWrapper to use GPU.
As a workaround for having just one GPU the tests are executed sequentially.

I also had to alter uploading CFO image to be loaded by KinD CLI.
The reason is that GPU image uses Docker and doesn't interact well with locally created registry.

Edit:
Thinking whether we should add GPU tests next to existing nonGPU ones, or rather replace them. Having both will increase number of tests, on the other side non GPU tests are faster and can be ran anywhere.

Verification steps

Run e2e PR check - GPU is made available on KinD cluster and tests use it.

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[sutaakar]

@astefanutti Ready for second round of review.
Right now GH action runs both CPU and GPU tests. Thinking whether we should run only GPU tests on PR check, leaving CPU just for possible local execution (will speed up PR check).
Dropped MNIST fashion example, all tests are using "plain" MNIST, to keep changes smaller. Can be ported to some other example later if needed.

[astefanutti]

Great!

Yes CPU is more for local development. It's OK to only run GPU on CI now.

Also if we are to refactor the MNIST test, I think it'd be good to remove PyTorch Lightning and use plain PyTorch, which should Lower the maintenance effort.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment