Tune the current LoRA-affinity algorithm to reduce flapping

[ahg-g]

/assign @kaushikmitr

[k8s-ci-robot]

@ahg-g: GitHub didn't allow me to assign the following users: kaushikmitr.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @kaushikmitr

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kaushikmitr]

/assign @kaushikmitr

[kaushikmitr]

@ahg-g This issue is addressed in this PR: #78
based on controller changes also needed to make some manifest updates: #81