✨ Refactor rukpak convert (now with testing!)

[perdasilva]

Description

The Convert function in the rukpak/convert package is very simplistic, hard to extend and lacking in testing. This PR adds a bundle rendering framework composed:

• ResourceGenrator: a function that takes a bundle and produces one or more kubernetes resources
• BundleRenderer: a struct that orchestrates generators to produce the rendered bundle resources

ResourceGenerators added:

• BundleDeploymentGenerator: produces appsv1.Deployments from the bundle CSV
• BundlePermissionsGenerator: produces Roles and RoleBindings from the permissions in the CSV
• BundleClusterPermissionsGenerator: produces ClusterRoles and ClusterRoleBindings from the clusterPermissions in the CSV
• BundleServiceAccountGenerator: produces corev1.ServiceAccount resources from the permissions and clusterPermissions in the CSV
• BundleCRDGenerator: produces CRDs based on the bundle CRDs
• BundleResourceGenerator: produces the resources included in the bundle that are not the CSV

Reviewer Checklist

• API Go Documentation
• Tests: Unit Tests (and E2E Tests, if appropriate)
• Comprehensive Commit Messages
• Links to related GitHub Issue(s)

[netlify]

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	33dbb54
🔍 Latest deploy log	https://app.netlify.com/sites/olmv1/deploys/67f90ac6effd020008a0935e
😎 Deploy Preview	https://deploy-preview-1893--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[joelanford]

I always have to refresh my memory on this question: CSV annotations are copied into the pod annotations, but not the deployment annotations? That may very well be the case, it's just that it never obvious if there is anything that motivates this.

[perdasilva]

I had the same questions in my mind. This is probably a game of telephone, but I just wanted to make sure we had what we currently have in the convert function. I don't know the motivation behind this =S

[codecov]

Codecov Report

Attention: Patch coverage is 94.55587% with 19 lines in your changes missing coverage. Please review.

Project coverage is 69.97%. Comparing base (ed205c5) to head (33dbb54).

Files with missing lines	Patch %	Lines
...-controller/rukpak/render/generators/generators.go	93.38%	6 Missing and 3 partials ⚠️
...ternal/operator-controller/rukpak/render/render.go	87.50%	4 Missing and 2 partials ⚠️
...nternal/operator-controller/rukpak/util/testing.go	88.57%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1893      +/-   ##
==========================================
+ Coverage   69.51%   69.97%   +0.46%     
==========================================
  Files          68       72       +4     
  Lines        5357     5486     +129     
==========================================
+ Hits         3724     3839     +115     
- Misses       1401     1412      +11     
- Partials      232      235       +3     

Flag	Coverage Δ
e2e	50.46% <62.75%> (-0.23%)	⬇️
unit	58.16% <93.12%> (+0.70%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[anik120]

The test cases are chef's kiss

[anik120]

I realized that while testing is being added to test the convert functionalities, the testing is exclusively on the new components (ResourceGenerators, BundleRenders etc).

Maybe a way to break this PR would have to be add tests in the first PR that tests the conversion exhaustively.

Then the new PR would refractor and add the new components, with minimal changes to the tests, while the pieces being called would obviously need to change, the behavior would not change in the existing tests.

That way we can be sure that we're not regressing on any functionality while refactoring and introducing the new components.

[anik120]

This feels a bit weird to me. I get the feeling that these changes are incorporating something new, to something existing. But do we have to keep the existing layout?

What are your thoughts on the following:

var PlainConverter = render.BundleRenderer 

func (r BundleRenderer) Convert(rv1 RegistryV1, installNamespace string, targetNamespaces []string) (*Plain, error) {
.
.
}

PlainConverter.Convert(...)

And then take that one step further:

var RegistryV1ToPlainBundleRenderer = render.BundleRenderer 

RegistryV1ToPlainBundleRenderer.Convert(...)

func (r BundleRenderer) Convert(rv1 RegistryV1, installNamespace string, targetNamespaces []string) (*Plain, error) {
.
.
}

ie get rid of Converter totally, with the Convert function defined in the render package as a BundleRenderer component.

Isn't the PR essentially replacing Converter with Renderer?

[perdasilva]

Eventually I'd like to do away with the convert package all together in favor of the renderer. I just didn't want to also carry over all the logic checking the install and target namespaces, and validating install modes, and conversion support, etc. to the renderer (yet). In hindsight, I wish I had hidden everything inside the Convert() function for now to also reduce touch points with other parts of the code (leaving that for later).

I don't quite get the code suggesting. Seems its to rename PlainConverter to RegistryV1ToPlainBundleRenderer?

[anik120]

Eventually I'd like to do away with the convert package all together in favor of the renderer.

I was talking about this - getting rid of the Converter in favor of using render.BundleRenderer directly, instead of wrapping renderer.BundleRender inside a Converter and then using Converter. That'd mean

package render 

type BundleRenderer struct{
.
.
}

func (r BundleRenderer) Convert (rv1 RegistryV1, installNamespace string, targetNamespaces []string) (*Plain, error)}
.
.
}

would have to be defined.

But that would also mean

carry over all the logic checking the install and target namespaces, and validating install modes, and conversion support, etc. to the renderer (yet)

has to be done. It didn't look like this'd be too much of a heavy lift to me, but maybe I'm missing something

[perdasilva]

it's not about the effort, it's just about the scope of the change in the PR - and I'd also like to consider what to do with the Helm converter as part of this, and that will definitely influence everything up from that. I'd also like to think over whether that logic should live in the renderer itself or outside of it, or just in the Plain renderer, etc. It might have made things tidier, but I don't think it's necessary for this PR (though I do think its necessary).

[perdasilva]

I realized that while testing is being added to test the convert functionalities, the testing is exclusively on the new components (ResourceGenerators, BundleRenders etc).

Maybe a way to break this PR would have to be add tests in the first PR that tests the conversion exhaustively.

Then the new PR would refractor and add the new components, with minimal changes to the tests, while the pieces being called would obviously need to change, the behavior would not change in the existing tests.

That way we can be sure that we're not regressing on any functionality while refactoring and introducing the new components.

I started with that. But, because the convert function was so monolithic it meant each test would need to set itself up in a way that it can get through to the logic it wants to test, e.g. the csv would need to support AllNamespaces (which is not super important when you want to check that strategy deployment spec is properly converted to a deployment resource, etc. It also meant that, for the permissions tests, the resulting resource names would carry the hash (which is also not so important when you want to check that spec.permissions -> rbav1.Role and rbacv1.RoleBinding). Also, the convert function returns all generated resources, which means we need to try and find the expected ones in the pile. Not the end of the world, but adds complexity to write and in case of failure. So, I pivoted away towards adding the regression tests (#1895) , which, tbf might not be as thorough as unit testing, I thought it still gave me enough of an indication that everything would be ok after refactor. Having said all that, the generator unit tests were portable enough that I brought them over to the current rukpak/convert package (#1912). Maybe as we move through, their short livedness will outweigh their drawbacks and give us extra security. Wdyt?

[anik120]

I started with that. But, because the convert function was so monolithic it meant each test would need to set itself up in a way that it can get through to the logic it wants to test..... So, I pivoted away towards adding the regression tests (#1895) , which, tbf might not be as thorough as unit testing, I thought it still gave me enough of an indication that everything would be ok after refactor. Having said all that, the generator unit tests were portable enough that I brought them over to the current rukpak/convert package (#1912). Maybe as we move through, their short livedness will outweigh their drawbacks and give us extra security. Wdyt?

That makes a lot of sense thank you for the context!! #1895 being present basically addresses the concern I was brining up, albeit in a different form - which makes sense given the challenges you described 👍🏽