Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update OKD-based CRC build (2022-02) #1123

Closed
justkrys opened this issue Feb 26, 2022 · 42 comments · Fixed by okd-project/okd.io#7
Closed

Update OKD-based CRC build (2022-02) #1123

justkrys opened this issue Feb 26, 2022 · 42 comments · Fixed by okd-project/okd.io#7

Comments

@justkrys
Copy link

justkrys commented Feb 26, 2022 •
edited
Loading

Describe the bug

Same issue as #636. OKD CRC certificates are expired.

Also, an observation in the hope it helps:
This is my OKD/OpenShift "What the heck is it" first experience. The minimal commitment for me is CRC. Oops, it's out of date and broken. :(
I wonder if either longer cert expiry or an automated monthly build of OKD CRC would make for a better first time experience.
As it stands now, I either have to signup for a Redhat 30-day Trial or commit to a full OKD install (say on AWS or whatever) just to be able to do some hands on exploring. Either path is less palatable than a working OKD CRC.
Not to mention that I only find this out after several Gib of downloads.
Anyway, hope you take this in the kind spirit it is offered. :)

Version

4.9.15, I guess.

How reproducible

100%. Certs expire every 30 days. The CRC is dated Nov 2021.

Log bundle

INFO Kubelet client certificate has expired, renewing it... [will take up to 8 minutes] 
INFO Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 8 minutes] 
Failed to renew TLS certificates: please check if a newer CodeReady Containers release is available: Temporary error: certificate /var/lib/kubelet/pki/kubelet-server-current.pem still expired (x60)

WARN A new version (1.39.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/1.39.0/crc-linux-amd64.tar.xz
@vrutkovs
Copy link
Member

cc @cgruver

@cgruver
Copy link

cgruver commented Feb 26, 2022

@justkrys Fully understand your frustration. The OKD build of CRC hasn't been getting much attention since it's all volunteer, and for a while I've been the sole volunteer... :-)

If it helps, you can get the official CRC free of charge, and it does not require a 30 day trial.

It does require that you sign up for a Red Hat developer account to get a pull secret, but that's also free and gets you access to some nice free books and other resources. The CRC install does not limit you to a 30 day trial.

https://cloud.redhat.com/openshift/create/local

@vrutkovs vrutkovs mentioned this issue Feb 27, 2022
Closed
@everflux
Copy link

So is okd crc officially dead and only redhat crc is an option?
Is okd going to get shelved as well?

@cgruver
Copy link

cgruver commented Feb 27, 2022

OKD CRC isn't dead, it just doesn't have a community invested in it yet.

OKD has a vibrant community around it, so it's not going anywhere. But, that really is the difference. There's enough interest in OKD, and users of OKD that there are folks both in and out of Red Hat who spend time contributing to it.

CRC just hasn't found that level of interest. I built and maintained the first few releases of CRC for OKD as a favor. But I have never been a user of it.

Thus, if I have to choose where to invest my FOSS community contributions, it's going to be on projects that I am actively using. This is just the nature of FOSS.

I really would encourage @everflux and @justkrys, if you have some time, we'd love for you to join the OKD community.

There are a few other folks who have expressed interest in getting a CRC group formed to maintain it. They'd love for you to join them. https://github.com/openshift/community#okd-working-group-meetings

I've published some info on how to build CRC for OKD. My forks of crc and snc are likely out of date now, but it might get you started.

https://upstreamwithoutapaddle.com/home-lab/okd-crc/

Cheers

@justkrys
Copy link
Author

Thanks very much for the information and alternative path. I will check that out as a starting point.

Also thank you for your invitation to join the community. :) Unfortunately, I leaned long ago that I do not have the right personality type for any sustained engagement with any community. But I will check things out. Who knows, things change.

Anyway, thanks again for the help. :D

@justkrys
Copy link
Author

One thought, though, if OKD CRC is not getting the love it needs, maybe a word about that in the install docs might be in order. Perhaps an open invitation for help. Just so people are not surprised and confused.

Thanks again!

@cgruver
Copy link

cgruver commented Feb 27, 2022

@justkrys It takes all types of personalities to make a community. Come on over.

:-)

@gbraad
Copy link

gbraad commented Mar 11, 2022

This is my OKD/OpenShift "What the heck is it" first experience. The minimal commitment for me is CRC. Oops, it's out of date and broken. :(

This issue comes from the upstream OpenShift release that prevents certificates to exist longer than 30 days. We from @code-ready/crc-team have tried to resolve this for a long time, but has been denied

@gbraad
Copy link

gbraad commented Mar 14, 2022

@cgruver, could you work with @praveenkumar to create an okd bundle that can be used with the new crc? We might be able to add a preset to test okd as an option. Let's see what can be done to unify this.

@gbraad
Copy link

gbraad commented Mar 17, 2022 •
edited
Loading

My proposal is as follows:

image

The recent addition of presets to crc 2.0 allows for more flexibility in the bundles we can offer. The simplest idea would be to use an OKD preset that uses the FCOS based OKD VM instead.

@cgruver, it would be important for us to understand what specific changes have been made to use this bundle instead of the regular OCP based one.

@cgruver
Copy link

cgruver commented Mar 18, 2022

@gbraad Let me get some time set aside for this, and I'll work with @praveenkumar.

@kai-uwe-rommel
Copy link

If the OKD based CRC is no longer useable, it should be taken offline to not confuse and frustrate potential users.

@JaimeMagiera
Copy link
Contributor

I’ve been asking for a volunteer to do a fresh build. So far no takers. At the WG Docs Subcommittee meeting Tuesday, I’ll add this topic to the agenda.

@gbraad
Copy link

gbraad commented Apr 16, 2022 via email

@gbraad
Copy link

gbraad commented May 19, 2022

@JaimeMagiera can the volunteer reach out to us? getting the preset working and let us do part of the upstream releases will solve the problem long-term

@cgruver
Copy link

cgruver commented May 19, 2022

Hey guys, apologies that I'm MIA here. #DayJob has consumed everything.

@JaimeMagiera Did you get a volunteer?

@JaimeMagiera
Copy link
Contributor

@gbraad, @cgruver: We have not received a volunteer yet. Lots of people asking about the status. No one stepping up though. I'll make another push beginning of next week.

@cgruver
Copy link

cgruver commented May 19, 2022

@JaimeMagiera I may be able to set aside some time over in the next couple of weeks. I'll keep you posted.

@gbraad
Copy link

gbraad commented May 20, 2022

It is not a thing you do in just a few mins. We are still in the process to automate some stuff. We might pick this up also soon. Any assistance is welcomed. Currently we are looking into enabling upstream bundles in our regular builds... This could become a replacement for the okd-crc fork.

@titou10titou10
Copy link

It seems OpenShift CodeReady is now OpenShift Local...
Don't know if there is an OKD "flavor" of OpenShift Local...

@gbraad
Copy link

gbraad commented May 30, 2022 •
edited
Loading

OpenShift Local is a rebrand of CodeReady Containers, but the command and project are still named crc.
We are moving CRC to be the upstream effort for also OKD, so yes... it will hopefully support this soon as a preset.
But I do not think there will be a flavour of this as OpenShift Local is a productization of CRC (a released version of crc with OCP)...

You will see crc releases to be made on our github page and hopefully this issue tracker can be seen as preset-/bundle specific.

Does this answer your question?

@gbraad gbraad mentioned this issue Jun 7, 2022
Closed
@myjoshtheflame
Copy link

If the OKD based CRC is no longer useable, it should be taken offline to not confuse and frustrate potential users.

Spot on ..l just started exploring Openshift and I am extremly confused if I should use OKD or CRC but hanging here and there to be honest

@VGerris
Copy link

VGerris commented Jun 23, 2022 •
edited
Loading

hi, can anyone explain the root cause of the issue and provide a possible workaround? I started with the latest crc that supposedly contains openshift 4.10 and then wanted to try out crc OKD. I removed the machines and ~/.crc and got the certificate renewal error. Is the cause that the 'older' installer has certificates that are no longer valid for some reason?
Does it mean recreating the installer should fix it?

Any workarounds are appreciated, thank you.

I get:
INFO Kubelet client certificate has expired, renewing it... [will take up to 8 minutes]
INFO Kubelet serving certificate has expired, waiting for automatic renewal... [will take up to 8 minutes]

Failed to renew TLS certificates: please check if a newer CodeReady Containers release is available: Temporary error: certificate /var/lib/kubelet/pki/kubelet-server-current.pem still expired (x57)

If I use the newer crc binary from openshift 4.10 I get :
crc-linux-2.4.1-amd64$ ./crc start
file '/home/ubuntu/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle' does not exist

Looks like a repackaging of the image with the new crc could fix it, I just do not know how to do that. Would be more convenient if the image was not hardcoded.

[update] seems this made it work ( after running setup with old binary and using new one to start):
./crc start -b /home/ubuntu/.crc/cache/crc_libvirt_4.8.0-0.okd-2021-10-24-061736.crcbundle
[/update]

@JaimeMagiera
Copy link
Contributor

Yes. That is the overall issue. The certificate expires and the installer needs to be recreated. We (the OKD Working Group) are looking for a volunteer to build a fresh installer. We just don't have the cycles.

@VGerris
Copy link

VGerris commented Jun 23, 2022 via email

@praveenkumar
Copy link

As part of crc-2.x we have decoupled the bundle from the crc binary and as part of setup we download it. It will allow us to generate the bundle adhoc fashion and upload it publicly and renew the bundle when cert expire. Bundle generation happen from https://github.com/code-ready/snc/ and have instruction about how to create an OKD bundle. Our (crc team) near term goal is to put more document/how-to for bundle creation and also update the bundle to latest okd release sooner. We are working on having those bundles as part of container image which will allow us to upload those quickly to quay.io than waiting for any internal/fedora infra.

@VGerris
Copy link

VGerris commented Jun 26, 2022

thanks, docs are appreciated. the latest crc binary with oc 4.10 "just works" on my ubuntu 22.04 now, when I tried to follow instructions for snc I got into all kind of redhat related things to get libvirt and other stuff installed.
When working around it in the script, eventually it complaiend about tcp for libvirt - not sure why because it seems everything works as it should now. Anyway, happy to give it another shot if there are more docs, perhaps I can add Ubuntu support if wished for. Thank you

@gbraad
Copy link

gbraad commented Aug 22, 2022

crc-org/blog#6
Functionality has been added to the 2.7.1 of CRC: https://github.com/code-ready/crc/releases/tag/v2.7.1

@gbraad gbraad mentioned this issue Aug 22, 2022
Merged
@gbraad
Copy link

gbraad commented Aug 22, 2022

One of the next steps is to describe the automated generation flow. Perhaps a terraform or pulumi definition of the setup might help.

@LorbusChris
Copy link
Contributor

code-ready/blog#6 Functionality has been added to the 2.7.1 of CRC: https://github.com/code-ready/crc/releases/tag/v2.7.1

https://crc.dev/blog/posts/2022-08-22-okd-on-crc/

@kaovilai
Copy link

kaovilai commented Oct 4, 2022

Awesome!

Too bad I can't Circle/TravisCI this for https://app.circleci.com/pipelines/github/kaovilai/reliable-ci/15/workflows/e65e2e00-0e06-44da-9598-a0d1a0e593a3/jobs/22

@gbraad
Copy link

gbraad commented Oct 8, 2022

We need to update the info on https://www.okd.io/crc/
The current version published there is nearly unusable due to cert rotation

any idea where we can propose these changes?

/cc @vrutkovs @JaimeMagiera @cgruver

@gbraad
Copy link

gbraad commented Oct 8, 2022

@praveenkumar is a longer term cert extension possible for the OKD bundle too or is this already published?

@cgruver
Copy link

cgruver commented Oct 8, 2022

The site content is here: https://github.com/okd-project/okd.io

Those images are almost a year old too. It's the last CRC for OKD that I built manually.

I'll bring it up in the next OKD working group meeting. For now, that content should probably just be removed until we have an automated process to build OKD bundles.

@kai-uwe-rommel
Copy link

Are you sure it is worth investing scarce manpower into CRC for OKD? The CRC from Red Hat based on OCP is basically free to use, too (you just need a free Red Hat developer account).

@VGerris
Copy link

VGerris commented Oct 11, 2022 via email

@vrutkovs vrutkovs mentioned this issue Oct 11, 2022
Merged
@vrutkovs
Copy link
Member

Good news, this has been resolved in CRC itself. Now it has presets and OKD is included. See https://crc.dev/blog/posts/2022-08-22-okd-on-crc/

@praveenkumar
Copy link

@praveenkumar is a longer term cert extension possible for the OKD bundle too or is this already published?

No, I still have to figure out this for OKD bundles. I will discuss that internally with @vrutkovs next week and try to create next bundle with cert validity of 1 year.

@VGerris
Copy link

VGerris commented Oct 12, 2022 via email

@praveenkumar
Copy link

Or have a refresh mechanism.

@VGerris cert refresh mechanism already present and rotation happen successfully, it just take more time (extra for cert rotation) than usual.

@VGerris
Copy link

VGerris commented Nov 16, 2022

hi, ok, how does one do that?

@praveenkumar
Copy link

hi, ok, how does one do that?

@VGerris End user doesn't have to do anything, it just happen automatic when cert expire.

@VGerris VGerris mentioned this issue May 26, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update CRC docs vrutkovs/okd.io