AutoTLS CA rotation

[Jimvin]

Currently the CA certificates create by secret-operator are valid for two years and signed service certificates for 1 day. We should improve the lifecycle handling for TLS certificates to ensure that new certificates are minted and rotated in when required. We should also generate metrics or alerts for certificate expiration, especially for the CA.

### Must have
- [x] New CA certificate should be generated some time in advance of the current one expiring
- [x] After a new CA is provisioned, old CA should still be used for Some Time(tm) to ensure that old peers will trust them (at least until those peers' own certs expire)
- [x] All valid CA certificates must be added to truststores (ca.crt and truststore.p12)

### Nice to have
- [ ] Clean up expired CA certificates?

[soenkeliebau]

ping @lfrancke

It would be nice to get this in the refinement and implementation queue. Not super urgent, but a low hanging fruit that I am 100% sure will become super-urgent due to customer demands at some point in time if we don't fix it before then.

[lfrancke]

For anyone reading this later, these are the docs (nightly right now): https://docs.stackable.tech/home/nightly/secret-operator/secretclass.html#_certificate_authority_rotation

[lfrancke]

If configured not to provision its own CA, a warning will be issued when there is less than 1 year remaining.

Where is that warning generated?

Expired certificates will currently not be deleted automatically, and should be cleaned up manually.

It is under the CA heading but does this apply to all certificates?
(Or do they go away with the pod they are attached to anyway?)

I think this could use an admonition of some sorts instead of being in-line.
@fhennig Thoughts?

[fhennig]

yes it could be a "NOTE". We don't really have any guidelines for when to use an admonition or when not to though

[soenkeliebau]

It is under the CA heading but does this apply to all certificates?
(Or do they go away with the pod they are attached to anyway?)

Normal certificates die with the pods they were created for. Unless running in rootless mode they are only ever "stored" in ram.

[nightkr]

Where is that warning generated?

In the operator logs, currently. I can understand if we don't want to mention it to avoid overpromising, until we expose it better.

[nightkr]

It is under the CA heading but does this apply to all certificates?
(Or do they go away with the pod they are attached to anyway?)

As @soenkeliebau mentioned, we don't store pod certificates at all, outside of the temporary pod volume.

I think this could use an admonition of some sorts instead of being in-line.
@fhennig Thoughts?

It could be, but I think it's also a relatively low priority problem for people to worry about. I don't want to overuse admonitions to the point where people become blind to them.

[lfrancke]

Thanks!
Could you please do a follow-up PR with these two changes:

1. Either remove the information about the warning or mention that it can be found in the operator logs
2. Make the deletion thing a NOTE admonition

I understand that you have a different opinion on the second but I find those things to actually help because they break up the "monotony" of long texts.

[nightkr]

Sure.

[soenkeliebau]

Where is that warning generated?

In the operator logs, currently. I can understand if we don't want to mention it to avoid overpromising, until we expose it better.

What would a better exposure look like here? I'll be happy to create an issue for it.
In the status of the SecretClass?

Random thougth that was sparked by this, should we start a list of "status stuff you should add to your alerting if you want to be informed about what we think you should do" page in the docs?

[nightkr]

@soenkeliebau
I was primarily thinking of a warning Event, but writing it into the status also sounds like a good idea.

[soenkeliebau]

Events are really short lived and would die pretty soon, so unless someone would alert on that event they might miss it..