x509: subjectis not in the allowed list, x509: certificate signed by unknown authority

[kvaps]

Bug Report

What did you do?
I'm trying to install OLM on vanila Kubernetes cluster

What did you expect to see?

Working packageserver apiservices

What did you see instead? Under which circumstances?

# kubectl get apiservices v1.packages.operators.coreos.com
NAME                               SERVICE                     AVAILABLE                      AGE
v1.packages.operators.coreos.com   olm/packageserver-service   False (FailedDiscoveryCheck)   5h52m

Logs of kube-apiserver:

E1104 14:43:09.934268       1 available_controller.go:437] v1.packages.operators.coreos.com failed with: failing or missing response from https://10.103.42.53:5443/apis/packages.operators.coreos.com/v1: bad status from https://10.103.42.53:5443/apis/packages.operators.coreos.com/v1: 401

logs of packageserver:

E1104 14:58:17.992093      12 authentication.go:53] Unable to authenticate the request due to an error: [x509: subject with cn=microservices-kubernetes-pki-front-proxy-client is not in the allowed list, x509: certificate signed by unknown authority]
I1104 14:58:17.992172      12 httplog.go:90] verb="GET" URI="/apis/packages.operators.coreos.com/v1" latency=1.069742ms resp=401 UserAgent="Go-http-client/2.0" srcIP="10.28.36.177:49534": 

Environment

• operator-lifecycle-manager version: v0.17.0, v0.16.1, v0.15.1 installed using manual method from releases page.

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:18:22Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
  

• Kubernetes cluster kind:

  vanilla kubernetes deployed using helm-chart kubefarm
  (https://github.com/kvaps/kubefarm)

Possible Solution

Looks like there is something wrong with my front-proxy-client certificate.

I also checked certificate generated for the apiservice, and wasn't able to verify it:

# kubectl get secret packageserver-service-cert -o go-template='{{ index .data "olmCAKey" | base64decode }}'  > /tmp/ca.crt
# kubectl get secret packageserver-service-cert -o go-template='{{ index .data "tls.crt" | base64decode }}'  > /tmp/tls.crt
# openssl verify -CAfile /tmp/ca.crt  /tmp/tls.crt 
O = "Red Hat, Inc."
error 18 at 0 depth lookup: self signed certificate
error /tmp/tls.crt: verification failed

The certificates are:

# cat /tmp/ca.crt /tmp/tls.crt
-----BEGIN CERTIFICATE-----
MIIBZzCCAQ6gAwIBAgIIF8O1hSz5VrswCgYIKoZIzj0EAwIwGDEWMBQGA1UEChMN
UmVkIEhhdCwgSW5jLjAeFw0yMDExMDQxNDM4MzVaFw0yMjExMDQxNDM4MzVaMBgx
FjAUBgNVBAoTDVJlZCBIYXQsIEluYy4wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AATUDJiBpiOHBJErl8g0GY/A51J4czQQJRywuJBRt/nBoPtHD1hkblAF4v1RnKaj
uZNT5MgbUNEdkeMznL1Txvdxo0IwQDAOBgNVHQ8BAf8EBAMCAoQwHQYDVR0lBBYw
FAYIKwYBBQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0E
AwIDRwAwRAIgZE9mQgJNtYiLWqlNBoOReVltNLCsiMhWvVaG5aWrMrECID7VGDka
CLk37Rh41RorZAKldwm6p/UNIpyZB4W/FISO
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIBrTCCAVKgAwIBAgIIIA79GirS/8UwCgYIKoZIzj0EAwIwGDEWMBQGA1UEChMN
UmVkIEhhdCwgSW5jLjAeFw0yMDExMDQxNDM4MzVaFw0yMjExMDMxNDM4MzVaMBgx
FjAUBgNVBAoTDVJlZCBIYXQsIEluYy4wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AAQfkQ6hUhYwfK31OJhUaI8O1ex1pAWPSOeJ92itBtk4YHXv38NlJOL7VVGq/AWf
oCuKXv+n2DwgFLoTiUcZH7UMo4GFMIGCMA4GA1UdDwEB/wQEAwIChDAdBgNVHSUE
FjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADBDBgNVHREEPDA6
ghlwYWNrYWdlc2VydmVyLXNlcnZpY2Uub2xtgh1wYWNrYWdlc2VydmVyLXNlcnZp
Y2Uub2xtLnN2YzAKBggqhkjOPQQDAgNJADBGAiEAr9uKsggDGudj+UxhQrpVEnWL
v29z+O3eVtONYLmp+CsCIQDjfTOqhQziEcW0Pl4PcKzakBq5M99/nOt9fJcvkcK/
Cw==
-----END CERTIFICATE-----

the certs are:

Additional context
I'm sure that it is not related, but better to mention it here: to access the cluster I'm using Konnectivity server:
https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

[kvaps]

openssl s_client shows that server expecting certificates, my apiserver client certificate microservices-kubernetes-pki-front-proxy signed by microservices-kubernetes-pki-front-proxy-ca

# openssl s_client -connect localhost:5443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 O = "Red Hat, Inc."
verify error:num=18:self signed certificate
verify return:1
depth=0 O = "Red Hat, Inc."
verify return:1
---
Certificate chain
 0 s:O = "Red Hat, Inc."
   i:O = "Red Hat, Inc."
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBqzCCAVKgAwIBAgIISZhU48ngOlswCgYIKoZIzj0EAwIwGDEWMBQGA1UEChMN
UmVkIEhhdCwgSW5jLjAeFw0yMDExMDQyMTEwMDhaFw0yMjExMDMyMTEwMDhaMBgx
FjAUBgNVBAoTDVJlZCBIYXQsIEluYy4wWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AATmmPTfH6BC3P/0rkER+sLbnp2BCCn2UK27CYa6eZai/o/ZhuQpbsQSpON9hK+I
s3ynsf5f84BtOdoLL5fRRFWmo4GFMIGCMA4GA1UdDwEB/wQEAwIChDAdBgNVHSUE
FjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADBDBgNVHREEPDA6
ghlwYWNrYWdlc2VydmVyLXNlcnZpY2Uub2xtgh1wYWNrYWdlc2VydmVyLXNlcnZp
Y2Uub2xtLnN2YzAKBggqhkjOPQQDAgNHADBEAiBfG9FVxLpX7CnaMeNBRyGL+C29
nud+mMT3SzZQG9jZ9gIgY219vouVEmhyUFL/Cgc8eGHO6Zz422xVwjwG5LPoQ40=
-----END CERTIFICATE-----
subject=O = "Red Hat, Inc."

issuer=O = "Red Hat, Inc."

---
Acceptable client certificate CA names
O = microservices-kubernetes, CN = microservices-kubernetes-pki-ca
O = microservices-kubernetes, CN = microservices-kubernetes-pki-front-proxy-ca
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1033 bytes and written 377 bytes
Verification error: self signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: EC2BB3122E51F6453719D9B6E1A83A2937CFC161581662867AB88CBB0BD3C21B
    Session-ID-ctx: 
    Resumption PSK: 279909897C759A7208E0776A9EAFA8661762A0F751A97359414BAEEBE628EE33
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 7a 7a 76 d4 97 73 94 02-d3 76 69 28 ad 7c fb 12   zzv..s...vi(.|..
    0010 - a9 bd cd 70 31 df 4c d7-77 d6 c2 f4 a2 60 db ff   ...p1.L.w....`..
    0020 - f4 f7 67 b0 cc 59 4d 13-c7 4a 01 48 d7 f2 e0 c7   ..g..YM..J.H....
    0030 - 86 2a 9a 2c 5a c5 4a 4b-c3 71 d1 23 e5 74 96 0e   .*.,Z.JK.q.#.t..
    0040 - f1 25 11 cb 99 37 c2 6d-d2 32 c0 ec ce 81 2d cc   .%...7.m.2....-.
    0050 - 59 37 8f 32 42 4d 5d 32-21 51 4a 4b c8 38 4e 78   Y7.2BM]2!QJK.8Nx
    0060 - 28 78 6c 7a a6 b6 f9 7f-12 d5 80 c1 80 c2 cd c5   (xlz............
    0070 - 19                                                .

    Start Time: 1604524529
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

[kvaps]

As mentioned in jet/kube-webhook-certgen#25 (comment) openssl might have problems to check the issuer of certificate, so I installed libressl istead to perform this check, but check is still failed cause wrong issuer:

# kubectl get secret packageserver-service-cert -o go-template='{{ index .data "olmCAKey" | base64decode }}'  > /tmp/ca.crt
# kubectl get secret packageserver-service-cert -o go-template='{{ index .data "tls.crt" | base64decode }}'  > /tmp/tls.crt
# libressl-openssl verify -CAfile /tmp/ca.crt  /tmp/tls.crt
/tmp/tls.crt: O = "Red Hat, Inc."
error 29 at 1 depth lookup:subject issuer mismatch
O = "Red Hat, Inc."
error 29 at 1 depth lookup:subject issuer mismatch

Thus olm-operator generating wrong certificate?

[exdx]

Hi @kvaps

I'm looking into a similar issue tracked here https://bugzilla.redhat.com/show_bug.cgi?id=1880928 at the moment.

There does seem to be an issue with the packageserver cert handling, but from what I've seen it's because the packageserver doesn't recognize the cert of the api-server and therefore returns that certificate signed by unknown authority error during a cert rotation. The packageserver should be authorizing the api-server via the extension-apiserver-authentication configmap.

I don't believe the issue is with the packageserver cert as its the client (in this case the api-server) that's getting back an error - the packageserver can talk to the api-server ok. Would you agree?

I'm fairly sure the Konnectivity service is causing the issues in this case. Could you verify other api-services you install on a cluster with Konnectivity work as expected?

[kvaps]

Hi @exdx,

Many thanks for the help, you're right the problem was related to missing cn for apiserver client certificate in requestheader-allowed-names in extension-apiserver-authentication confimap.

Problem is not related with the Konnectivity service.