nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 #10452

StefanLobbenmeierObjego · 2023-09-27T11:54:04Z

What happened:

I upgraded from 4.7.2 to 4.8.0 and expected everything to still work.

After the upgrade a pod no longer got the correct requests. I investigated why and discovered that the nginx.conf is no longer generated correctly:

That is the ingress.yaml of the pod that now gets wrong requests:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin-oauth
  namespace: tools
  annotations:
     cert-manager.io/acme-challenge-type: http01
     cert-manager.io/cluster-issuer: letsencrypt
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/use-regex: "true"
     nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Script-Name /pgadmin;

in v 4.7.2 I still get the snippet in my configuration:

proxy_set_header X-Script-Name /pgadmin;

and in v4.8.0 it is now missing:

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

/nginx-ingress-controller --version

NGINX Ingress controller
Release: v1.9.0
Build: 4bd3d6b
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.4
WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1

Environment:

Cloud provider or hardware configuration: Azure
OS (e.g. from /etc/os-release): Ubuntu 22.04.1 LTS
Kernel (e.g. uname -a): Linux nginx-ingress-nginx-controller-7f58f9c446-6l2dt 5.15.0-1030-azure Nginx Ingress Controller ConfigMap support for header size #37-Ubuntu SMP Mon Dec 12 19:15:51 UTC 2022 x86_64 Linux
Install tools:
- Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
Basic cluster related info:
- kubectl version
  Client Version: v1.28.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.25.4
  WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1
- kubectl get nodes -o wide
  NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
  aks-intmain-10577288-vmss000000 Ready agent 238d v1.25.4 10.60.0.116 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1
  aks-intmain-10577288-vmss000001 Ready agent 238d v1.25.4 10.60.2.39 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1
  aks-intmain-10577288-vmss000002 Ready agent 238d v1.25.4 10.60.2.148 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1
How was the ingress-nginx-controller installed:
- If helm was used then please show output of helm ls -A | grep -i ingress
  nginx ingress 29 2023-09-27 13:47:27.895964 +0200 CEST deployed ingress-nginx-4.8.0 1.9.0
- If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  USER-SUPPLIED VALUES:
  controller:
  metrics:
  enabled: true
  podAnnotations:
  prometheus.io/port: 10254
  prometheus.io/scrape: true
  service:
  annotations:
  service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
  externalTrafficPolicy: Local
- If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
- if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
Current State of the controller:
- kubectl describe ingressclasses
  Name: nginx
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/instance=nginx
  app.kubernetes.io/managed-by=Helm
  app.kubernetes.io/name=ingress-nginx
  app.kubernetes.io/part-of=ingress-nginx
  app.kubernetes.io/version=1.9.0
  helm.sh/chart=ingress-nginx-4.8.0
  Annotations: meta.helm.sh/release-name: nginx
  meta.helm.sh/release-namespace: ingress
  Controller: k8s.io/ingress-nginx
  Events:
- kubectl -n <ingresscontrollernamespace> get all -A -o wide
  that is a huge amount of text I would like to avoid
- kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  Name: nginx-ingress-nginx-controller-7995ddbbc4-fgh5g
  Namespace: ingress
  Priority: 0
  Service Account: nginx-ingress-nginx
  Node: aks-intmain-10577288-vmss000001/10.60.2.39
  Start Time: Wed, 27 Sep 2023 13:47:39 +0200
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/instance=nginx
  app.kubernetes.io/managed-by=Helm
  app.kubernetes.io/name=ingress-nginx
  app.kubernetes.io/part-of=ingress-nginx
  app.kubernetes.io/version=1.9.0
  helm.sh/chart=ingress-nginx-4.8.0
  pod-template-hash=7995ddbbc4
  Annotations: prometheus.io/port: 10254
  prometheus.io/scrape: true
  Status: Running
  IP: 10.60.2.109
  IPs:
  IP: 10.60.2.109
  Controlled By: ReplicaSet/nginx-ingress-nginx-controller-7995ddbbc4
  Containers:
  controller:
  Container ID: containerd://d8b9d655b431edc48a4c185891d6ec784f57df0837f2c318678416bf2130bc7c
  Image: registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60
  Image ID: registry.k8s.io/ingress-nginx/controller@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60
  Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP
  Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
  Args:
  /nginx-ingress-controller
  --publish-service=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
  --election-id=nginx-ingress-nginx-leader
  --controller-class=k8s.io/ingress-nginx
  --ingress-class=nginx
  --configmap=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
  --validating-webhook=:8443
  --validating-webhook-certificate=/usr/local/certificates/cert
  --validating-webhook-key=/usr/local/certificates/key
  State: Running
  Started: Wed, 27 Sep 2023 13:47:41 +0200
  Ready: True
  Restart Count: 0
  Requests:
  cpu: 100m
  memory: 90Mi
  Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
  Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
  Environment:
  POD_NAME: nginx-ingress-nginx-controller-7995ddbbc4-fgh5g (v1:metadata.name)
  POD_NAMESPACE: ingress (v1:metadata.namespace)
  LD_PRELOAD: /usr/local/lib/libmimalloc.so
  Mounts:
  /usr/local/certificates/ from webhook-cert (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvswc (ro)
  Conditions:
  Type Status
  Initialized True
  Ready True
  ContainersReady True
  PodScheduled True
  Volumes:
  webhook-cert:
  Type: Secret (a volume populated by a Secret)
  SecretName: nginx-ingress-nginx-admission
  Optional: false
  kube-api-access-tvswc:
  Type: Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds: 3607
  ConfigMapName: kube-root-ca.crt
  ConfigMapOptional:
  DownwardAPI: true
  QoS Class: Burstable
  Node-Selectors: kubernetes.io/os=linux
  Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
  Events:
  Type Reason Age From Message
Normal Scheduled 3m16s default-scheduler Successfully assigned ingress/nginx-ingress-nginx-controller-7995ddbbc4-fgh5g to aks-intmain-10577288-vmss000001
Normal Pulling 3m17s kubelet Pulling image "registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60"
Normal Pulled 3m15s kubelet Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60" in 1.724907226s
Normal Created 3m15s kubelet Created container controller
Normal Started 3m15s kubelet Started container controller
Normal RELOAD 3m12s nginx-ingress-controller NGINX reload triggered due to a change in configuration
- kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
  Name: nginx-ingress-nginx-controller
  Namespace: ingress
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/instance=nginx
  app.kubernetes.io/managed-by=Helm
  app.kubernetes.io/name=ingress-nginx
  app.kubernetes.io/part-of=ingress-nginx
  app.kubernetes.io/version=1.9.0
  helm.sh/chart=ingress-nginx-4.8.0
  Annotations: meta.helm.sh/release-name: nginx
  meta.helm.sh/release-namespace: ingress
  service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
  Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  Type: LoadBalancer
  IP Family Policy: SingleStack
  IP Families: IPv4
  IP: 10.60.17.227
  IPs: 10.60.17.227
  LoadBalancer Ingress: 20.238.248.36
  Port: http 80/TCP
  TargetPort: http/TCP
  NodePort: http 32493/TCP
  Endpoints: 10.60.2.109:80
  Port: https 443/TCP
  TargetPort: https/TCP
  NodePort: https 30555/TCP
  Endpoints: 10.60.2.109:443
  Session Affinity: None
  External Traffic Policy: Local
  HealthCheck NodePort: 31434
  Events:
Current state of ingress object, if applicable:
- kubectl -n <appnnamespace> get all,ing -o wide
- kubectl -n <appnamespace> describe ing <ingressname>
- If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag
Others:
- Any other related information like ;
  - copy/paste of the snippet (if applicable)
  - kubectl describe ... of any custom configmap(s) created and in use
  - Any other related information that may help

That is the ingress.yaml of the pod that now gets wrong requests:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin-oauth
  namespace: tools
  annotations:
     cert-manager.io/acme-challenge-type: http01
     cert-manager.io/cluster-issuer: letsencrypt
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/use-regex: "true"
     nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Script-Name /pgadmin;

How to reproduce this issue:

Anything else we need to know:

The text was updated successfully, but these errors were encountered:

k8s-ci-robot · 2023-09-27T11:54:11Z

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bmv126 · 2023-09-27T15:10:26Z

It may be because of #10393

You need to set below to true.
allow-snippet-annotations

StefanLobbenmeierObjego · 2023-09-27T21:54:01Z

That looks very likely, thanks! I will try tomorrow

longwuyuan · 2023-09-28T01:29:25Z

/remove-kind bug

StefanLobbenmeierObjego · 2023-09-28T08:40:15Z

Thanks, that was it ❤️

kmasuhr · 2023-10-19T12:27:09Z

It would be great if this breaking change was added to the changelog.

xuxiaowei-com-cn · 2023-12-21T07:42:50Z

[root@center-dev-1 ~]# kubectl -n ingress-nginx get deployments.apps ingress-nginx-controller -o yaml | grep configmap
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"},"spec":{"minReadySeconds":0,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx"}},"strategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"}},"spec":{"containers":[{"args":["/nginx-ingress-controller","--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller","--election-id=ingress-nginx-leader","--controller-class=k8s.io/ingress-nginx","--ingress-class=nginx","--configmap=$(POD_NAMESPACE)/ingress-nginx-controller","--validating-webhook=:8443","--validating-webhook-certificate=/usr/local/certificates/cert","--validating-webhook-key=/usr/local/certificates/key"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"LD_PRELOAD","value":"/usr/local/lib/libmimalloc.so"}],"image":"registry.k8s.io/ingress-nginx/controller:v1.9.1@sha256:605a737877de78969493a4b1213b21de4ee425d2926906857b98050f57a95b25","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["/wait-shutdown"]}}},"livenessProbe":{"failureThreshold":5,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"controller","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":443,"name":"https","protocol":"TCP"},{"containerPort":8443,"name":"webhook","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"100m","memory":"90Mi"}},"securityContext":{"allowPrivilegeEscalation":true,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":101},"volumeMounts":[{"mountPath":"/usr/local/certificates/","name":"webhook-cert","readOnly":true}]}],"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"ingress-nginx","terminationGracePeriodSeconds":300,"volumes":[{"name":"webhook-cert","secret":{"secretName":"ingress-nginx-admission"}}]}}}}
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
[root@center-dev-1 ~]#

[root@center-dev-1 ~]# kubectl -n ingress-nginx get configmaps ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"allow-snippet-annotations":"false"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"}}
  creationTimestamp: "2023-10-13T02:25:27Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "32089604"
  uid: 73614794-5de6-4bcd-968e-6ad874c68e20
[root@center-dev-1 ~]#

smanghi1 · 2024-03-13T09:53:27Z

if we set 'allow-snippet-annotations' value to true, won't we get CVE-2021-25742 as per this issue?
kubernetes/kubernetes#126811
or do we have any guidelines to use this directive?

StefanLobbenmeierObjego · 2024-04-03T16:04:48Z

@Lingkar since you eyed this:

The reason we implemented this kubernetes/kubernetes#126811

StefanLobbenmeierObjego added the kind/bug Categorizes issue or PR as related to a bug. label Sep 27, 2023

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority labels Sep 27, 2023

strongjz added this to [SIG Network] Ingress NGINX Sep 27, 2023

StefanLobbenmeierObjego changed the title ~~configuration-snippet broke in helm 4.8.0~~ nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 Sep 27, 2023

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed kind/bug Categorizes issue or PR as related to a bug. labels Sep 28, 2023

StefanLobbenmeierObjego closed this as completed Sep 28, 2023

github-project-automation bot moved this to Done in [SIG Network] Ingress NGINX Sep 28, 2023

Annopaolo mentioned this issue Dec 12, 2023

ADI: ingress requires nginx allowSnippetAnnotations set to true astarte-platform/astarte-kubernetes-operator#352

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 #10452

nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 #10452

StefanLobbenmeierObjego commented Sep 27, 2023

k8s-ci-robot commented Sep 27, 2023

Uh oh!

bmv126 commented Sep 27, 2023

Uh oh!

StefanLobbenmeierObjego commented Sep 27, 2023

Uh oh!

longwuyuan commented Sep 28, 2023

Uh oh!

StefanLobbenmeierObjego commented Sep 28, 2023

Uh oh!

kmasuhr commented Oct 19, 2023

Uh oh!

xuxiaowei-com-cn commented Dec 21, 2023

Uh oh!

smanghi1 commented Mar 13, 2024

Uh oh!

StefanLobbenmeierObjego commented Apr 3, 2024 •

edited

Loading

Uh oh!

nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 #10452

nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0 #10452

Comments

StefanLobbenmeierObjego commented Sep 27, 2023

/nginx-ingress-controller --version

k8s-ci-robot commented Sep 27, 2023

Uh oh!

bmv126 commented Sep 27, 2023

Uh oh!

StefanLobbenmeierObjego commented Sep 27, 2023

Uh oh!

longwuyuan commented Sep 28, 2023

Uh oh!

StefanLobbenmeierObjego commented Sep 28, 2023

Uh oh!

kmasuhr commented Oct 19, 2023

Uh oh!

xuxiaowei-com-cn commented Dec 21, 2023

Uh oh!

smanghi1 commented Mar 13, 2024

Uh oh!

StefanLobbenmeierObjego commented Apr 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanLobbenmeierObjego commented Apr 3, 2024 •

edited

Loading