the_real_ip not capturing correct client ip from cloudfront

[timm088]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Possible bug?

NGINX Ingress controller version: 0.17.1
Kubernetes version (use kubectl version): 1.10.3

• Cloud provider or hardware configuration: AWS (EKS)
• Others:

What happened:
When using use-proxy-protocol: "true", config within nginx.conf is created with
real_ip_header proxy_protocol;

If i add an additional message to the logs to print $http_x_forwarded_for, we see the 'real' client ip.

However, $the_real_ip contains what we assume is the last IP of the x-forwarded-for, not the first (which is the real client ip).

What you expected to happen:

Client ip would be stored in $the_real_ip

Anything else we need to know:

Only appears to happen with cloudfront > elb > ingress
When we bypass cloudfront, the ELB only has a single entry in x-forwarded-for. Looks to be related to multi entry x-forwarded-for

[aledbf]

@timm088 are you setting https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#proxy-real-ip-cidr ?

[aledbf]

something like

# your vpc subnet where ELB resides in
XX.XXX.XXX.XXX/XX;

# AWS CloudFront IP/CIDR range
52.84.0.0/15,54.182.0.0/16,54.192.0.0/16,54.230.0.0/16,54.239.128.0/18,54.239.192.0/19,54.240.128.0/18,204.246.164.0/22,204.246.168.0/22,204.246.174.0/23,204.246.176.0/20,205.251.192.0/19,205.251.249.0/24,205.251.250.0/23,205.251.252.0/23,205.251.254.0/24,216.137.32.0/19;

(please check the addresses)

[timm088]

Thanks @aledbf, appreciate your help.

We did try that, setting multiple ranges and still resulted in an incorrect IP shown in logs.

For the moment, we have decided to switch to using an NLB , and disabling proxy_protocol, which has resulted in the behavior we required for the ingress controller (NLB's are pure TCP, and have no http header concept)

[timm088]

Unfortunately, NLBs turned out to be too experimental in their implementation.

I can definitely confirm @aledbf that no matter what I see to try, proxy-real-ip-cidr configmap does not seem to work as its intended.

I can confirm when i alter log message format, x-forwarded-for has the true client (real) ip.

[eseliger]

@timm088 were you able to find a viable solution for the problem? I'm also running into this.

[timm088]

abandoned from our side for now, returned to ELBs due to some instability in NLBs being 'alpha' supported only. @eseliger

[estahn]

@timm088 @aledbf We experience the same issue. I believe the main issue is real_ip_header proxy_protocol; is incompatible with X-Forwarded-For. Or at least that there is a mixture of proxy protocol and some HTTP header.

proxy-real-ip-cidr will not help, as it results in the following:

	real_ip_header      proxy_protocol;
	real_ip_recursive   on;
	set_real_ip_from    13.124.199.0/24;
	set_real_ip_from    144.220.0.0/16;
        ...

Cloudfront will transfer the address via X-Forwarded-For, but the realip module is looking at proxy protocol.

[timm088]

We have gone with using https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#compute-full-forwarded-for

This will ensure you retain the entire structure of x-forwarded-for when using cloudfront

Its not the best (depending on what the downstream service is doing here), but we've found it to be the most reliable.

[estahn]

@timm088 Yes, that works and will keep the IP in HTTP_X_ORIGINAL_FORWARDED_FOR as well. But I wonder what is the effect on features like rate limiting.

[estahn]

I wonder if forcing the use of real_ip_header X-Forwarded-For with proxy protocol works and resolve this issue. Also what happens if you have some traffic coming through Cloudfront and some bypass the Cloudfront distribution. Imo real_ip_recursive should work on proxy_protocol and X-Forwarded-For.

ingress-nginx/rootfs/etc/nginx/template/nginx.tmpl

Lines 127 to 134 in 8cee8d5

	{{/* Enable the real_ip module only if we use either X-Forwarded headers or Proxy Protocol. */}}
	{{/* we use the value of the real IP for the geo_ip module */}}
	{{ if or $cfg.UseForwardedHeaders $cfg.UseProxyProtocol }}
	{{ if $cfg.UseProxyProtocol }}
	real_ip_header proxy_protocol;
	{{ else }}
	real_ip_header {{ $cfg.ForwardedForHeader }};
	{{ end }}

[estahn]

Here is a possible solution to the issue: https://www.getpagespeed.com/server-setup/nginx/how-to-use-multiple-real-ip-headers-with-nginx