HTTP headers from load balancer not specifying real client IP

Hi all,

I just created a fresh v3 rack in AWS. I’m trying to pass along the true external client IP to my application, but it looks like that data is being stripped somewhere along the way.

Here’s the raw HTTP request data I see when I run a dummy service with a simple HTTP listener:

X-Request-ID: 771434122d4833d982832008d49dde9c
X-Real-IP: 10.1.85.112
X-Forwarded-For: 10.1.85.112
X-Forwarded-Host: <my host parameter>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Scheme: https
<other client HTTP headers>

10.1.85.112 happens to be the private IP of one of the rack EC2 instances.

Is there any way to get the real external IP, or is that not possible in v3 racks?

Thanks,

Nick

1 Like

I just noticed that the same issue is happening for me as well. I checked all the request headers, and the real IP doesn’t seem to be present anywhere, I’m just getting an internal IP.

I tried editing the nginx-configuration ConfigMap and adding the line:

"use-proxy-protocol": "true"

But this took down all my staging servers (including the Convox API and kubectl), and I had to manually fix it with Terraform (terraform apply.) I really need to figure this out since I have auditing requirements and rate limiting based on IP addresses.

Some good news: I was reading through the Convox docs for AWS: Convox

I discovered the proxy_protocol option, and this is set to false by default.

I believe this can be set by running convox rack params set proxy_protocol=true

The bad news is that this requires 5-10 minutes of downtime. But this is critical to my application so I’ll have to bite the bullet and get it done.

I’m also going to set up monitoring to make sure this never happens again. I added a healthcheck endpoint to ensure that the detected client IP isn’t a private or loopback address.


UPDATE: I set up a new test rack and ran convox rack params set proxy_protocol=true. There was only 1 minute of downtime, and this fixed the issue. The app is now reporting the correct client IP!


Update from Nov 8, 2022

I notified all my customers last week that we were going to have some downtime today, so I’m going to be setting proxy_protocol=true on my racks.

I also discovered that this new option was added on the 1st of June, 2022:

https://github.com/convox/convox/releases/tag/3.5.0

  • AWS - Add proxy protocol parameter (client’s real source ip) (#440)

This new release introduces the proxy_protocol=true/false rack parameter to enable AWS NLB proxy-protocol. A use case for proxy-protocol is if you’d like to preserve the client’s source IP of incoming requests. Available via the x-forwarded-for header key.

If you’re running a non console managed rack just make sure to have aws cli and jq installed in your terminal.

Enterprise customers should first update the console app to use the enterprise.convox.com/console:2.2.3 image version.

Important: When changing this parameter in an existing rack the ingress traffic will be unavailable between 5 to 10 minutes so make sure to plan it beforehand.

Upgrading from my old version of 3.3.4 to 3.5.0 also caused a brief blip of downtime, even before setting proxy_protocol=true. If you manage your own terraform config, I would recommend combining the proxy_protocol=true and 3.5.0 upgrade into the same terraform apply so that you don’t need multiple outages.


Side note: It’s always a bit scary to upgrade my Convox release, and I never know if it will cause a lot of downtime or not. So I’ve been holding off on these updates for a while.

It would be really really really nice if I could set up a blue/green deployment for my whole Convox infrastructure, so that I can upgrade the whole system and then seamlessly switch to the new servers by updating the DNS records. This would require DNS challenges in LetsEncrypt so that I can issue an SSL cert before updating the DNS (HTTP challenges would cause a bit of downtime), so I have to keep adding this manually using kubectl.