Some good news: I was reading through the Convox docs for AWS: Convox
I discovered the proxy_protocol
option, and this is set to false
by default.
I believe this can be set by running convox rack params set proxy_protocol=true
The bad news is that this requires 5-10 minutes of downtime. But this is critical to my application so I’ll have to bite the bullet and get it done.
I’m also going to set up monitoring to make sure this never happens again. I added a healthcheck endpoint to ensure that the detected client IP isn’t a private or loopback address.
UPDATE: I set up a new test rack and ran convox rack params set proxy_protocol=true
. There was only 1 minute of downtime, and this fixed the issue. The app is now reporting the correct client IP!
Update from Nov 8, 2022
I notified all my customers last week that we were going to have some downtime today, so I’m going to be setting proxy_protocol=true
on my racks.
I also discovered that this new option was added on the 1st of June, 2022:
- AWS - Add proxy protocol parameter (client’s real source ip) (#440)
This new release introduces the proxy_protocol=true/false
rack parameter to enable AWS NLB proxy-protocol. A use case for proxy-protocol is if you’d like to preserve the client’s source IP of incoming requests. Available via the x-forwarded-for
header key.
If you’re running a non console managed rack just make sure to have aws cli
and jq
installed in your terminal.
Enterprise customers should first update the console app to use the enterprise.convox.com/console:2.2.3
image version.
Important: When changing this parameter in an existing rack the ingress traffic will be unavailable between 5 to 10 minutes so make sure to plan it beforehand.
Upgrading from my old version of 3.3.4
to 3.5.0
also caused a brief blip of downtime, even before setting proxy_protocol=true
. If you manage your own terraform config, I would recommend combining the proxy_protocol=true
and 3.5.0
upgrade into the same terraform apply
so that you don’t need multiple outages.
Side note: It’s always a bit scary to upgrade my Convox release, and I never know if it will cause a lot of downtime or not. So I’ve been holding off on these updates for a while.
It would be really really really nice if I could set up a blue/green deployment for my whole Convox infrastructure, so that I can upgrade the whole system and then seamlessly switch to the new servers by updating the DNS records. This would require DNS challenges in LetsEncrypt so that I can issue an SSL cert before updating the DNS (HTTP challenges would cause a bit of downtime), so I have to keep adding this manually using kubectl
.