I figured something out. In case anyone else finds this and needs to increase the limit, here’s a quick step-by-step guide. (And for my own future reference.)
A few things to know:
- The resource type for the Nginx ingress controller is
ing
- You can call
kubectl edit ing to edit the config and annotations in your text editor. I use VS Code for this, by running export KUBE_EDITOR="code --wait"
- You can show a list of namespaces with
kubectl get namespaces. We will be using one called <rack name>-system. In this example, my rack name is europe-v3.
Steps
- Switch to your Convox rack:
convox switch europe-v3
- Fetch kubeconfig:
convox rack kubeconfig > /tmp/kubeconfig
- Configure
kubectl to use settings in /tmp/kubeconfig:
export KUBECONFIG=/tmp/kubeconfig
- Show a list of ingress controllers:
kubectl get ing -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
europe-v3-docspring web <none> web.docspring.*******.convox.cloud,eu.docspring.com,app-eu.docspring.com + 2 more... *******.elb.eu-central-1.amazonaws.com 80, 443 156d
europe-v3-system api <none> api.*******.convox.cloud *******.elb.eu-central-1.amazonaws.com 80, 443 156d
europe-v3-system kubernetes <none> api.*******.convox.cloud *******.elb.eu-central-1.amazonaws.com 80, 443 156d
- Show all the details about your
web ingress controller:
kubectl describe ing web -n europe-v3-docspring
Here you can see all the Annotations that are used to configure Nginx:
Name: web
Labels: app=docspring
atom=***************
provider=k8s
rack=europe-v3
release=***************
service=web
system=convox
type=service
Namespace: europe-v3-docspring
Address: ***************.elb.eu-central-1.amazonaws.com
Ingress Class: <none>
Default backend: <default>
TLS:
cert-web terminates web.docspring.***************.convox.cloud
cert-web-domains terminates eu.docspring.com,app-eu.docspring.com,api-eu.docspring.com
Rules:
Host Path Backends
---- ---- --------
web.docspring.***************.convox.cloud
web:4001 (*.*.*.*:4001,*.*.*.*:4001,*.*.*.*:4001 + 1 more...)
eu.docspring.com
web:4001 (*.*.*.*:4001,*.*.*.*:4001,*.*.*.*:4001 + 1 more...)
app-eu.docspring.com
web:4001 (*.*.*.*:4001,*.*.*.*:4001,*.*.*.*:4001 + 1 more...)
api-eu.docspring.com
web:4001 (*.*.*.*:4001,*.*.*.*:4001,*.*.*.*:4001 + 1 more...)
Annotations: alb.ingress.kubernetes.io/scheme: internet-facing
cert-manager.io/cluster-issuer: letsencrypt
convox.com/backend-protocol: http
convox.com/idles: false
kubernetes.io/ingress-class: nginx
nginx.ingress.kubernetes.io/backend-protocol: http
nginx.ingress.kubernetes.io/proxy-connect-timeout: 60
nginx.ingress.kubernetes.io/proxy-read-timeout: 60
nginx.ingress.kubernetes.io/proxy-send-timeout: 60
nginx.ingress.kubernetes.io/server-snippet:
keepalive_timeout 60s;
grpc_read_timeout 60s;
grpc_send_timeout 60s;
client_body_timeout 60s;
nginx.ingress.kubernetes.io/ssl-redirect: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 12m (x58 over 156d) nginx-ingress-controller Scheduled for sync
Normal Sync 12m (x45 over 116d) nginx-ingress-controller Scheduled for sync
- Edit the ingress controller in your text editor:
kubectl edit ing web -n europe-v3-docspring
Add nginx.ingress.kubernetes.io/proxy-buffer-size: "16k" line in the metadata: => annotations: section:
(Or set this to something like "128k" if you need larger buffers.)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# ...
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
- Save and close the file
- Show a list of your ingress controller pods:
kubectl get pods -l name=ingress-nginx -n europe-v3-system
NAME READY STATUS RESTARTS AGE
ingress-nginx-bd6b85668-48cn8 1/1 Running 0 116d
ingress-nginx-bd6b85668-kzbz5 1/1 Running 0 156d
- Describe your ingress pod:
kubectl describe pod ingress-nginx-bd6b85668-48cn8 -n europe-v3-system
You should see an event at the bottom saying that Nginx has been reloaded with the new configuration. (You don’t need to restart anything, and there shouldn’t be any downtime.)
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RELOAD 23m (x20 over 116d) nginx-ingress-controller NGINX reload triggered due to a change in configuration
- If you want to be extra sure that this worked, you can confirm that the Nginx config has been updated. You can do this by starting a bash session in the pod:
kubectl exec -it ingress-nginx-bd6b85668-48cn8 -n europe-v3-system -- bash
bash-5.1$ ls -l
total 116
...
-rw-r--r-- 1 www-data www-data 41517 Sep 29 21:09 nginx.conf
-rw-r--r-- 1 www-data www-data 2656 Mar 24 2021 nginx.conf.default
...
- Check the
proxy_buffer_size and proxy_buffers values in nginx.conf:
grep "proxy_buffer_size\|proxy_buffers" nginx.conf
proxy_buffer_size 4k;
proxy_buffers 4 4k;
proxy_buffer_size 16k;
proxy_buffers 4 16k;
...
You’ll still see some 4k values, but these are just for some internal services. All the server { ... } configuration blocks for your application should be updated to use the new value (16k in this case.)
Caveats
It looks like you will need to set this annotation whenever you deploy your app or update Convox. This might cause some temporary downtime with 502 errors. (I’m not sure about this though.)
Notes on Terraform
I also noticed that this doesn’t seem to be part of my Terraform configuration. I refreshed the state by running terraform refresh, and was hoping that I would see the new annotation in my terraform.tfstate file. But then I realized that the Terraform config doesn’t actually know anything about my apps, so this part must be managed directly by the Convox application and stored somewhere else.
I’d be interested to know if there’s a way to import all of my application resources into the Terraform .tfstate file, or maybe store a backup of the Convox data. (I’m still a bit fuzzy on how this works.)
Anyway, I’m really glad that I figured out how to fix the issue, and I hope this will be available as an option in the future. I’m going to go back to using CSP and rel=preload headers, and hopefully won’t see any more 502 errors.
