Cloud

We spent hours debugging an IP whitelisting issue, but the problem was with one missing NLB setting

In one of our customer environments, IP whitelisting was configured correctly. NGINX ingress annotations were right. CIDR blocks were right. And it still wasn't working - every request was denied, even the requests from whitelisted CIDR ranges. Three hours later, we found the problem, and it had nothing to do with NGINX.

What was the issue then?

One missing load balancer config. Yes. That's correct. This particular load balancer config led to spending three hours troubleshooting to find the root cause. Before we reveal the missing config, let me explain the use case in detail.

This particular app is deployed in an EKS cluster and exposed to the outside with an ingress NGINX. The ingress NGINX controller has a load-balancer type service, which automatically creates an NLB in AWS to expose to external/internal traffic. Later, our client wanted to allow access to this specific app to a set of CIDR ranges only. Hence, we decided to add IP whitelisting for nginx level. You may think that this could be achieved using security groups. Yes, you are correct. But the issue is that it applies to all the backend resources; here, we only needed this whitelisting for a specific app.

Initially, we were troubleshooting the NGINX level, suspecting there was a misconfiguration for IP whitelisting. But while looking into NGINX access logs, we noticed something weird. All the source IPs were the same. Then we realized there was something wrong.

On further analysis, we figured out that these private IPs are load balancer internal IPs, which shed a light💡. By default, NLB uses SNAT with ingress. (SNAT is a network-level feature where a load balancer translates the source IP address of a client packet to one of the private IP addresses of the load balancer before forwarding it to the target.) This was the issue. Although we had whitelisted IPs, the rule always dropped the packets as the source IPs were overridden with NLB internal IPs, and the load balancer's internal IPs were not in the allow list.

How did we fix it?

The fix was simple. We just had to set externalTrafficPolicy to Local in the ingress nginx helm chart values.yaml. This basically enables the preserve client IP option in the NLB target group attributes.