Nginx resolver explained

Nginx resolver explained

Nginx resolver is playing very important part in creating fault tolerant setups, especially when it comes to the free open source version. In this article I’ll explain why we need Nginx resolver and how it works.

I remember the moment about a year or so ago when I came to the office and found people screaming that one of our sites was hacked. When I opened it in my browser I saw completely different site, definitely not ours..

After spending hours trying to figure out the problem, I was finally able to identify the root cause. It wasn’t a hack attack after all, it was something much more simple.. Turned out AWS load balancer changed IP and Nginx cached the old one (we used static reverse proxy configuration).

Let me remind how we usually configure Nginx as reverse proxy:

location / {
    proxy_pass http://service-999999.eu-west-2.elb.amazonaws.com;
} 

This super simple configuration will work just fine until one day / week / month later it will stop. Yes it will stop when AWS changes IP behind ELB DNS record.

But what exactly happens? Turns out when you specify hostname for proxy_pass like we did in our example, Nginx will resolve it once on startup or reload and then cache resulting IP address. For us Nginx was sending requests to the IP that was re-assigned from ELB to someone else EC2 instance. Fun right?

What can we do to avoid the problem?

Well, there are multiple ways:

  1. Initially we wrote a small script that would monitor DNS pointer for the ELB and if it changed, we would reload Nginx. That worked, but we had to run additional piece in our infrastructure which we didn’t like.

  2. Then there is Nginx Plus which provides special resolve flag that you can set on your upstream servers. It will honor DNS TTL and update them automatically if the pointer changes (as per official documentation). I’ve yet to try it. The problem with Nginx Plus that it costs 2.5K / per instance / per year..

  3. There is a module on GitHub called nginx-upstream-dynamic-servers, but it doesn’t have recent updates at the time of writing.. (but do have opened Issues). Please let me know in the comments below if you use it.

  4. Some time ago I stumbled upon an interesting free Nginx resolver alternative which I’ll describe below.

Free Nginx resolver alternative

server {
  listen        80;
  server_name   example.com;

  location / {

    resolver 172.16.0.23;

    set $upstream_endpoint http://service-999999.eu-west-2.elb.amazonaws.com;

    proxy_pass $upstream_endpoint$request_uri;
  }
}

So here we use our famous Nginx resolver directive (172.16.0.23 is AWS default resolver, you can use Google 8.8.8.8, or your own).
When proxy_pass command is getting $variable instead of URI, it uses DNS resolver in case cache entry for the IP has expired. This little handy config secret is exactly what we need!

URI Caveat when using $variable in proxy_pass with trailing slash

Normally in Nginx if we put trailing slash / at the end of the static proxy_pass directive, it will remove the part matched by the location and only proxy remaining part of the URI:

location /test/ {
    proxy_pass http://127.0.0.1:80/;
}

If we get /test/path, only /path will be sent down.

When we use $variable for the proxy_pass with trailing slash, only / will be proxied!

resolver 172.16.0.23;
set $upstream_endpoint http://service-999999.eu-west-2.elb.amazonaws.com/; # Trailing slash
location /test/ {
  proxy_pass $upstream_endpoint;
}

Now request to /test/path will proxy only / which is not what we would expect.

If we really need to preserve trailing slash functionality, we need to remove trailing slash from the $upstrem_endpoint and use rewrite inside of the location:

resolver 172.16.0.23;
set $upstream_endpoint http://service-999999.eu-west-2.elb.amazonaws.com; # No trailing slash
location /test/ {
  rewrite ^/test/(.*) /$1 break;
  proxy_pass $upstream_endpoint;
}

Finally our /test/path request will be converted to desired /path.

This post turned out a bit more technical than I expected, but it’s well worth spending extra bit of time to understand all the examples above. If you plan to use Nginx in the environment with changing ips behind DNS pointers (which is almost everywhere), then you need to either get Nginx Plus, or learn Nginx resolver secrets.

Wake up, Neo... The matrix has you...

Join our growing UNDERGROUND MOVEMENT of Rain Makers. Just drop your email below and your life will never be the same again.

Feel free to reach out on Twitter, Facebook or Instagram.

I won't send you spam. Unsubscribe at any time. Powered by ConvertKit

7 thoughts on “Nginx resolver explained”

  1. For this little hack – will it honor DNS TTL or Nginx will cache the first received IP address from DNS for a longer period of time?

  2. Hey, just would like to say thank you for your time posting this!
    It was really helpful for me and the URI caveat for using trailing slashes was a godsend!
    Thank you!

  3. Thanks for posting this. We use AWS load balancers as backends and the variable trick was useful to make sure we don’t continue using old IP addresses when the backend changes.

  4. I’ve been struggling to understand why adding a variable to proxy_pass was breaking things. Your post clarified a lot and helped me solve my issue. Many thanks!!

Leave a Reply to Edgaras Cancel reply

Your email address will not be published. Required fields are marked *