23 April 2014

This was suppose to be 10 minutes fix..

Or so I thought. If you read my blog you probably know that for the most part I’m doing operations stuff for FastCompany. That means sometimes I deal with little tricky problems like the one I’m going to talk about today.


After recent switch to a new CDN provider, we discovered wrong Client IP address in our Nginx access logs. I started debugging process by taking sample of the headers with tcpdump ( quite useful for this sort of stuff! ) on one of the of the Nginx router boxes.

    sudo tcpdump -A -vvvv -s 9999 -i eth0 port 80 > /tmp/sample

Here is part of packet:

    X-Geo: US
    X-Timer: S1398278909.637000322,VS0,VS3
    X-Varnish: 58497648
    X-Varnish: 506864951
    X-Varnish: 2932897992
    X-Varnish: 1531049198
    X-Forwarded-Port: 80
    X-Forwarded-Proto: http

I would like us to focus on two headers from this sample X-Real-IP and X-Forwarded-For. It turns out our CDN provider was passing Client-IP in both of them, but in our Nginx access logs we saw (last IP of the X-Forwarded-For).

Turns out there is not much information or specifications about those X- headers, but I found out that:

  • X-Forwarded-For is usually used by proxies to carry original Client IP through intermediary hops. Otherwords each time request goes through proxy, it should add current request IP to the list. More details here. The format should look like this pretty much:

      X-Forwarded-For: client, proxy1, proxy2

    There is one problem though (as with many standards implementations) it’s not mandatory to use, meaning some proxies will add the header and some will not.. And it’s quite easy to spoof as well. Let say you want to hide your real IP - to do that you can just send a request with X-Forwarded-For: spoof and proxy will gladly add request IP to the list. The result will look like this X-Forwarded-For: spoof realip. As you can see, you can’t just extract leftmost IP, because it might be forged (you also need to keep that in mind if you are using that X-Forwarded-For in the application for some kind of IP based logic).

  • X-Real-IP to be honest, I didn’t find any good info or specs about this one.. Looking through numerous blog posts about passing true Client IP to the backend it seem like it’s common to set both X-Real-IP and X-Forwarded-For. Anyways if you know more about the history of this header, you are more than welcomed to do it in the comments section below.

Nginx time to shine

At this point we have both X-Forwarded-For and X-Real-IP in the request headers (and both contain precious Client-IP, well to be honest, you cant 100% say what’s real client IP, because packets travel through many proxies between client and us. I guess that’s another story) but Nginx uses current request IP.

To change that behaviour we need to use Nginx real_ip module (I don’t think it’s compiled by default). The module exposes couple control knobs which we can use to alter client IP.

If you look at the documentation of the module, you will notice that by default Nginx will use X-Real-IP header (real_ip_header) if it’s present in the request. Since we have Client IP set by CDN to be X-Real-IP, there is no need to do anything else and we should see correct IP in the logs after enabling the module.

There are couple other important things though: set_real_ip_from (set addresses allowed to influence client IP change) and real_ip_recursive.

Lets talk about second one. Since Nginx (whith real_ip module) provides a way to extract client IP from X-Forwarded-For it’s common to see real_ip_header set to X-Forwarded-For, but if you won’t enable real_ip_recursive, you will get rightmost IP inetead of lefmost.. Here is super useful ServerFault post describing the problem and solution.

If you are lazy to read, I’ll summarize - in order to protect yourself from IP spoof, and get REAL CLIENT IP (Morpheus: How do you define ‘real’?) you need to enable real_ip_recursive and set known proxies using set_real_ip_from. Nginx will remove IPs matching known proxies and then use rightmost IP which should be the IP you are looking for!

Looking for hosting?

If you are looking for shared hosting and not sure which one to choose, ncmOnline provides the list of fastest web hosting companies. Most popular choice so far seem to be offering from A2 (full review: a2 hosting reviews).

Contact me on Codementor

Some popular ones

My books recommendations

Great book for operations people. Helped me to design and build solid deployment pipelines. Awesome advices on automated testing as well. The author advocates against feature branches, every commit goes to master! Scary? I know, but it actually makes sense once you get the idea. Read the book to find out more.

One of those rare books where every word counts!

Classics from John Allspaw who is SVP of Infrastructure and Operations at Etsy (and used to work for Flickr). The book covers very important topics like metrics collection, continuous deployment, monitoring, dealing with unexpected traffic spikes, dev and ops collaboration and much more. Def recommend if you are starting out in the operations field or been doing it for a while ( in latter case you probably read this book already :).

This book is must read for every software engineer, no matter which language you use! It will change your perspective on writing code. I was amazed by the quality of material - very detailed and up to the point.

"The only way to make the deadline -- the only way to go fast -- is to keep the code as clean as possible at all times."

blog comments powered by Disqus