Nginx proxy_read_timeout catch or double php script execution
I was working on import script recently. Expected execution time was more than 60 seconds, the default value of proxy_read_timeout directive in Nginx that’s why Nginx displayed nice 504 Gateway Timeout error every time I run the script, even though the script was still running on the back end.
I didn’t pay attention until I noticed that my import script gets executed multiple times even though I executed it just once (by going to script url in the browser).. That problem was driving me nuts because I was trying to figure what triggers second script execution. After countless hours spent on debugging I finally figured that the problem was caused by Nginx proxy_read_timeout directive..
It turned out that after read_timeout interval passed, Nginx would call that script one more time trying to get the response, so for me it meant that two jobs were executing in parallel updating same resource (database in my case) at the same time, causing unexpected behaviour..
To demonstrate the problem I wrote super simple php script:
<?php error_log('script run'); sleep(10); ?>
and in this case your proxy_read_timeout should be less that 10 seconds. In the error log you will see 2 entries instead of one.
For most scripts that behaviour is not a problem, but make sure you increase Nginx read_timeout for critical scripts which are taking some time (more than default 60 seconds) to execute..
Some popular ones
- Story behind X-Forwarded-For and X-Real-IP headers (23 Apr 2014)
- Internal redirect to another domain with proxy_pass and Nginx (14 Oct 2013)
- Secure data bag items with chef solo (04 Aug 2013)
My books recommendations
Great book for operations people. Helped me to design and build solid deployment pipelines. Awesome advices on automated testing as well. The author advocates against feature branches, every commit goes to master! Scary? I know, but it actually makes sense once you get the idea. Read the book to find out more.
One of those rare books where every word counts!
Classics from John Allspaw who is SVP of Infrastructure and Operations at Etsy (and used to work for Flickr). The book covers very important topics like metrics collection, continuous deployment, monitoring, dealing with unexpected traffic spikes, dev and ops collaboration and much more. Def recommend if you are starting out in the operations field or been doing it for a while ( in latter case you probably read this book already :).
This book is must read for every software engineer, no matter which language you use! It will change your perspective on writing code. I was amazed by the quality of material - very detailed and up to the point.
"The only way to make the deadline -- the only way to go fast -- is to keep the code as clean as possible at all times."
blog comments powered by Disqus