16 April 2014

You probably heard about recent security hole discovered in openssl library called Heartbleed. If not read:

http://heartbleed.com/

http://www.openssl.org/news/secadv_20140407.txt

Anyways here is the catch - turns out Amazon can roll critical updates to all images based of Amazon AMI! (well, not all to be honest, you can still control that process as you will find out if you read links below, but this feature is ON by default)

Here I have couple relevant links describing the process:

https://aws.amazon.com/amazon-linux-ami/faqs/#auto_update

https://aws.amazon.com/amazon-linux-ami/faqs/#lock

You can also check out Amazon response to Heartbleed related question on one of the forums:

For the Amazon Linux AMI, we are actively working on updates, and we hope to push them out on Tuesday. We want to make sure that we get the testing right, because the Amazon Linux AMI (by default) applies Critical and Important security updates on initial launch, unless you configure it otherwise.

This could have an interesting impact on much older Amazon Linux AMIs that are not “locked on launch” to a specific repository version, and we want to carefully test.

Ok, autoupdates sound good and that could (or should?) be treated as a GOOD feature. Unfortunately it can break your app if you missed it. Here is how:

Imagine you replaced your CI box which prepares builds and got new openssl library, then new build happens and compiles against new version which you then roll to production boxes ( which you didn’t replace ) and your app suddenly stop working..

Or even simpler than that - you start new boxes (maybe because of traffic spike) and those come up broken, because you built your nodejs app aginst OpenSSL 1.0.1e 11 Feb 2013 and on your new boxes you get OpenSSL 1.0.1g-fips 7 Apr 2014. Tadaaa. Hope you wont spend couple hours debugging that..

Anyways - the takeaway here is that you need to monitor those critical updates. And don’t be too puzzled one day if you see something new on your backed AMI like I did with this openssl update :)

Contact me on Codementor


Some popular ones

My books recommendations

Great book for operations people. Helped me to design and build solid deployment pipelines. Awesome advices on automated testing as well. The author advocates against feature branches, every commit goes to master! Scary? I know, but it actually makes sense once you get the idea. Read the book to find out more.

One of those rare books where every word counts!

Classics from John Allspaw who is SVP of Infrastructure and Operations at Etsy (and used to work for Flickr). The book covers very important topics like metrics collection, continuous deployment, monitoring, dealing with unexpected traffic spikes, dev and ops collaboration and much more. Def recommend if you are starting out in the operations field or been doing it for a while ( in latter case you probably read this book already :).

This book is must read for every software engineer, no matter which language you use! It will change your perspective on writing code. I was amazed by the quality of material - very detailed and up to the point.

"The only way to make the deadline -- the only way to go fast -- is to keep the code as clean as possible at all times."


blog comments powered by Disqus