The Bourne Shell – fifth film in the trilogy. Jason Bourne tries to patch servers against obscure bug. No fights or car chases. Avoid.
— Charles Arthur (@charlesarthur) September 27, 2014
To nobody’s delight and amusement, this has been the year of critical, remotely-exploitable security vulnerabilities. First we had the Heartbleed bug in OpenSSL, and last week, we had Shellshock, an environment-variable handling bug in Bash, the Bourne-Again Shell. Here are some tips for detecting & fixing security vulnerabilities like this across a fleet of servers using Chef, and then making sure that your fleet is actually safe.
Detecting Vulnerable Bash
One great feature of Chef is that it has a program, Ohai, which collects all system information at the beginning of a Chef Client run. Ohai can be extended using an easy-to-use plugin system to collect additional information. Since Ohai does not by default collect information on shells, I’ve written a quick plugin that you can distribute to all your nodes to check if Bash is vulnerable or not. Here it is:
To use the plugin, create a new cookbook (let’s call it remediate-shellshock) and then use the custom plugin functionality in the Ohai community cookbook to drop off additional plugins.
Remember to put the plugin file (bash.rb) into the files/default/plugins directory in the new cookbook.
As you can tell, I could have written a similar plugin to test for and report on versions of OpenSSL that are vulnerable to Heartbleed as well. There are also many other enhancements possible to this code; for example, I could extend the plugin to check whether /bin/sh is Bash and flag those as vulnerable or not.
Because all the Ohai data is posted to the Chef server at the end of a run, it’s now easy to report on machines that are vulnerable to Shellshock across your entire infrastructure, using knife search:
knife search node 'bash_shellshock_vulnerable:true'
Repairing Vulnerable Bash
Once you’ve detected that your Bash is vulnerable, it’s simple to write recipe code to update your Bash package to the latest version. For example, the following would suffice:
package 'bash' do action :upgrade end
or, to combine the results from the aforementioned Ohai plugin and only upgrade if a vulnerable version is found:
package 'bash' do
action :upgrade
only_if { node['languages']['bash']['shellshock_vulnerable'] }
end
We get a lot of questions about whether Chef is good for “patch management” like this. Often what customers mean by “patch management” is a system that can schedule patching for certain days/times, as well as apply patches to groups of servers. While it is possible to use Chef for this (for an example, check out Brian Flad’s auto-patch cookbook), external systems like RedHat Satellite or Windows Server Update Services (WSUS) have a richer feature set in this area. It’s easy to integrate Chef with them.
Failing Chef Runs if Bash is Vulnerable
Another approach that community member Jeff Blaine has taken is to fail Chef runs immediately in a recipe if Bash is vulnerable. To that end, he has written a cookbook called bash-cve-2014-7169 which runs exactly the same test that I’ve put in the Ohai plugin. When his Chef runs fail, he is alerted to the failing machines, and can take the remediation steps as above.
Summary
Unfortunately, while zero-day critical security vulnerabilities are a fact of life, Chef provides you the ability to detect, repair, and audit security vulnerabilities across your entire infrastructure very quickly. We’ve heard amazing stories from some of our customers who were able to repair the Heartbleed vulnerability across tens of thousands of servers in a matter of hours. A faster response time naturally means a lower risk to your customers, so this is truly a situation in which speed matters.
