Chef Client 12.1.0 Multipackage Installs

This is a guest post by Phil Dibowitz, Production Engineer at Facebook

At the Chef Community Summit this year, as with all Summits, there was a Hack Day. My hack project was to extend Chef’s `package` resource to be able to handle multiple packages at once. Why? There are two primary use-cases:

* When provisioning a new system, installing your base set of packages one at a time is very time consuming. It means setting up an HTTP transaction to download the package, writing the package to disk, ending the HTTP transaction, starting a yum/apt/etc. transaction, unpacking the package to disk, closing/committing the transaction, and then rinse and repeat for each package. Being able to download all of those packages in one HTTP transaction saves a ton of time and installing them all at once saves some more. Even a minimal attempt at grouping one set of packages shaved 4 minutes off of provisioning time for us.
* Sometimes you have two or more packages that must be updated in a single transaction… perhaps because a file moved between them (or because upstream packagers set up poor dependencies). Chef was entirely incapable of handling this (without, for example, using an `execute` resource to call out to yum/apt/etc. directly)

It was a fairly large undertaking (26 files changed, 955 insertions, 266 deletions), and I had a lot of help from Lamont Granquist, but on February 4th, it was finally merged and will be available in Chef 12.1.0. The result – in addition to some cleaned-up code (and some uglied-up code) – is you can now do this:

[ruby]
package [‘coreutils’, ‘lspci’, …] do
action :upgrade
end
[/ruby]

This will not only keep the whole set of packages up-to-date, it’ll only upgrade the necessary subset (if any) and tell you which packages were upgraded (if any).

Note that this can make for some ugly logs – particularly on long lists of packages, so we recommend this instead:

[ruby]
package ‘base OS packages’ do
package_name [‘coreutils’, ‘lspci’, …]
action :upgrade
end
[/ruby]

Using this format has the advantage that any notifications or subscriptions that point to to this will not need to be updated if you add/remove a package from the list.

But we don’t just support `:upgrade`, you can also do:

[ruby]
package ‘my packages’ do
package_name [‘package1’, ‘package2’]
version [‘version1’, ‘version2’]
action :install
end
[/ruby]

Again, this will figure out all the changes that need to happen to converge the state of these packages and do it all as one single yum/apt transaction.

And of course the same with `:remove` and `:purge`. Multipackage rules are supported on platforms that use the `yum` or `apt` providers. While the base `package` provider fully supports it, the underlying subclasses must also support it, and only `yum` and `apt` have had the appropriate surgery.

In addition, Lamont wrote the Multipackage cookbook which you can use to build a list of packages throughout your run that will be installed in a single multipackage call. On Chef versions < 12.1.0 it will fall back to a loop of single-package rules for backward compatibility. I hope this helps you cook up even more awesome!

Avatar
Phil Dibowitz

Phil Dibowitz has been working in systems engineering for 12 years and is currently a production engineer at Facebook. Initially, he worked on the Traffic Infrastructure team automating load balancer configuration management as well as designing and building the production IPv6 infrastructure. Phil now leads the team responsible for rebuilding the configuration management system from the ground up. Prior to Facebook, he worked at Google managing the GMail environment, and at Ticketmaster, where he co-authored and open sourced a configuration management tool called Spine. Phil also contributes to and maintains various open source projects and has spoken around the community at conferences and LUGs on a variety of topics from Configuration Management to Path MTU Discovery to X509.