Chef 101: The Road to Best Practices

Chef Community Summits are right around the corner, and they’re easily my favorite event of the year. The community comes together and decides what conversations they want to have, and that becomes the day’s agenda. Last year, I had the privilege of sitting in on a session about teaching Chef in academic settings. Over the course of the conversation, someone pointed out that in their automation classes, they spend some time teaching traditional system administration practices first. They found this made it easier for students to relate to the problems automation tools like Chef solve by providing context for the types of challenges operations teams have historically had to face.

A year later, and that conversation has stuck with me and informed how I approach questions about best practices. When someone asks a question about best practices, myself included, they’re usually looking for a ‘what’ and a ‘how’. It’s important, however, not to lose sight of the ‘why’ as well. Any best practice can easily become an anti-pattern if we don’t have a clear idea of how it benefits us. Once again, context can be everything.

With that in mind, I’d like to do something a little bit different. We’ll be taking a look at some practices that can help you get a running start with your Chef Automation, but through the eyes of Sam, an IT professional and newcomer to Chef. Through Sam’s efforts, we’ll see some of the early pitfalls Chef users can run into, and what solutions exist to ensure our automation efforts don’t lose momentum as they evolve.

Quick Links

Prologue: What do I need to know?

This post is designed to be beginner-friendly. You don’t need any coding experience to follow along — just an interest in automation, and curiosity about how best to implement it with Chef. That said, in order to ensure we’re all on the same page, we’ll need to define some terms before we dive into Sam’s narrative.

Chef Cookbooks – Cookbooks are how Chef Code is organized. A cookbook contains many files, but the most important of them are recipes, which define what configurations we wish to apply to the systems we manage.

Chef Supermarket – The Chef Supermarket is a publicly available repository of user-created cookbooks that are ready to use. Some are authored by Chef Software and its various partners, and some are written by the Chef community at large. In either case, cookbooks on the Supermarket are open source, and can be inspected or modified to suit an individual or organizations specific needs.

Chef Server – A Chef Server is where cookbooks are stored and provides information about each system you manage. Each managed node has a run list, which defines which recipes need to be run on that particular system. It also maintains a search index of information about our environment so that we can easily identify and manage discrete subsets as needed.

Chef Client – The Chef Client is an agent that runs on each managed system. It communicates with the chef server to pull down any cookbooks required by its run list, executes the recipes therein locally, and reports its results back to the Chef Server when complete.

ChefDK – The Chef Development Kit contains everything we’ll need to get started writing and working with chef cookbooks, including a commandline utility for communicating with the chef server, and testing tools so that we can validate cookbooks locally before updating our environments.

One final note before we dive into our story. All of the examples provided are functional, and you can follow along with Sam’s adventure, provided you have access to a chef server, and a node to manage. That said, I encourage readers to first read through the full story. At the end, I’ll be providing some helpful links for hands-on activities, and at that point, you should be able to come back through the article, and apply some of the concepts our friend Sam will be learning. And with that…

Part One: The Chef Client Cookbook

Sam’s journey, like ours today, starts with a blog post. In a recent interview, Mike Rich of IHG talked about his own Chef implementation journey. In that conversation, he provides a fantastic piece of advice for newcomers:

“…get the client to report back once an hour, or even once a day, and just look at the data that it gives you and the value that it gives you out of the box.”

A phrase you hear a lot is “don’t try to boil the ocean”, and it’s probably the most important advice you can receive. Automation is inherently additive; when you automate a component in your environment, that automation persists when you start working on the next piece, so even minor changes add up to profound benefits fairly quickly. This advice also dovetails nicely into the first practice we’ll be covering today:

Implementation: Use the chef-client cookbook.

While there are obviously exceptions to every best practice rule, one thing that’s common of nearly every real-world environment I’ve encountered is the use of the chef-client cookbook. It’s a cookbook available on the Chef Supermarket, written and maintained by Chef Software, and is responsible for configuring the chef-client to run on a regular interval, so that we can ensure that our nodes’ configurations are always kept up to date. This doesn’t take any additional action in and of itself, but like Mike says, it’ll start giving us some data about our environments right out of the box, so it’s a good low-risk first project to get Sam rolling.

After setting up their Chef Server, Sam sets a goal of uploading the chef-client cookbook, and configuring a node to consume it and start reporting in to the Chef Server. The first step is to have a look at the cookbook on the Chef Supermarket.

On the Supermarket, we can see a cookbook’s authors and contributors, an easy-to-reference README, supported platforms, and version history. We can also download the cookbook directly, or view its source on github. From here, Sam decides to clone the project out of git to their local workstation:

git clone https://github.com/chef-cookbooks/chef-client.git

From there, Sam does some reading of the chef documentation and learns that the Development Kit contains a tool called “knife” which can be used to upload new cookbooks to the chef server with the command:

knife cookbook upload chef-client

…and it’s here that Sam hits their first roadblock:

Uploading chef-client  [8.1.8]
ERROR: Cookbook chef-client depends on cookbooks which are not currently
ERROR: being uploaded and cannot be found on the server.
ERROR: The missing cookbook(s) are: 'cron' version '>= 2.0.0', 'logrotate' version '>= 1.9.0', 'windows' version '>= 2.0.0'

Problem: Missing dependencies!

Chef supports defining dependencies in cookbooks, which is a boon to seasoned Chef developers, and as often a stumbling block for newcomers. It means that if another cookbook already defines part of what we’re looking to achieve, we needen’t re-write it, but simply depend on it, and write only the net-new functionality we wish to implement. That said, the flip side of the coin is that we can find ourselves, as Sam has, in what IT professionals affectionally dub “dependency hell”, where we now have a list of new cookbooks we need to install before we can get moving. Even if Sam were to download the three cookbooks highlighted, they may have dependencies of their own that must be satisfied before they can be uploaded.

Thankfully, while I’ve been giving exposition, Sam has done some more digging in the documentation, and discovered that the ChefDK has a tool that can help us here: berkshelf.

Implementation: Use berkshelf to manage dependencies

Berkshelf is a dependency management tool that can install or upload the full dependency chain for a cookbook without us needing to hunt them down individually. In this instance, since all Sam cares about is getting the chef-client cookbook installed, this allows for doing so quickly without spending the afternoon downloading dependencies. To install cookbooks locally, run:

berks install

And to upload them to the chef server:

berks upload

Now Sam can use the knife utility to confirm that the cookbooks were uploaded successfully:

$ knife cookbook list
chef-client       8.1.8
compat_resource   12.19.0
cron              4.1.3
logrotate         2.2.0
ohai              5.2.0
test              0.1.0
windows           3.1.3

Now that the cookbooks are installed, and Sam can move onto the fun part — bootstrapping their first server!

Implementation: Bootstrapping a node.

Note: Bootstrapping Windows 
In the example below, Sam is bootstrapping an Ubuntu server
over SSH. The chef-client cookbook also supports windows, but
SSH is often not an option in windows environments.
Knife can also be used to bootstrap nodes over WinRM.
In later examples we'll see the "knife ssh" command, and it too
has a "knife winrm" counterpart for windows systems. 

In Chef, bootstrapping a node consists of a few things:

  • Installing the chef client
  • Creating a configuration file, defining Chef Server and authentication details
  • Installing any self-signed certs that need to be trusted, if applicable

Knife has a built-in bootstrap command that can take care of all these steps in one shot. After consulting documentation and tutorials, Sam takes care of their first server bootstrap like so:

knife bootstrap server1.fakedomain.tld -x ubuntu -i ~/.ssh/sam_admin.pem --sudo -N server1 -r 'recipe[chef-client]'

The bootstrap command allows us to use a few flags to define some details, so let’s take a quick look at what Sam’s done here.

server1.fakedomain.tld – no flag, just the hostname or IP address of the server being bootstrapped.

-x ubuntu – The username to use when connecting

-i ~/.ssh/sam_admin.pem – An authentication key for ssh. ( -P can be used for password authentication instead)

–sudo – Execute commands with superuser privilege. Typically required for new software installation.

-N server1 – the “node name” to use when registering with the Chef Server

-r ‘recipe[chef-client]’ – the run list to be applied to our node after registration. In this case, run the default recipe out of the chef-client cookbook.

These are just a few of the options available. For a full list of flags for any knife command, simply run it with the –help flag for descriptions of all options and usage examples.

Note: Cookbooks vs. Recipes  
Observant readers may note that even though "chef-client" is a cookbook, 
our run list refers to a recipe. What's the deal with that?! This is a 
common stumbling block for new users, and the short answer is that 
the chef client executes recipes, and cookbooks are simply where those 
recipes are stored. Typically in a run list item, the notation is
recipe[COOKBOOK_NAME::RECIPE_NAME]. A notable exception is our example above, which simply contained recipe[chef-client]. This is shorthand notation for `recipe[chef-client::default]`. Whenever a cookbook is specified without a recipe, that just means that the default recipe will be executed.

Success! After bootstrapping server1, Sam is able to see it by querying the chef server with knife:

$ knife node list
server1

Thinking back to the quote from Mike Rich, he mentioned having a look at the data you get after installing the client. Indeed, now that server1 has been bootstrapped, Sam quickly learns that a ton of information is being stored about that node. To get a brief summary of that information, Sam runs:

$ knife node show server1
 Node Name:   server1
 Environment: _default
 FQDN:        server1.fakedomain.tld
 IP:          12.34.56.78
 Run List:    recipe[chef-client]
 Roles:
 Recipes:     chef-client, chef-client::default, chef-client::service, chef-client::init_service
 Platform:    ubuntu 14.04

Beyond that, Sam can now query chef about various system information pertaining to that node. For example:

$ knife search node 'name:server1' -a shells
 1 items found

 server1:
   shells:
     /bin/sh
     /bin/dash
     /bin/bash
     /bin/rbash
     /usr/bin/tmux
     /usr/bin/screen

Here Sam’s asked the Chef Server to return all nodes who’s name is “server1” and display the value of those nodes’ “shells” attribute.

In this case, the search returned one node, and we can see a list of all the installed shells on that server. But where did this information come from? Even when we’re not applying any configuration changes, every chef client run first runs a tool called ohai, which collects system profiling information that is sent back to the Chef Server along with the results of our run. This means that right out of the box we have a way to learn all manner of information about the systems we manage, and as we bootstrap more and more servers, can easily query Chef for information across our environments. Here are a few more examples of the sort of data that can be pulled out of Chef:

$ knife search node 'name:server1' -a memory.total -a filesystem.by_mountpoint./.percent_used -a ec2.public_ipv4
 1 items found

 server1:
   ec2.public_ipv4:                         34.213.47.248
   filesystem.by_mountpoint./.percent_used: 25%
   memory.total:                            7659520kB

Here we see that not only can Sam pull data about disk usage and memory, but it’s even auto-detected that their node is in Amazon, and allows us to query for AWS internals, like their node’s public IP address. It’s worth noting that this data would be collected even if Sam’s node was bootstrapped without a run-list. The chef-client cookbook simply ensures that the client is run on a regular interval, so that this data is always kept up to date.

Part One: Summary

Even though Sam isn’t yet applying any configurations beyond those for the chef-client itself, they’re already collecting valuable information about the systems they manage, and have gotten past one of the biggest roadblocks new users can come across. As use of Chef evolves, Sam will learn that this data can be used a number of ways when writing automation code. The data collected by ohai allows chef recipes to take conditional action based on the results returned — for example, rather than hard coding the memory tuning for a database or webserver, you can define the resource allocation as a percentage of available resources, allowing the same code to be used in different environments, even if their hardware profiles differ. It can also be used to query chef for information about other nodes, allowing for components like load balancers or monitoring servers to dynamically pull a list of its peers from the chef server, and automatically add new nodes as they’re created.

Before we can get there, however, Sam will need to learn a bit more about how those configuration parameters are managed in Chef, which brings us to…

Part Two: Attributes

After getting the initial setup taken care of, Sam notices that nodes are checking in once every half hour. This may be fine, but what if they prefer hourly, or even daily converges? How might that be accomplished easily?

Problem: The default values provided by the chef-client cookbook need to be changed.

Sam’s first thought is that since the cookbook is open source, it should be easy enough to edit so as to update its check-in interval. However, much as installing the chef-client cookbook is a near-universal best practice, so too is directly editing a community cookbook a near-universal anti-pattern. Consider what happens when a new version of the cookbook is released. It may have updated functionality that Sam will want to take advantage of, but to do so, they’ll first need to re-apply their update, as they will with each subsequent release. After some digging, Sam discovers a likely solution: attributes.

Implementation: Determine the proper attributes to update to change chef-client’s check-in frequency.

Just as cookbooks contain recipes that will be executed by our nodes, they contain variables called ‘attributes‘ that can be used to modify that execution without altering any of the underlying code. To see what default attributes are set in the chef-client cookbook, Sam can refer to its README on the Chef Supermarket, or now that they have a node bootstrapped, can pull that information from the Chef Server the same way they were able to query their node’s profiling information.

$ knife search node 'name:server1' -a chef_client
 1 items found

 server1:
   chef_client:
     backup_path:    /var/lib/chef
     bin:            /usr/bin/chef-client
     cache_path:     /var/cache/chef
     chkconfig:
       start_order: 98
       stop_order:  2
     conf_dir:       /etc/chef
     config:
       chef_server_url:        https://api.chef.io/organizations/chef-101-blog
       client_fork:            true
       exception_handlers:
       node_name:              server1
       report_handlers:
       start_handlers:
       validation_client_name: chef-validator
       verify_api_cert:        true
     cron:
       append_log:            false
       environment_variables:
       hour:                  0,4,8,12,16,20
       log_file:              /dev/null
       mailto:
       minute:                0
       path:
       use_cron_d:            false
       weekday:               *
     daemon_options:
     init_style:     init
     interval:       1800
     load_gems:
     log_dir:        /var/log/chef
     log_file:       client.log
     log_perm:       640
     log_rotation:
       options:    compress
       postrotate: /etc/init.d/chef-client reload >/dev/null || :
       prerotate:
     logrotate:
       frequency: weekly
       rotate:    12
     reload_config:  true
     run_path:       /var/run/chef
     splay:          300
     systemd:
       restart: always
       timeout: false
       timer:   false
     task:
       frequency:          minute
       frequency_modifier: 30
       password:
       user:               SYSTEM

It turns out there’s a lot that we can configure here! For now, though, Sam is going to focus on two attributes in particular: interval and splay.

From the README we can see what those two attributes control:

  • node['chef_client']['interval'] – Sets Chef::Config[:interval] via command-line option for number of seconds between chef-client daemon runs. Default 1800.
  • node['chef_client']['splay'] – Sets Chef::Config[:splay] via command-line option for a random amount of seconds to add to interval. Default 300.

The interval allows us to define the frequency of our chef client runs, whereas the splay adds some randomness to that interval so that as we bootstrap more servers, they don’t all try and check in at exactly the same time. Since the attribute is in seconds, if Sam wants nodes to check in every hour, that would mean setting the interval to 3600 seconds, or 60 minutes.
Since Sam’s environment is small, splay is less of a concern, but setting it to half their interval seems a reasonable default setting to start with. Now that Sam’s found the right attributes to update, the question remains, how are those updates applied?

In Chef, there are a number of places one can update attributes, as well as a defined precedence hierarchy to determine which attribute is used if it’s defined multiple places. In total, there are 15 levels in that hierarchy, as shown in this table, where “1” is the lowest priority and “15” is the highest:

Thankfully, most of these would only be needed for very specific use-cases. For most day-to-day operations, there are only a few levels of precedence we need concern ourselves with. The two most important to understand are the ones at each extreme:

1 – default attributes defined in an attribute file.
These are the defaults provided with each cookbook, like the chef_client configurations we looked at earlier. These ensure that each attribute has a default value, but will be used if and only if the same attribute isn’t defined elsewhere.

15 – automatic attributes collected by ohai
These are the attributes we saw in part one, like our memory and IP information. Because these values are dynamically polled on each chef client run, they are considered canonical, and cannot be overridden by user-defined values.

In order for Sam to successfully override the default values provided in the chef-client cookbook, that attribute will need to be set in one of the precedence levels in between. After some research, Sam determines that a “role” looks like the best place to make that update.

Implementation: Creating a base role

A role is a way to collect multiple recipes and attributes organized by server type. For example, a ‘web-frontend’ role might contain recipes for installing a web server, caching server, and deployment of an application. From the chart above, we see that default attributes defined in roles have a precedence of ‘4’, and should supplant the attributes defined in the chef-client cookbook.

Note: Editing with knife 
Certain commands, like the "knife role create" and "knife node edit"
require setting up a preferred text editor. This can be done by 
setting an EDITOR environmental variable, but the easiest
way is to update your knife.rb to include a line like:
knife[:editor]="/path/to/editor"

To get started, Sam will once again use knife to create a “base” role to be applied to all servers, and define their chef-client configuration.

knife role create base --description "Base configuration for all servers."

This opens up an editor with some pre-formatted json that can be updated with the desired attributes and run list. After creating the role, Sam modifies it so that it will run the chef-client recipe, and update attributes so that servers check in hourly with a 30-minute splay:

{
  "name": "base",
  "description": "Base configuration for all servers.",
  "json_class": "Chef::Role",
  "default_attributes": {
    "chef_client": {
      "interval": 3600,
      "splay": 1800
    }
  },
  "override_attributes": {

  },
  "chef_type": "role",
  "run_list": [
    "recipe[chef-client]"
  ],
  "env_run_lists": {

  }
}

After saving this role, it can be added to newly bootstrapped nodes by substituting “-r role[base]” for “-r recipe[chef-client]”. Existing nodes can be edited with knife, via the “knife node edit” command, so Sam updates server one like so:

knife node edit server1

As with the role creation, this opens server1’s node definition in an editor, so that the run list can be updated. Sam updates things so that the new configuration reads:

{
  "name": "server1",
  "chef_environment": "_default",
  "normal": {
    "tags": [
    ]
  },
  "policy_name": null,
  "policy_group": null,
  "run_list": [
    "role[base]"
  ]
}

The next time server1 checks in, it will pull down its updated run list, and update configurations per Sam’s newly-defined attributes. Instead of waiting for the next automatic check-in, however, Sam opts to apply this change immediately by using the “knife ssh” command, which allows remote execution on any nodes to which we have SSH access.

knife ssh 'name:server1' 'sudo chef-client' -x ubuntu -i ~/.ssh/sam_admin.pem

Note that the flags are similar to the ones used in bootstrapping, and also that rather than specifying a specific host, we’re specifying a query, as with the knife search command. This means that we can effectively run commands on multiple systems, provided they have the same credentials. For example, if instead of 'name:server1', we were to substitute 'recipes:chef-client', it would run the command on all servers that contain the chef-client cookbook in their run list.

In Sam’s case, knife ssh starts returning output from server1 with details of the chef client run being applied. Sam notices that most of these are informational, notifying that specified configurations are already up to date:

server1.fakedomain.tld * directory[/var/run/chef] action create (up to date)

This is an important function of how Chef operates. In Chef recipes, we define the desired state of our nodes, and when chef runs, it will first evaluate whether or not said node is already in that state. If so, it simply informs us that the configuration is up to date, and moves on to its next task. Since Sam already configured the chef client during bootstrap, the /var/run/chef directory was already created, and no action needed to be taken on this subsequent run.

However, since Sam *did* make a change to chef’s configuration, further down the run, we get some different output:

server1.fakedomain.tld * template[/etc/default/chef-client] action create
server1.fakedomain.tld - update content in file /etc/default/chef-client from 27f1d4 to 12ac41
server1.fakedomain.tld --- /etc/default/chef-client 2017-09-27 16:41:24.870259999 +0000
server1.fakedomain.tld +++ /etc/default/.chef-chef-client20170927-17056-6or5d9 2017-09-27 18:42:52.534259999 +0000
server1.fakedomain.tld @@ -1,4 +1,4 @@
server1.fakedomain.tld CONFIG=/etc/chef/client.rb
server1.fakedomain.tld -INTERVAL=1800
server1.fakedomain.tld -SPLAY=300
server1.fakedomain.tld +INTERVAL=3600
server1.fakedomain.tld +SPLAY=1800

Because the interval and splay attributes were changed in the base role, chef detects that Sam’s /etc/default/chef-client template is out of date, and informs Sam of the change being applied.

When the chef client run completes, Sam can now confirm that the chef server has up-to-date information about server1’s configuration:

knife search node 'role:base' -a chef_client.interval -a chef_client.splay
1 items found

server1:
chef_client.interval: 3600
chef_client.splay: 1800

Success!

Part Two: Review

Sam’s taken a big step, and can now modify the execution of recipes without yet having to write a single line of chef code! I mentioned earlier that most users will only need to use a few levels of precedence, and excepting automatic attributes, which are not directly user-modifyable, I’ve found that the magic number is usually three.

  1. default attributes set in cookbooks
  2. role-based attributes to define per-type defaults
  3. environmental attributes to define per-environment defaults

In practice, say we have a few simple shopping cart applications we’re responsible for. We might have a single cookbook that defines the basic configurations for our webserver of choice, but a per-role definition of root directories, database credentials, etc. Even in a single application, however, we might have specific configurations, like different payment gateways for development and production that can be overridden at a per-environment level.

Every user and organization is unique, and this is one instance where a universal best practice is harder to define, but ultimately all of our systems need to be administered by humans, and each level of precedence we use increases the complexity of our environments, so a general rule of thumb is to define attributes consistently in as few places as possible to provide the requisite level of granularity in configuration.

Part Three: Wrapper Cookbooks

After some research, Sam has created some environments on the Chef server, ‘development’ and ‘production’, to help organize nodes, and has decided that when defining attributes in an environment, “override” attributes will be used. Since environmental override attributes have a precedence of “12” per the chart in Part Two, role defaults have a precedence of “4”, and cookbook defaults have a precedence of “1”, this will create a hierarchy very similar to what we outlined in our previous review section.

Sam has also re-evaluated the changes made thus far, and determined that the chef-client check-in interval doesn’t need to differ between development and production, and so decides to leave that configuration where it is. That said, as new automation is created, Sam now has a way to separate production and development configurations as the need arises.

Via the same process used to update its run list in Part Two, Sam modifies server1 so that it now belongs to the development enviornment.

...
"chef_environment": "development",
...

When bootstrapping a new node, the environment can be set with a flag, similarly to how Sam configured server1’s run list in Part 1:

knife bootstrap server2.fakedomain.tld -x ubuntu -i ~/.ssh/sam_admin.pem --sudo -N server2 -E production -r 'role[base]'

Before long, however, Sam uncovers a new issue. So far, the data pulled out of Chef has been quite useful, and Sam would like to try having servers check in more frequently. However, ideally this would be done first only in development to assess any performance impact before promoting to production. Since both the development and production server are consuming the same role, if Sam were to update it, the change would be consumed by both environments. What’s worse, the same problem will arise if any new recipes are added to the base role — there’s no easy way to evaluate them in development before the change is taken live in production!

Problem: Updates to roles cannot be selectively applied

Because of this very issue, many consider using roles in Chef to be an anti-pattern, and it can be a point of frustration for new users. A common solution to this problem is to use a wrapper cookbook. Wrapper cookbooks make use of the dependency feature we covered in Part One to allow us to depend on and include multiple recipes, similar to the way Sam used the run_list parameter in their role in Part Two. Unlike roles, however, cookbooks are versionable, and multiple versions of the same cookbook can be stored on a Chef Server. This not only allows us to more easily track the development history of cookbooks we create, but provide a means to more safely evaluate the changes we make. Chef environments can be configured with cookbook version constraints, where only the defined versions will be consumed by that environment, even if a newer version has been made available.

Implementation: Create a wrapper cookbook to replace the base role

To kick things off, the ChefDK provides a “chef” utility that can be used to generate a new cookbook from a template to get Sam’s wrapper created quickly.

$ chef generate cookbook base_wrapper
Generating cookbook base_wrapper
- Ensuring correct cookbook file content
- Committing cookbook files to git
- Ensuring delivery configuration
- Ensuring correct delivery build cookbook content
- Adding delivery configuration to feature branch
- Adding build cookbook to feature branch
- Merging delivery content feature branch to master

The chef generate command creates a cookbook skeleton for Sam to work with. Using the ‘tree’ utility, we can get an idea of how cookbooks are structured:

$ tree cookbooks/base_wrapper/
cookbooks/base_wrapper/
├── Berksfile
├── README.md
├── chefignore
├── metadata.rb
├── recipes
│ └── default.rb
├── spec
│ ├── spec_helper.rb
│ └── unit
│ └── recipes
│ └── default_spec.rb
└── test
└── smoke
└── default
└── default_test.rb

7 directories, 8 files

In order to have their wrapper cookbook invoke chef-client with the same attributes defined in their base role, Sam will need to make three simple modifications to the default cookbook skeleton they generated:

1) Define a dependency on chef-client in metadata.rb

Every cookbook has a metadata file that defines its name, version, and any dependencies it might have. To add a dependency on the ‘chef-client’ cookbook, Sam updates it like so:

name 'base_wrapper'
maintainer 'The Authors'
maintainer_email 'you@example.com'
license 'All Rights Reserved'
description 'Installs/Configures base_wrapper'
long_description 'Installs/Configures base_wrapper'
version '0.1.0'
chef_version '>= 12.1' if respond_to?(:chef_version)

depends 'chef-client'

2) Define the default behavior in recipes/default.rb

We mentioned in Part One that cookbooks contain a “default” recipe that defines what actions should be performed if no more specific recipe is specified. A simple wrapper cookbook defines recipes to be included from its dependent cookbooks, which provides similar functionality to defining a run_list in a role. To invoke the chef-client cookbook, Sam modifies default.rb to read:

#
# Cookbook:: base_wrapper
# Recipe:: default
#
# Copyright:: 2017, The Authors, All Rights Reserved.

include_recipe 'chef-client::default'

Here we see that Sam has explicitly defined the “default” recipe, but as with our run-list, `include_recipe ‘chef-client’` is functionally equivalent.

3) Define the attributes that need to be modified in attributes/default.rb

The attributes directory in a cookbook is used to define, you guessed it, the default attributes for that cookbook, and a wrapper cookbook is no different. To create a default attributes file for their wrapper cookbook, Sam runs a similar “chef generate” command:

$ chef generate attribute cookbooks/base_wrapper default
Recipe: code_generator::attribute
* directory[cookbooks/base_wrapper/attributes] action create
- create new directory cookbooks/base_wrapper/attributes
* template[cookbooks/base_wrapper/attributes/default.rb] action create
- create new file cookbooks/base_wrapper/attributes/default.rb
- update content in file cookbooks/base_wrapper/attributes/default.rb from none to e3b0c4
(diff output suppressed by config)

Then, Sam updates the newly created attributes/default.rb file to read:

default['chef_client']['interval'] = '3600'
default['chef_client']['splay'] = '1800'

With those files updated, Sam then uploads to the Chef Server with berkshelf, exactly the same way they did the chef-client cookbook initially:

$ cd cookbooks/base_wrapper && berks install && berks upload
Resolving cookbook dependencies...
Fetching 'base_wrapper' from source at .
Fetching cookbook index from https://supermarket.chef.io...
Using base_wrapper (0.1.0) from source at .
Using chef-client (8.1.8)
Using windows (3.1.3)
Using logrotate (2.2.0)
Using compat_resource (12.19.0)
Using cron (4.1.3)
Using ohai (5.2.0)

Just as before, berks is able to pull the full dependency list, even though we only specified the chef-client dependency explicitly.

After uploading the cookbook, Sam updates their nodes’ run lists, so that instead of ‘role[base]’, they now read ‘recipe[base_wrapper]’. Just as before, Sam can now apply those updates to both nodes via the knife ssh command:

knife ssh 'name:server*' 'sudo chef-client' -x ubuntu -i ~/.ssh/sam_admin.pem

Once complete, Sam can confirm now that the nodes no longer contain any roles in their run lists, but that the proper attribute values are still set:

$ knife search node 'name:server*' -a roles -a chef_client.interval -a chef_client.splay
2 items found

server1:
chef_client.interval: 3600
chef_client.splay: 1800
roles:

server2:
chef_client.interval: 3600
chef_client.splay: 1800
roles:

With that complete, Sam can now make use of the aforementioned environmental version pinnings to lock production on this initial version of base_wrapper before making any changes. With knife, Sam can update the production environment by running:

knife environment edit production

To pin production to version 0.1.0 of base_wrapper, Sam updates it to read:

{
"name": "production",
"description": "",
"cookbook_versions": {
"base_wrapper": "= 0.1.0"
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {

},
"override_attributes": {

}
}

Finally, now Sam can update base_wrapper’s attributes file with some new values — a 15 minute check-in with 5 minute splay:

default['chef_client']['interval'] = '900'
default['chef_client']['splay'] = '300'

Since this represents a new version of the cookbook, the version in metadata.rb is updated:

...
long_description 'Installs/Configures base_wrapper'
version '0.1.1'
chef_version '>= 12.1' if respond_to?(:chef_version)
...

After uploading, Sam runs `knife cookbook list -a` which will return all available cookbooks and versions on the Chef Server

$ knife cookbook list -a
base_wrapper 0.1.1 0.1.0
chef-client 8.1.8
compat_resource 12.19.0
cron 4.1.3
logrotate 2.2.0
ohai 5.2.0
test 0.1.0
windows 3.1.3

Next to base_wrapper, we see that both versions 0.1.0 and 0.1.1 are available, and since production is locked down to the former, Sam should safely be able to run the same knife ssh command to run chef-client on both servers, but only the development instance should have its attributes changed. After running, Sam confirms with another knife search:

$ knife search node 'name:server*' -a cookbooks.base_wrapper -a chef_client.interval -a chef_client.splay
2 items found

server1:
chef_client.interval: 900
chef_client.splay: 300
cookbooks.base_wrapper:
version: 0.1.1

server2:
chef_client.interval: 3600
chef_client.splay: 1800
cookbooks.base_wrapper:
version: 0.1.0

Success! Once Sam is satisfied with development’s performance, production’s environment pinning can be updated to start consuming the new configuration, and now Sam has a solid foundation for any future development.

Part Three: Review

Wrapper cookbooks provide a great way to get started with community cookbooks quickly, all while sidestepping some common early pitfalls. Beyond the versioning issue we covered in Part Three, there are a few other distinct benefits we get right out of the gate by starting with a wrapper cookbook:

  • Fewer artifacts to manage: Sam’s initial implementation required downloading a cookbook, installing its dependencies, creating a separate role, and uploading it as well. With a wrapper cookbook, the same result can be achieved without needing to create a separate role, or even needing to check anything out of git — simply generate the wrapper, define dependencies, and upload with berkshelf, and everything else comes along for the ride.
  • Simplified Attribute precedence: By using a wrapper cookbook, Sam gets the same attribute behavior as with the initial role configuration. Attributes in the wrapper cookbook supersede those defined in its upstream dependencies, and environmental overrides still take precedence over both.
  • Greater flexibility: Policyfiles are a recently implemented feature of chef that allow role, environment and cookbook data to be combined into a single, promotable artifact, greatly simplifying some implementations. Because of their feature overlap, policyfiles are not compatible with chef roles, but as wrapper cookbooks aren’t architecturally different than any other cookbook, they can be easily integrated into policyfile workflows.

Epilogue: Where do we go from here?

If you’ve stuck with me this long, you’re probably anxious to start getting some hands-on experience of your own.

Learn Chef Rally is a phenomenal resource to get that process rolling, and in particular, the Infrastructure Automation track will guide you through the process of bootstrapping nodes of your own against a chef server, introduce you to the testing tools in ChefDK, and leave you with everything you’d need to follow along at any point of Sam’s Adventure.

The Integrated Compliance track will expand upon what we’ve covered here by guiding you through creating another wrapper cookbook, this time around the Audit cookbook, which can be used to scan your nodes to see how they perform against your security requirements.

While we saw a great example of a wrapper cookbook here, we’ve only scratched the surface of what they can do. For a more detailed run-down of use cases and examples, my colleague Jerry Aldrich has written a wonderful guide to Writing Wrapper Cookbooks of your own.

If the note about policyfiles piqued your interest, and you’d like to learn more, Chef community member Michael Hedgpeth has put together an invaluable guide that gives a beginner-friendly overview of how policyfiles work, and why you might want to implement them.

For small environments, the individual bootstrapping that Sam’s doing should be sufficient for most needs, but can be cumbersome when configuring larger environments. Look know further than a recent post by David Echols for some strategies for bootstrapping large environments quickly.

Author Nick Rycar

Nick is a Technical Product Marketing Manager working out of Chef HQ in Seattle. When he's not busy preparing product demos, he's torturing his colleagues with terrible puns and needlessly esoteric pop-culture trivia. Mostly he's just another confused New York transplant in the Pacific Northwest.

  • gnana sekar

    Hello Nick Rycar,
    Thank’s for sharing the information, I have started my learning process in Devops in one of the online training institute. I hvae one doubt (i.e)how can we say that chef devops and devops they both are different give me one example, waiting for your answer.

    Thank’s
    gnanasekar

    • Nicolas Rycar

      Hi, Gnanasekar! I hope the learning is going well.

      There isn’t necessarily a formal distinction between “devops” and “chef devops”, but at a high level, the concept of adopting devops practices boils down to combining practices in traditional development and operations to create workflows that can be more broadly applied across teams and encourage collaboration between them.

      That said, we have a ton of material available covering how we apply these principles at Chef, along with some background on the devops movement as a whole. A good place to start would be this landing page, which includes an informative overview PDF with links to talks and articles you may find useful: https://www.chef.io/solutions/devops/

      I hope that helps!