I would like to provide an update to the community about the incident that impacted Chef-client run failures yesterday and give you a sense of what has been happening behind the scenes. As many of you know, this was a result of a protest by a former employee — I will provide more color on this later in the blog. Right now, my primary concern is for the safety of our people and community members, some of whom I’m concerned to see being singled out during this difficult process. My hope is that our community will show care and respect for each other as we work through valid differences of opinion being expressed.
I am happy to report that as of this blog, no impact to our software persists and our customers’ service has been 100% restored.
Let’s start with the facts.
On Thursday, September 19th an action was performed by a trusted community member in violation of the standards of open source software (OSS) development. The individual yanked several RubyGems that they authored while employed by Chef. In order to remove the gems, they first removed the other owners and took unilateral action to yank the gems, violating established processes for making OSS changes and improperly removing property which Chef owned. This ownership has been established through the Github history of commits, licenses, etc. The individual did not have Chef’s permission to remove these items from the RubyGems site.
Now, let’s talk about the impact.
The gems that were removed include important dependencies that a number of users rely upon for the operation of their Chef deployments. Their removal caused Chef-client runs and builds of Chef-related code to fail in some environments. The Chef team pursued two paths to remediate the issue. The first was to work with the RubyGems organization, who determined the gems were owned by Chef and began efforts to restore gems to the repository and Chef access to the RubyGems namespace. The second was to prepare a forked version of the gems and publish modified cookbooks. Thankfully, the first path was completed this morning (September 20) and no future impact to customers is expected.
During remediation steps, we inadvertently changed the name of the Author in the gem ‘chef-sugar’ to Chef Software, Inc. This was incorrectly done and a mistake as we had people trying to do multiple things at once to get things working. As soon as we were aware of this mistake, we changed the Author back to the original.
Could this happen again?
We are doing everything we can to ensure this cannot happen again to our customers. As a result, we are working to correct some challenges in open source software delivery. The tension is that we want and need to honor the collaboration that is Open Source software development and balance that with the needs of organizations that depend on our software to operate their businesses.
Again, thank you to all of you for being part of the Chef community. Let’s work through this together.
Below is a detailed timeline of events as they unfolded. We will continue to provide updates as necessary while the situation evolves.
Thursday, September 19 2019 Timeline (Approximate times in PST)
|7:30 AM||An issue with Chef-client run failures was reported.|
|8:46 AM||The removal of three RubyGems from the RubyGems repository by a former Chef employee was identified as the cause of the run failures.|
|10:46 AM||Chef engineering contacted RubyGems to ask that gems be restored to the repository and access to the RubyGems namespace be given back to Chef.|
|10:47 AM||As a remediation strategy while waiting for the RubyGems decision, Chef renamed and re-published several versions of each of the gems under Chef ownership to maintain continuity for our users.|
|10:49 AM||Chef inadvertently changed the author name to Chef Software for one gem chef-sugar (removing original author and replacing with ‘Chef Software, Inc’).|
|11:51 AM||RubyGems requested documentation that the rights to these gems belong to Chef Software.|
|1:20 PM||Chef became aware of the author name mistake and changed it back to the rightful author.|
|3:12 PM||Chef responded to the RubyGems organization with documentation of the author’s employment dates and signed Proprietary Information and Inventions Agreement (PIIA) demonstrating ownership.|
|9:00 PM||After reviewing the documentation provided by Chef Software, RubyGems deemed that the ownership and request for restoration was legitimate. RubyGems agreed to restore the gems and namespace.|
Friday, September 20 2019 Timeline (Approximate times in PST)
|11:28 AM||Version 0.9.0 of chef-api was restored to the RubyGems site|
|11:32 AM||All other versions of chef-api were restored to the RubyGems site|
|11:45 AM||All versions of chef-sugar were restored to the RubyGems site|
|11:50 AM||All versions of community-zero and stove were restored to the RubyGems site|
Here are some personal reflections I have posted separately.