Chef: Local Cookbook Development



  1. Execute the command as follows:
    chef generate cookbook test_cookbook
    cd test_cookbook
    Take a look at some of the default files like, spec-helper.rb, default.rb file
  2. We will now create a generator that we will use to create cookbook with our customization:
    mkdir generator
    chef generate generator generator/lcd_origin
    cd generator/lcd_origin/templates/default
    vi insert text of your choice. Save and close.
  3. Edit kitchen.yml.erb change the contents to match the following, as we will be using docker driver and centos 7.2 image:
      name: docker
      privileged: true
      use_sudo: false
      name: chef_zero
      # You may wish to disable always updating cookbooks in CI or other testing environments.
      # For example:
      #   always_update_cookbooks: <%%= !ENV['CI'] %>
      always_update_cookbooks: true
      name: inspec
      - name: centos-7.2
          run_command: /usr/lib/systemd/systemd
      - name: default
          - recipe[<%= cookbook_name %>::default]
            - test/smoke/default
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>

    Make sure you are not adding any tabs as they are invalid in ymls.

  4. Edit ../../files/default/spec_helper.rb Add the following lines:
    RSpec.configure do |config|
      config.platform = 'centos'
      config.version = '7.2.1511'
    end<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
  5. So to tell chef to use our generator when generating cookbooks:
    goto the lcd_origin directory
    mkdir ~/.chef
    vi ~/.chef/config.rb Add the contents as below:

    cookbook_path ['~/chef/cookbooks']
    local_mode true
    if File.basename($PROGRAM_NAME).eql?('chef') && ARGV[0].eql?('generate')
      chefdk.generator.license = "all_rights"
      chefdk.generator.copyright_holder = "Student Name"
      chefdk.generator_cookbook = "/root/cookbooks/lcd_origin" = ""
      chefdk.generator_cookbook = "~/generator/lcd_origin"
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>

    Create directory chef/cookbooks
    mkdir -p ~/chef/cookbooks
    And then generate a cookbook
    cd ~/chef
    chef generate cookbook cookbooks/lcd_web
    If you check the README vi cookbooks/lcd_web/ You will see the text we added previously.
    We can generate attributes by following command:
    chef generate attribute cookbooks/lcd_web default
    We can genrate recipe by:
    chef generate recipe cookbooks/lcd_web users

  6. These attributes can be checked here in lcd_origin/attributes/default.rb
    The recipe can be checked with lcd_origin/recipes/users.rb

Test Driven Development




    1. You can check the chefspec default file  spec/unit/recipes/default_spec.rb
    2. Now let’s run the default_spec
      chef exec rspec
      If you get any warning about platform and platform_version then change the following in your code:
      runner = ‘centos’, version: ‘7.2.1511’)
      For some of you the platform and the platform_version might be already added. It might be ubuntu and 16.04 change it to centos and 7.2.1511 resp.
      Then execute the rspec again. This will give an output similar to below:
      Finished in 0.48374 seconds (files took 1.83 seconds to load)
      2 examples, 0 failures
    3. Now change the default_spec.rb to match the following to check if httpd package is installed or not:
      # Cookbook:: lcd_web
      # Spec:: default
      # Copyright:: 2017, Student Name, All Rights Reserved.
      require 'spec_helper'
      describe 'lcd_web::default' do
        context 'CentOS' do
          let(:chef_run) do
            # for a complete list of available platforms and versions see:
            runner = 'centos', version: '7.2.1511')
          it 'converges successfully' do
            expect { chef_run }.to_not raise_error
          it 'installs httpd' do
            expect(chef_run).to install_package('httpd')

      Then run chef exec rspec You will get an error as below:



      1) lcd_web::default CentOS installs httpd
      Failure/Error: expect(chef_run).to install_package(‘httpd’)

      expected “package[httpd]” with action :install to be in Chef run. Other p ackage resources:

      # ./spec/unit/recipes/default_spec.rb:23:in `block (3 levels) in ‘

      Finished in 0.63225 seconds (files took 1.68 seconds to load)
      3 examples, 1 failure

      Failed examples:

      rspec ./spec/unit/recipes/default_spec.rb:22 # lcd_web::default CentOS installs httpd

      It failed because there was no httpd package present in the system because our cookbook doesn’t do so. Let’s add httpd installation to our default recipe.

      package ‘httpd’ do
      As the default action is install we do not need to specify anything.
      Now let’s execute the rspec: chef exec rspec
      Now the output will be as follows:

      Finished in 0.73318 seconds (files took 1.67 seconds to load)
      3 examples, 0 failures

    4. Add the following below installs httpd task:
          it 'enables the httpd service' do
            expect(chef_run).to enable_service('httpd')
          it 'starts the service' do
            expect(chef_run).to start_service('httpd')
      Add the following in default.rb recipe:
      service 'httpd' do
        action [:start, :enable]
      <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>

      Then run: chef exec rspec

    5. Now what if I add something in the default recipe that might break my cookbook:
      add the line in default.rb
      include_recipe ‘something’
      Then run rspec you will get error as below:
      Failure/Error: runner.converge(described_recipe)Chef::Exceptions::CookbookNotFound:
      Cookbook something not found. If you’re loading something from another cookbook, make sure you configure the dependency in your metadata
      # /tmp/chefspec20170918-4764-5svx8rfile_cache_path/cookbooks/lcd_web/recipes/default.rb:7:in `from_file’
      # ./spec/unit/recipes/default_spec.rb:15:in `block (3 levels) in ‘
      # ./spec/unit/recipes/default_spec.rb:31:in `block (3 levels) in ‘Finished in 0.81827 seconds (files took 1.71 seconds to load)
      5 examples, 4 failuresFailed examples:rspec ./spec/unit/recipes/default_spec.rb:18 # lcd_web::default CentOS converges successfully
      rspec ./spec/unit/recipes/default_spec.rb:22 # lcd_web::default CentOS installs httpd
      rspec ./spec/unit/recipes/default_spec.rb:26 # lcd_web::default CentOS enables the httpd service
      rspec ./spec/unit/recipes/default_spec.rb:30 # lcd_web::default CentOS starts the service

So you can see the test will fail if the cookbook dependency failes as well. You can remove the include_recipe line and re execute the rspec.

Test Kitchen Configuration


  1. Edit .kitchen.yml to match the following:
      name: docker
      privileged: true
      use_sudo: false
      name: chef_zero
      # You may wish to disable always updating cookbooks in CI or other testing environments.
      # For example:
      #   always_update_cookbooks: <%= !ENV['CI'] %>
      always_update_cookbooks: true
      name: inspec
      - name: centos-7.2
      - name: centos-6.8
      - name: dev
          run_command: /usr/sbin/init
          - recipe[la_java::default]
        attributes: { 'java': { 'jdk_version': '7' } }
            - 8080:80
        excludes: centos-6.8
      - name: prod
          run_command: /sbin/init
          - recipe[la_java::default]
        attributes: { 'java': { 'jdk_version': '6' } }
            - 8081:80
        includes: centos-6.8
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>

    Execute the command kitchen list
    If you get error as follows:
    >>>>>> ——Exception——-
    >>>>>> Class: Kitchen::ClientError
    >>>>>> Message: Could not load the ‘docker’ driver from the load path. Please en sure that your driver is installed as a gem or included in your Gemfile if using Bundler.
    >>>>>> ———————-
    >>>>>> Please see .kitchen/logs/kitchen.log for more details
    >>>>>> Also try running `kitchen diagnose –all` for configuration

    Then install the chef gem kitchen-docker with:  chef gem install kitchen-docker
    Then the kitchen list output will be as follows:
    [root@arati7111 lcd_web]# kitchen list
    Instance Driver Provisioner Verifier Transport Last Action Last Error
    dev-centos-72 Docker ChefZero Inspec Ssh
    prod-centos-68 Docker ChefZero Inspec Ssh
    Then run: kitchen converge
    Then kitchen verify
    Thus java 7 will be installed in centos 7 and java 6 on centos 6

  2. If you comment the includes and excludes statements from .kitchen.yml then we will have 4 test cases.
  3. You can download the java cookbook with
    knife cookbook site download
    and add it to the run_list

Using Test Kitchen


Execute the following commands

  1. mkdir test_kitchen
  2. cd test_kitchen
  3. kitchen init
    If you get the following error while executing kitchen init:Successfully installed kitchen-vagrant-1.2.1
    Fetching: mixlib-install-3.6.0.gem (100%)
    Successfully installed mixlib-install-3.6.0
    Fetching: ffi-1.9.18.gem (100%)
    Building native extensions. This could take a while…
    ERROR: Error installing kitchen-vagrant:
    ERROR: Failed to build gem native extension./usr/bin/ruby extconf.rb
    mkmf.rb can’t find header files for ruby at /usr/share/include/ruby.h

Gem files will remain installed in /home/user/.gem/ruby/gems/ffi-1.9.18 for inspection.
Results logged to /home/user/.gem/ruby/gems/ffi-1.9.18/ext/ffi_c/gem_make.out

Then execute the kitchen init command with sudo rights

  • If even then the issue is not resolved execute the following command:
    chef gem list kitchen-vagrant
    Then you will have output as follows:  kitchen-vagrant (1.2.1)
    chef gem uninstall kitchen-vagrant -v 1.2.1
    Then rerun kitchen init with sudo rights
  • chef generate cookbook my_cookbook
    You can see that the test/integration/default directory
    In the cookbook you can see the test/smoke/default directory
  • Execute kitchen list
    If you get an error as follows:
    >>>>>> ——Exception——-
    >>>>>> Class: Kitchen::UserError
    >>>>>> Message: Vagrant 1.1.0 or higher is not installed. Please download a package from
    >>>>>> ———————-
    That means you will have to install vagrant. you can use the following command:
    rpm -ivh again execute kitchen list  you will get an output as below:
    Instance Driver Provisioner Verifier Transport Last Action Last Error
    default-ubuntu-1404 Vagrant ChefSolo Busser Ssh
    default-centos-72 Vagrant ChefSolo Busser Ssh
  • Execute kitchen create
    You will get an error as follows: The provider ‘virtualbox’ that was requested to back the machine ‘default’
    You will need to install virtualbox first: Follow this tutorial for the same:
    If you are not able to install vbox then just change the driver name from vagrant to docker in .kitchen.yml

Then execute kitchen converge
Then execute kitchen setup
Then you can execute kitchen destroy to delete the created instances


Create a cookbook with chef generate cookbook lcd_web. Edit the following file test/smoke/default/default_test.rb. Add the following lines
[ ‘net-tools’, ‘httpd’ ].each do |pkg|
describe package(pkg) do
it { should be_installed }

Then execute kitchen verify it will fail the test as below:

System Package
∅ net-tools should be installed
expected that `System Package net-tools` is installed
System Package
∅ httpd should be installed
expected that `System Package httpd` is installed

Now lets add the chef code to install these packages
Add the following lines in recipes/default.rb
[ ‘net-tools’, ‘httpd’].each do |pkg|
package pkg do
action :install

Then execute kitchen converge
Then run kitchen verify
The output should be somthing as below:
System Package
✔ net-tools should be installed
System Package
✔ httpd should be installed

Static Code Analysis

foodcritic -l
FC001: Use strings in preference to symbols to access node attributes
FC002: Avoid string interpolation where not required
FC004: Use a service resource to start and stop services
FC005: Avoid repetition of resource declarations
FC006: Mode should be quoted or fully specified when setting file permissions
FC007: Ensure recipe dependencies are reflected in cookbook metadata
FC008: Generated cookbook metadata needs updating
FC009: Resource attribute not recognised
FC010: Invalid search syntax
FC011: Missing README in markdown format
FC012: Use Markdown for README rather than RDoc
FC013: Use file_cache_path rather than hard-coding tmp paths
FC014: Consider extracting long ruby_block to library
FC015: Consider converting definition to a Custom Resource
FC016: LWRP does not declare a default action
FC017: LWRP does not notify when updated
FC018: LWRP uses deprecated notification syntax

Now cd to lcd_web cookbook
execute: foodcritic .
output: Checking 2 files
FC008: Generated cookbook metadata needs updating: ./metadata.rb:3
FC064: Ensure issues_url is set in metadata: ./metadata.rb:1
FC065: Ensure source_url is set in metadata: ./metadata.rb:1
FC067: Ensure at least one platform supported in metadata: ./metadata.rb:1
FC078: Ensure cookbook shared under an OSI-approved open source license: ./metadata.rb:1

Lets check our metadata.rb file
For error FC067 add the line in metadata.rb: supports ‘centos’
The other errors are not of our concern so we will exclude them with:
foodcritic . -t ~FC008 -t ~FC064 -t ~FC065 -t ~FC078
There should be no errors.

Now we want to persist these exclusion so we will create .foodcritic file:
vi .foodcritic and add the following lines:




Save and close. Then execute foodcritic .

Lets have a look at rubocop. To install rubocop execute gem install rubocop

You can have a look at it’s help with command rubocop –help

If you run rubocop in lcd_we cookbook you might get output as follows:

init_test/lcd_web/recipes/default.rb:7:2: C: Space inside square brackets detected.
[ ‘net-tools’, ‘httpd’].each do |pkg|
bin/mixlib-install:11:11: C: Prefer single-quoted strings when you don’t need string interpolation or special symbols.
version = “>= 0”
bin/mixlib-install:15:32: C: Prefer single-quoted strings when you don’t need string interpolation or special symbols.
str = str.dup.force_encoding(“BINARY”) if str.respond_to? :force_encoding
bin/mixlib-install:17:15: C: Avoid the use of Perl-style backrefs.
version = $1

32 files inspected, 53 offenses detected

You might have errors like “line too long” etc. By default the line length specified in 80, Now let’s see if we can override that setting.
We will be needing todo file for that let’s generate it by the command:
rubocop –auto-gen-config
Then edit the file .rubocop.yml and add the line: inherit_from: .rubocop_todo.yml
Then execute rubocop you will see that all of our offenses have been masked.
Let’s have a look at the .rubocop_todo.yml file.
Now in the code you will see code blocks like follows:

# Offense count: 1
# Cop supports --auto-correct.
# Configuration parameters: EnforcedStyle, SupportedStyles.
# SupportedStyles: auto_detection, squiggly, active_support, powerpack, unindent
    - 'generator/lcd_origin/recipes/recipe.rb'

This denotes that there i an indentation issue with recipe.rb file.
rubocop has the feature to auto-correct small issues, you will need to comment code block one by one and then execute rubocop –auto-correct
Output will be something like: generator/lcd_origin/metadata.rb:5:1: C: [Corrected] 1 trailing blank lines detected.

So in this way we can correct some issues with rubocop. You can repeat the same for rest of the issues.
For the other issues which cannot be correct with rubo-cop you can add rules for the same in .rubocop.yml

We will change the LineLength rule. Add the following in .rubocop.yml file:
Max: 200
Then comment the LineLength block in .rubocop_todo.yml file.

Then execute rubocop You will see there are no offenses.


Edit file recipes/default.rb. Comment everything and add the following:

lazy_message = "Hello World"

file 'lazy_message' do
  path '/tmp/lazy.txt'
  content "#{lazy_message}"

execute 'yum-makecache' do
  command 'yum makecache'
  notifies :create, 'file[message]', :immediately
  action :nothing

package 'bind-utils' do
  action :install
  notifies :run, 'execute[yum-makecache]', :before

file 'message' do
  path '/tmp/message.txt'
  content lazy { "#{lazy_message}" }

lazy_message = "Goodbye World"

In this code firstly value of lazy_meesage will be set to Hello World. Then file /tmp/lazy.txt will get generated then makecache block will ge executed which in turn will call the file “message” block. But the file content will be wriiten at last due to lazy tag. Then package bind-utils will be installed but before that yum makecache will be called.
Then execute kitchen converge
kitchen login
lazy.txt will contain “Hello World” and message.txt will contain “Goodbye world”



ElasticSearch Issues

  • java.lang.IllegalArgumentException: unknown setting [node.rack] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Node level attributes used for allocation filtering, forced awareness or other node identification / grouping must be prefixed with node.attr. In previous versions it was possible to specify node attributes with the node. prefix. All node attributes except of and node.ingest must be moved to the new node.attr. namespace.

  • Unknown setting mlockall

Replace the bootstrap.mlockall with bootstrap.memory_lock

  • Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

Edit:  /etc/security/limits.conf and add the following lines

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Edit: /usr/lib/systemd/system/elasticsearch.service uncomment the line


Execute the following commands:

systemctl daemon-reload

systemctl elasticsearch start

  • Elasticsearch cluster health “red”: “unassigned_shards”

Execute the following command:

Elasticsearch’s cat API will tell you which shards are unassigned, and why:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

Each row lists the name of the index, the shard number, whether it is a primary (p) or replica ® shard, and the reason it is unassigned:

constant-updates        0 p UNASSIGNED NODE_LEFT node_left[NODE_NAME]

If the unassigned shards belong to an index you thought you deleted already, or an outdated index that you don’t need anymore, then you can delete the index to restore your cluster status to green:

curl -XDELETE 'localhost:9200/index_name/'
  • ElasticSearch nodes not showing hardware metrics

Execute the following command:

curl localhost:9200/_nodes/stats?pretty

It will show you the error root cause. If the error is:

“failures” : [
“type” : “failed_node_exception”,
“reason” : “Failed node [3kOQUA2IQ-mnD74ER3O6SQ]”,
“caused_by” : {
“type” : “illegal_state_exception”,
“reason” : “environment is not locked”,
“caused_by” : {
“type” : “no_such_file_exception”,
“reason” : “/opt/apps/elasticsearch/nodes/0/node.lock”

Then just restart elasticsearch service. It is caused when data directory is deleted while elasticsearch is still running.

  • Elasticsearch service does not start and no logs are captured in elasticsearch.log
    The issue can be found in /var/log/messages, mainly this issue is because of java not installed ot JAVA_HOME not set.
    The issue might also be because improper jvm settings.


Salt stack issues

  • The function “state.apply” is running as PID

Restart salt-minion with command: service salt-minion restart

  • No matching sls found for ‘init’ in env ‘base’

Add top.sls file in the directory where your main sls file is present.

Create the file as follows:

- apache

If the sls is present in a subdirectory elasticsearch/init.sls then write the top.sls as:

- elasticsearch.init
  • How to execute saltstack-formulas
    1. create file /srv/pillar/top.sls with content:
        - salt
    1. create file /srv/pillar/salt.sls with content:
        worker_threads: 2
          - roots
          - git
          - git://
          - git://
          - git://
          - git://
          - git://
          - git://
          - git://
          - git://
              - .*
              - '@runner'
              - '@wheel'
            - /srv/salt
            - /srv/pillar
          level: 'debug'
          server: 'gevent'
          host: ''
          port: '8080'
          cors: False
          tls: True
          certpath: '/etc/pki/tls/certs/localhost.crt'
          keypath: '/etc/pki/tls/certs/localhost.key'
          pempath: '/etc/pki/tls/certs/localhost.pem'
        master: localhost
    1. before you can use saltstack-formula you need to make one change to /etc/salt/master and add next config:
      - roots
      - git
      - git://
    1. restart salt-master (e.g. service salt-master restart)
    2. run salt-call state.sls salt.master
  • The Salt Master has cached the public key for this node

Execute the following command:

delete the exiting key on master by:

salt-key -d <minion-id>

then restart minion. Then reaccept the key on master:

salt-key -a <minion-id>

  • If salt-cloud is giving error as below:

Missing dependency: ‘netaddr’. The openstack driver requires ‘netaddr’ to be installed.

Execute the command: yum install python-netaddr

then verify if your provider is loaded with command: salt-cloud –list-providers

  • Remove dead minions keys in salt

salt-run manage.down removekeys=True

Diamond installation on centos 7

$ yum install make rpm-build python-configobj python-setuptools
$ git clone
$ cd Diamond
$ make buildrpm
Then use the package you built like this:

$ yum localinstall –nogpgcheck dist/diamond-4.0.449-0.noarch.rpm
$ cp /etc/diamond/{diamond.conf.example,diamond.conf}
$ $EDITOR /etc/diamond/diamond.conf
# Start Diamond service via service manager.

$ service diamond start

diamond-setup -C ElasticSearchCollector
diamond-setup -C NetworkCollector


  1. failed to connect socket to ‘/var/run/libvirt/libvirt-sock-ro’ no such file or directory

execute: egrep ‘(vmx|svm)’ /proc/cpuinfo

2. If the above commands returns with any output showing vmx or svm then your hardware supports VT else it does not.

yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer