Chef: Local Cookbook Development

 

Generators
generator1generator2generator3

  1. Execute the command as follows:
    chef generate cookbook test_cookbook
    cd test_cookbook
    Take a look at some of the default files like README.md, spec-helper.rb, default.rb file
  2. We will now create a generator that we will use to create cookbook with our customization:
    mkdir generator
    chef generate generator generator/lcd_origin
    cd generator/lcd_origin/templates/default
    vi README.md.erb insert text of your choice. Save and close.
  3. Edit kitchen.yml.erb change the contents to match the following, as we will be using docker driver and centos 7.2 image:
    ---
    driver:
      name: docker
      privileged: true
      use_sudo: false
    
    provisioner:
      name: chef_zero
      # You may wish to disable always updating cookbooks in CI or other testing environments.
      # For example:
      #   always_update_cookbooks: <%%= !ENV['CI'] %>
      always_update_cookbooks: true
    
    verifier:
      name: inspec
    
    platforms:
      - name: centos-7.2
        driver_config:
          run_command: /usr/lib/systemd/systemd
    
    suites:
      - name: default
        run_list:
          - recipe[<%= cookbook_name %>::default]
        verifier:
          inspec_tests:
            - test/smoke/default
        attributes:
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
    

    Make sure you are not adding any tabs as they are invalid in ymls.

  4. Edit ../../files/default/spec_helper.rb Add the following lines:
    RSpec.configure do |config|
      config.platform = 'centos'
      config.version = '7.2.1511'
    end<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
    
  5. So to tell chef to use our generator when generating cookbooks:
    goto the lcd_origin directory
    mkdir ~/.chef
    vi ~/.chef/config.rb Add the contents as below:

    cookbook_path ['~/chef/cookbooks']
    local_mode true
    if File.basename($PROGRAM_NAME).eql?('chef') && ARGV[0].eql?('generate')
      chefdk.generator.license = "all_rights"
      chefdk.generator.copyright_holder = "Student Name"
      chefdk.generator_cookbook = "/root/cookbooks/lcd_origin"
      chefdk.generator.email = "you@example.com"
      chefdk.generator_cookbook = "~/generator/lcd_origin"
    end
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
    

    Create directory chef/cookbooks
    mkdir -p ~/chef/cookbooks
    And then generate a cookbook
    cd ~/chef
    chef generate cookbook cookbooks/lcd_web
    If you check the README vi cookbooks/lcd_web/README.md. You will see the text we added previously.
    We can generate attributes by following command:
    chef generate attribute cookbooks/lcd_web default
    We can genrate recipe by:
    chef generate recipe cookbooks/lcd_web users

  6. These attributes can be checked here in lcd_origin/attributes/default.rb
    The recipe can be checked with lcd_origin/recipes/users.rb

Test Driven Development

tdd1.png
tdd2tdd3tdd4tdd5

ChefSpec

chefspec1chefspec2chefspec3chefspec4chefspec5

    1. You can check the chefspec default file  spec/unit/recipes/default_spec.rb
    2. Now let’s run the default_spec
      chef exec rspec
      If you get any warning about platform and platform_version then change the following in your code:
      runner = ChefSpec::ServerRunner.new(platform: ‘centos’, version: ‘7.2.1511’)
      For some of you the platform and the platform_version might be already added. It might be ubuntu and 16.04 change it to centos and 7.2.1511 resp.
      Then execute the rspec again. This will give an output similar to below:
      Finished in 0.48374 seconds (files took 1.83 seconds to load)
      2 examples, 0 failures
    3. Now change the default_spec.rb to match the following to check if httpd package is installed or not:
      #
      # Cookbook:: lcd_web
      # Spec:: default
      #
      # Copyright:: 2017, Student Name, All Rights Reserved.
      
      require 'spec_helper'
      
      describe 'lcd_web::default' do
        context 'CentOS' do
          let(:chef_run) do
            # for a complete list of available platforms and versions see:
            # https://github.com/customink/fauxhai/blob/master/PLATFORMS.md
            runner = ChefSpec::ServerRunner.new(platform: 'centos', version: '7.2.1511')
            runner.converge(described_recipe)
          end
      
          it 'converges successfully' do
            expect { chef_run }.to_not raise_error
          end
      
          it 'installs httpd' do
            expect(chef_run).to install_package('httpd')
          end
        end
      end
      

      Then run chef exec rspec You will get an error as below:

      .F.

      Failures:

      1) lcd_web::default CentOS installs httpd
      Failure/Error: expect(chef_run).to install_package(‘httpd’)

      expected “package[httpd]” with action :install to be in Chef run. Other p ackage resources:

      # ./spec/unit/recipes/default_spec.rb:23:in `block (3 levels) in ‘

      Finished in 0.63225 seconds (files took 1.68 seconds to load)
      3 examples, 1 failure

      Failed examples:

      rspec ./spec/unit/recipes/default_spec.rb:22 # lcd_web::default CentOS installs httpd

      It failed because there was no httpd package present in the system because our cookbook doesn’t do so. Let’s add httpd installation to our default recipe.

      package ‘httpd’ do
      end
      As the default action is install we do not need to specify anything.
      Now let’s execute the rspec: chef exec rspec
      Now the output will be as follows:

      Finished in 0.73318 seconds (files took 1.67 seconds to load)
      3 examples, 0 failures

    4. Add the following below installs httpd task:
          it 'enables the httpd service' do
            expect(chef_run).to enable_service('httpd')
          end
      
          it 'starts the service' do
            expect(chef_run).to start_service('httpd')
          end
      
      Add the following in default.rb recipe:
      service 'httpd' do
        action [:start, :enable]
      end
      <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
      

      Then run: chef exec rspec

    5. Now what if I add something in the default recipe that might break my cookbook:
      add the line in default.rb
      include_recipe ‘something’
      Then run rspec you will get error as below:
      Failure/Error: runner.converge(described_recipe)Chef::Exceptions::CookbookNotFound:
      Cookbook something not found. If you’re loading something from another cookbook, make sure you configure the dependency in your metadata
      # /tmp/chefspec20170918-4764-5svx8rfile_cache_path/cookbooks/lcd_web/recipes/default.rb:7:in `from_file’
      # ./spec/unit/recipes/default_spec.rb:15:in `block (3 levels) in ‘
      # ./spec/unit/recipes/default_spec.rb:31:in `block (3 levels) in ‘Finished in 0.81827 seconds (files took 1.71 seconds to load)
      5 examples, 4 failuresFailed examples:rspec ./spec/unit/recipes/default_spec.rb:18 # lcd_web::default CentOS converges successfully
      rspec ./spec/unit/recipes/default_spec.rb:22 # lcd_web::default CentOS installs httpd
      rspec ./spec/unit/recipes/default_spec.rb:26 # lcd_web::default CentOS enables the httpd service
      rspec ./spec/unit/recipes/default_spec.rb:30 # lcd_web::default CentOS starts the service

So you can see the test will fail if the cookbook dependency failes as well. You can remove the include_recipe line and re execute the rspec.

Test Kitchen Configuration

testkit1testkit2testkit3testkit4testkit5testkit6testkit7testkit8testkit9testkit10testkit11

  1. Edit .kitchen.yml to match the following:
    ---
    driver:
      name: docker
      privileged: true
      use_sudo: false
    
    provisioner:
      name: chef_zero
      # You may wish to disable always updating cookbooks in CI or other testing environments.
      # For example:
      #   always_update_cookbooks: <%= !ENV['CI'] %>
      always_update_cookbooks: true
    
    verifier:
      name: inspec
    
    platforms:
      - name: centos-7.2
      - name: centos-6.8
    
    suites:
      - name: dev
        driver:
          run_command: /usr/sbin/init
        run_list:
          - recipe[la_java::default]
        attributes: { 'java': { 'jdk_version': '7' } }
        driver_config:
          forward:
            - 8080:80
        excludes: centos-6.8
      - name: prod
        driver:
          run_command: /sbin/init
        run_list:
          - recipe[la_java::default]
        attributes: { 'java': { 'jdk_version': '6' } }
        driver_config:
          forward:
            - 8081:80
        includes: centos-6.8
    <span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>
    

    Execute the command kitchen list
    If you get error as follows:
    >>>>>> ——Exception——-
    >>>>>> Class: Kitchen::ClientError
    >>>>>> Message: Could not load the ‘docker’ driver from the load path. Please en sure that your driver is installed as a gem or included in your Gemfile if using Bundler.
    >>>>>> ———————-
    >>>>>> Please see .kitchen/logs/kitchen.log for more details
    >>>>>> Also try running `kitchen diagnose –all` for configuration

    Then install the chef gem kitchen-docker with:  chef gem install kitchen-docker
    Then the kitchen list output will be as follows:
    [root@arati7111 lcd_web]# kitchen list
    Instance Driver Provisioner Verifier Transport Last Action Last Error
    dev-centos-72 Docker ChefZero Inspec Ssh
    prod-centos-68 Docker ChefZero Inspec Ssh
    Then run: kitchen converge
    Then kitchen verify
    Thus java 7 will be installed in centos 7 and java 6 on centos 6

  2. If you comment the includes and excludes statements from .kitchen.yml then we will have 4 test cases.
  3. You can download the java cookbook with
    knife cookbook site download
    and add it to the run_list

Using Test Kitchen

kitchen3kitchen4kitchen5

Execute the following commands

  1. mkdir test_kitchen
  2. cd test_kitchen
  3. kitchen init
    If you get the following error while executing kitchen init:Successfully installed kitchen-vagrant-1.2.1
    Fetching: mixlib-install-3.6.0.gem (100%)
    Successfully installed mixlib-install-3.6.0
    Fetching: ffi-1.9.18.gem (100%)
    Building native extensions. This could take a while…
    ERROR: Error installing kitchen-vagrant:
    ERROR: Failed to build gem native extension./usr/bin/ruby extconf.rb
    mkmf.rb can’t find header files for ruby at /usr/share/include/ruby.h

Gem files will remain installed in /home/user/.gem/ruby/gems/ffi-1.9.18 for inspection.
Results logged to /home/user/.gem/ruby/gems/ffi-1.9.18/ext/ffi_c/gem_make.out

Then execute the kitchen init command with sudo rights

  • If even then the issue is not resolved execute the following command:
    chef gem list kitchen-vagrant
    Then you will have output as follows:  kitchen-vagrant (1.2.1)
    chef gem uninstall kitchen-vagrant -v 1.2.1
    Then rerun kitchen init with sudo rights
  • chef generate cookbook my_cookbook
    You can see that the test/integration/default directory
    In the cookbook you can see the test/smoke/default directory
  • Execute kitchen list
    If you get an error as follows:
    >>>>>> ——Exception——-
    >>>>>> Class: Kitchen::UserError
    >>>>>> Message: Vagrant 1.1.0 or higher is not installed. Please download a package from http://www.vagrantup.com/downloads.html.
    >>>>>> ———————-
    That means you will have to install vagrant. you can use the following command:
    rpm -ivh https://releases.hashicorp.com/vagrant/2.0.0/vagrant_2.0.0_x86_64.rpmThen again execute kitchen list  you will get an output as below:
    Instance Driver Provisioner Verifier Transport Last Action Last Error
    default-ubuntu-1404 Vagrant ChefSolo Busser Ssh
    default-centos-72 Vagrant ChefSolo Busser Ssh
    kitchen7kitchen8kitchen9
  • Execute kitchen create
    You will get an error as follows: The provider ‘virtualbox’ that was requested to back the machine ‘default’
    You will need to install virtualbox first: Follow this tutorial for the same:
    https://apassionatechie.wordpress.com/2017/10/09/install-virtualbox-on-centos-7/kitchen10.png
    If you are not able to install vbox then just change the driver name from vagrant to docker in .kitchen.yml

Then execute kitchen converge
kitchen11.png
Then execute kitchen setup
Then you can execute kitchen destroy to delete the created instances
kitchen13kitchen14

InSpec
inspec1inspec2inspec3inspec4inspec5inspec6inspec7inspec8

Create a cookbook with chef generate cookbook lcd_web. Edit the following file test/smoke/default/default_test.rb. Add the following lines
[ ‘net-tools’, ‘httpd’ ].each do |pkg|
describe package(pkg) do
it { should be_installed }
end
end

Then execute kitchen verify it will fail the test as below:

System Package
∅ net-tools should be installed
expected that `System Package net-tools` is installed
System Package
∅ httpd should be installed
expected that `System Package httpd` is installed

Now lets add the chef code to install these packages
Add the following lines in recipes/default.rb
[ ‘net-tools’, ‘httpd’].each do |pkg|
package pkg do
action :install
end
end

Then execute kitchen converge
Then run kitchen verify
The output should be somthing as below:
System Package
✔ net-tools should be installed
System Package
✔ httpd should be installed

Static Code Analysis

static1static2static3static4static5static6static7static8static9
foodcritic -l
FC001: Use strings in preference to symbols to access node attributes
FC002: Avoid string interpolation where not required
FC004: Use a service resource to start and stop services
FC005: Avoid repetition of resource declarations
FC006: Mode should be quoted or fully specified when setting file permissions
FC007: Ensure recipe dependencies are reflected in cookbook metadata
FC008: Generated cookbook metadata needs updating
FC009: Resource attribute not recognised
FC010: Invalid search syntax
FC011: Missing README in markdown format
FC012: Use Markdown for README rather than RDoc
FC013: Use file_cache_path rather than hard-coding tmp paths
FC014: Consider extracting long ruby_block to library
FC015: Consider converting definition to a Custom Resource
FC016: LWRP does not declare a default action
FC017: LWRP does not notify when updated
FC018: LWRP uses deprecated notification syntax

Now cd to lcd_web cookbook
execute: foodcritic .
output: Checking 2 files
x.
FC008: Generated cookbook metadata needs updating: ./metadata.rb:3
FC064: Ensure issues_url is set in metadata: ./metadata.rb:1
FC065: Ensure source_url is set in metadata: ./metadata.rb:1
FC067: Ensure at least one platform supported in metadata: ./metadata.rb:1
FC078: Ensure cookbook shared under an OSI-approved open source license: ./metadata.rb:1

Lets check our metadata.rb file
For error FC067 add the line in metadata.rb: supports ‘centos’
The other errors are not of our concern so we will exclude them with:
foodcritic . -t ~FC008 -t ~FC064 -t ~FC065 -t ~FC078
There should be no errors.

Now we want to persist these exclusion so we will create .foodcritic file:
vi .foodcritic and add the following lines:

~FC008

~FC064

~FC065

~FC078
Save and close. Then execute foodcritic .

Lets have a look at rubocop. To install rubocop execute gem install rubocop

You can have a look at it’s help with command rubocop –help

If you run rubocop in lcd_we cookbook you might get output as follows:

init_test/lcd_web/recipes/default.rb:7:2: C: Space inside square brackets detected.
[ ‘net-tools’, ‘httpd’].each do |pkg|
^
bin/mixlib-install:11:11: C: Prefer single-quoted strings when you don’t need string interpolation or special symbols.
version = “>= 0”
^^^^^^
bin/mixlib-install:15:32: C: Prefer single-quoted strings when you don’t need string interpolation or special symbols.
str = str.dup.force_encoding(“BINARY”) if str.respond_to? :force_encoding
^^^^^^^^
bin/mixlib-install:17:15: C: Avoid the use of Perl-style backrefs.
version = $1
^^

32 files inspected, 53 offenses detected

You might have errors like “line too long” etc. By default the line length specified in 80, Now let’s see if we can override that setting.
We will be needing todo file for that let’s generate it by the command:
rubocop –auto-gen-config
Then edit the file .rubocop.yml and add the line: inherit_from: .rubocop_todo.yml
Then execute rubocop you will see that all of our offenses have been masked.
Let’s have a look at the .rubocop_todo.yml file.
Now in the code you will see code blocks like follows:

# Offense count: 1
# Cop supports --auto-correct.
# Configuration parameters: EnforcedStyle, SupportedStyles.
# SupportedStyles: auto_detection, squiggly, active_support, powerpack, unindent
Layout/IndentHeredoc:
  Exclude:
    - 'generator/lcd_origin/recipes/recipe.rb'

This denotes that there i an indentation issue with recipe.rb file.
rubocop has the feature to auto-correct small issues, you will need to comment code block one by one and then execute rubocop –auto-correct
Output will be something like: generator/lcd_origin/metadata.rb:5:1: C: [Corrected] 1 trailing blank lines detected.

So in this way we can correct some issues with rubocop. You can repeat the same for rest of the issues.
For the other issues which cannot be correct with rubo-cop you can add rules for the same in .rubocop.yml

We will change the LineLength rule. Add the following in .rubocop.yml file:
LineLength:
Max: 200
Then comment the LineLength block in .rubocop_todo.yml file.

Then execute rubocop You will see there are no offenses.

Troubleshooting
ts1ts2ts3ts4ts5ts6

Edit file recipes/default.rb. Comment everything and add the following:

lazy_message = "Hello World"

file 'lazy_message' do
  path '/tmp/lazy.txt'
  content "#{lazy_message}"
end

execute 'yum-makecache' do
  command 'yum makecache'
  notifies :create, 'file[message]', :immediately
  action :nothing
end

package 'bind-utils' do
  action :install
  notifies :run, 'execute[yum-makecache]', :before
end

file 'message' do
  path '/tmp/message.txt'
  content lazy { "#{lazy_message}" }
end

lazy_message = "Goodbye World"

In this code firstly value of lazy_meesage will be set to Hello World. Then file /tmp/lazy.txt will get generated then makecache block will ge executed which in turn will call the file “message” block. But the file content will be wriiten at last due to lazy tag. Then package bind-utils will be installed but before that yum makecache will be called.
Then execute kitchen converge
kitchen login
lazy.txt will contain “Hello World” and message.txt will contain “Goodbye world”

 

Advertisements

ElasticSearch Issues

  • java.lang.IllegalArgumentException: unknown setting [node.rack] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Node level attributes used for allocation filtering, forced awareness or other node identification / grouping must be prefixed with node.attr. In previous versions it was possible to specify node attributes with the node. prefix. All node attributes except of node.masternode.data and node.ingest must be moved to the new node.attr. namespace.

  • Unknown setting mlockall

Replace the bootstrap.mlockall with bootstrap.memory_lock

  • Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

Edit:  /etc/security/limits.conf and add the following lines

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Edit: /usr/lib/systemd/system/elasticsearch.service uncomment the line

LimitMEMLOCK=infinity

Execute the following commands:

systemctl daemon-reload

systemctl elasticsearch start

  • Elasticsearch cluster health “red”: “unassigned_shards”

Execute the following command:

Elasticsearch’s cat API will tell you which shards are unassigned, and why:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

Each row lists the name of the index, the shard number, whether it is a primary (p) or replica ® shard, and the reason it is unassigned:

constant-updates        0 p UNASSIGNED NODE_LEFT node_left[NODE_NAME]

If the unassigned shards belong to an index you thought you deleted already, or an outdated index that you don’t need anymore, then you can delete the index to restore your cluster status to green:

curl -XDELETE 'localhost:9200/index_name/'
  • ElasticSearch nodes not showing hardware metrics

Execute the following command:

curl localhost:9200/_nodes/stats?pretty

It will show you the error root cause. If the error is:

“failures” : [
{
“type” : “failed_node_exception”,
“reason” : “Failed node [3kOQUA2IQ-mnD74ER3O6SQ]”,
“caused_by” : {
“type” : “illegal_state_exception”,
“reason” : “environment is not locked”,
“caused_by” : {
“type” : “no_such_file_exception”,
“reason” : “/opt/apps/elasticsearch/nodes/0/node.lock”
}
}

Then just restart elasticsearch service. It is caused when data directory is deleted while elasticsearch is still running.

  • Elasticsearch service does not start and no logs are captured in elasticsearch.log
    The issue can be found in /var/log/messages, mainly this issue is because of java not installed ot JAVA_HOME not set.
    The issue might also be because improper jvm settings.

 

Salt stack issues

  • The function “state.apply” is running as PID

Restart salt-minion with command: service salt-minion restart

  • No matching sls found for ‘init’ in env ‘base’

Add top.sls file in the directory where your main sls file is present.

Create the file as follows:

base:
'web*':
- apache

If the sls is present in a subdirectory elasticsearch/init.sls then write the top.sls as:

base:
'*':
- elasticsearch.init
  • How to execute saltstack-formulas
    1. create file /srv/pillar/top.sls with content:
    base:
      '*':
        - salt
    1. create file /srv/pillar/salt.sls with content:
    salt:
      master:
        worker_threads: 2
        fileserver_backend:
          - roots
          - git
        gitfs_remotes:
          - git://github.com/saltstack-formulas/epel-formula.git
          - git://github.com/saltstack-formulas/git-formula.git
          - git://github.com/saltstack-formulas/nano-formula.git
          - git://github.com/saltstack-formulas/rabbitmq-formula.git
          - git://github.com/saltstack-formulas/remi-formula.git
          - git://github.com/saltstack-formulas/vim-formula.git
          - git://github.com/saltstack-formulas/salt-formula.git
          - git://github.com/saltstack-formulas/users-formula.git
        external_auth:
          pam:
            tiger:
              - .*
              - '@runner'
              - '@wheel'
        file_roots:
          base:
            - /srv/salt
        pillar_roots:
          base:
            - /srv/pillar
        halite:
          level: 'debug'
          server: 'gevent'
          host: '0.0.0.0'
          port: '8080'
          cors: False
          tls: True
          certpath: '/etc/pki/tls/certs/localhost.crt'
          keypath: '/etc/pki/tls/certs/localhost.key'
          pempath: '/etc/pki/tls/certs/localhost.pem'
      minion:
        master: localhost
    1. before you can use saltstack-formula you need to make one change to /etc/salt/master and add next config:
    fileserver_backend:
      - roots
      - git
    gitfs_remotes:
      - git://github.com/saltstack-formulas/salt-formula.git
    1. restart salt-master (e.g. service salt-master restart)
    2. run salt-call state.sls salt.master
  • The Salt Master has cached the public key for this node

Execute the following command:

delete the exiting key on master by:

salt-key -d <minion-id>

then restart minion. Then reaccept the key on master:

salt-key -a <minion-id>

  • If salt-cloud is giving error as below:

Missing dependency: ‘netaddr’. The openstack driver requires ‘netaddr’ to be installed.

Execute the command: yum install python-netaddr

then verify if your provider is loaded with command: salt-cloud –list-providers

  • Remove dead minions keys in salt

salt-run manage.down removekeys=True

Diamond installation on centos 7

$ yum install make rpm-build python-configobj python-setuptools
$ git clone https://github.com/python-diamond/Diamond
$ cd Diamond
$ make buildrpm
Then use the package you built like this:

$ yum localinstall –nogpgcheck dist/diamond-4.0.449-0.noarch.rpm
$ cp /etc/diamond/{diamond.conf.example,diamond.conf}
$ $EDITOR /etc/diamond/diamond.conf
# Start Diamond service via service manager.

$ service diamond start

diamond-setup -C ElasticSearchCollector
diamond-setup -C NetworkCollector

Issues

  1. failed to connect socket to ‘/var/run/libvirt/libvirt-sock-ro’ no such file or directory

execute: egrep ‘(vmx|svm)’ /proc/cpuinfo

2. If the above commands returns with any output showing vmx or svm then your hardware supports VT else it does not.

yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer