Archive

Archive for June, 2016

Cloudera Manager Setup Using Chef [CentOS 6.6]

June 24, 2016 Leave a comment
This cookbook [currently as of now] can be used to setup a Cloudera Manager Server (Management Server) running on MySQL database.
But the intended use for this cookbook [rather a wishlist] is to do more. Simply put Auto Deployment of a Cloudera Hadoop Cluster using Chef, Python and Cloudera API. This will help create cluster for a development/test/preproduction/production environment on a click of a button.
  • Attributes
  • Recipe
  • Usage

Get the Cookbook.

Can be downloaded from the link. Cloudera Manager Setup

How would the Setup Look like.

Nodes and the runlist which will be assigned.
  • Cloudera Manager – Runlist cm_setup default runlist which will include all the installations.
  • All Other nodes – Runlist, will be assigning the Role base_node_install which we will create below.

What does cm_setup default cookbook have.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'
Creating a mysql setup. Setting up cloudera-manager on mysql.
include_recipe 'cm_setup::mysql_setup'
  include_recipe 'cm_setup::mysql_install'
  include_recipe 'cm_setup::mysql_configure'
  include_recipe 'cm_setup::mysql_user_setup
Installing mysql_connector.
include_recipe 'cm_setup::mysql_connector_setup'
Installing cloudera-daemons and agent.
include_recipe 'cm_setup::cloudera_install_setup'
  configuring database using the db script.
  Autostarting `cloudera-scm-server`.

What does the base_node_install Role have.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'

Role.

To setup non-mgmt nodes we can create a role and assign the nodes this role, so that the base setup on that node is completed.
Below is a JSON for the Role for base setup.
{
   "name": "base_node_install",
   "description": "Base Installation for Node other than the Clouder Manager Node.",
   "json_class": "Chef::Role",
   "default_attributes": {

   },
   "override_attributes": {

   },
   "chef_type": "role",
   "run_list": [
     "recipe[cm_setup::commons]",
     "recipe[cm_setup::sysctl_setup]",
     "recipe[cm_setup::ntpd_setup]",
     "recipe[cm_setup::users_setup]",
     "recipe[cm_setup::hostfile_setup]"
   ],
   "env_run_lists": {

   }
}
Creating role on the Chef Server.
┌─[ahmed][zubair-HP-ProBook][±][master U:2 ✗][~/work/chef-repo]
└─▪ knife role create base_node_install
Add the contents above to the role. Once we are done then we can list then using below command.
┌─[ahmed][zubair-HP-ProBook][±][master U:2 ✗][~/work/chef-repo]
└─▪ knife role list
base_node_install
testrole
Now we add the role to each on the nodes which act as a managed node like a namenode, standbynamenode, resourcemanager etc.

Before we assign the Role.

Before we assign the role, we need to bootstrap the node so that it is added to the Chef Server.
Below is the command to add the node to the Chef Server.
knife bootstrap  --ssh-port  --ssh-user  --ssh-password  --sudo
NOTE : The user we use should have sudo privileges so that chef can install the client on the node.
Here is the output for the vagrant node which was used to test the Cookbook.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ knife bootstrap 127.0.0.1 --ssh-port 2222 --ssh-user vagrant --ssh-password vagrant --sudo
Doing old-style registration with the validation key at /home/ahmed/work/chef-repo/.chef/happy-minds-validator.pem...
Delete your validation key in order to use your user credentials instead

Connecting to 127.0.0.1
127.0.0.1 -----> Installing Chef Omnibus (-v 12)
127.0.0.1 downloading http://ift.tt/1WbhOaj
127.0.0.1   to file /tmp/install.sh.3341/install.sh
127.0.0.1 trying wget...
127.0.0.1 el 6 x86_64
127.0.0.1 Getting information for chef stable 12 for el...
127.0.0.1 downloading http://ift.tt/292cTot
127.0.0.1   to file /tmp/install.sh.3361/metadata.txt
127.0.0.1 trying wget...
127.0.0.1 sha1    44e71beed0cc0db2481c3e3d2108ad218c32dade
127.0.0.1 sha256    e51559dc7747c03b446f9d1a3cdbb122f274352ba0ed7dd8fdac41e10514b9e2
127.0.0.1 url    http://ift.tt/28TZemS
127.0.0.1 version    12.11.18
127.0.0.1 downloaded metadata file looks valid...
127.0.0.1 downloading http://ift.tt/28TZemS
127.0.0.1   to file /tmp/install.sh.3361/chef-12.11.18-1.el6.x86_64.rpm
127.0.0.1 trying wget...
127.0.0.1 Comparing checksum with sha256sum...
127.0.0.1 Installing chef 12
127.0.0.1 installing with rpm...
127.0.0.1 warning: /tmp/install.sh.3361/chef-12.11.18-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
127.0.0.1 Preparing...                                                            (1########################################### [100%]
127.0.0.1    1:chef                                                               ( ########################################### [100%]
127.0.0.1 Thank you for installing Chef!
127.0.0.1 Starting the first Chef Client run...
127.0.0.1 Starting Chef Client, version 12.11.18
127.0.0.1 Creating a new client identity for localhost.localdomain using the validator key.
127.0.0.1 resolving cookbooks for run list: []
127.0.0.1 Synchronizing Cookbooks:
127.0.0.1 Installing Cookbook Gems:
127.0.0.1 Compiling Cookbooks...
127.0.0.1 [2016-06-24T15:25:45+02:00] WARN: Node localhost.localdomain has an empty run list.
127.0.0.1 Converging 0 resources
127.0.0.1
127.0.0.1 Running handlers:
127.0.0.1 Running handlers complete
127.0.0.1 Chef Client finished, 0/0 resources updated in 09 seconds

Logon to Chef Server and Edit Run List.

Chef Server and Edit Run List

Select Role to be Assigned.

Select Role to be Assigned

Assigned Role and Save.

Assigned Role and Save

Chef Role is Assigned.

Chef Role is Assigned

Executing sudo chef-client on Node.

Executing sudo chef-client on Node

Attributes.

Below are the set of attributes which can be changed as per requirement.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ tree
.
|____hosts_attr.rb
|____default.rb
|____mysql_attr.rb
|____sudo_attr.rb
|____cdh_attr.rb
|____sysctl_attr.rb
|____security_sssd_attr.rb
|____security_krb5_attr.rb
|____ntp_attr.rb

hosts_attr File

This file has the host information which need to be populated in the /etc/hosts file.
hostsfile cookbook http://ift.tt/1acwcZi.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat hosts_attr.rb

#
# Server informatoin for the `/etc/hosts` file changes this as required
#

default['etc_hosts_entries']['9.1.1.1']['hostname'] = 'server9.ahmed.com'
default['etc_hosts_entries']['9.1.1.1']['aliases']  = ['server9']
default['etc_hosts_entries']['9.1.1.1']['comment']  = 'Server9'
default['etc_hosts_entries']['9.1.1.1']['action']   = :create_if_missing

mysql_attr File.

This has parameters related to mysql more attributes can be twicked more information can be found on the base cookbook mysql, mysql_connector, database.
mysql cookbook for creating the mysql instance. mysql_connector cookbook for creating the connector. database cookbook to create database and database users.
File here.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat mysql_attr.rb
#
# MySQL connector attributes
# => http://ift.tt/28TZfqV
# => http://ift.tt/28TZfHv
#

default['mysql_connector']['j']['install_paths'] = ['/usr/share/java']
default['mysql_connector']['j']['version'] = '5.1.36'

#
# MySQL user, Configuration and services
#
# #
# # Installing `mysqld`
# # => http://ift.tt/1wkd5Cm
# # => http://ift.tt/292cSRs
# #
#
# #
# # Setting up user for the mysql database.
# # => http://ift.tt/292cMJL
# # => http://ift.tt/28TZH8i
# #
#
#

default['mysql']['configuring']['database_service_name'] = 'default'
default['mysql']['configuring']['database_name'] = 'cmdb'
default['mysql']['configuring']['database_root_password'] = 'root@123'

default['mysql']['configuring']['database_user'] = 'cmadmin'
default['mysql']['configuring']['database_password'] = 'cmadmin@123'
default['mysql']['configuring']['database_user_privileges'] = [:all]
default['mysql']['configuring']['database_user_privileges_host'] = '%'

default['mysql']['configuring']['host_ip'] = '127.0.0.1'
default['mysql']['configuring']['port'] = '3306'
default['mysql']['configuring']['version'] = '5.5'

hostsfile File

Creating users and sudo users on the server.
Config File.
┌─[ahmed][zubair-HP-ProBook][±][master U:3 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat sudo_attr.rb
#
# Adding sudo attributes
#

#
# User Setup
#
# #
# # Creating a admin user/group for clouderamanager
# # http://ift.tt/292cPVZ
# # http://ift.tt/28TZodR
#

default['users_setup']['groups'] = { 'sysadmin' => 2300, 'cmadmin' => 2301 }

#
# Creating sudo users
# => http://ift.tt/292cWkr
# => http://ift.tt/28TZaU5
#

default['authorization']['sudo']['groups'] = ['cmadmin', 'sysadmin']
default['authorization']['sudo']['users'] = ['cmadmin', 'vagrant', 'sysadminuser']
default['authorization']['sudo']['passwordless'] = true

cdh_attr.rb File

Here we can configure information related to cdh.
  • Creating Repository
  • Installation Packages
  • Services
yum cookbook http://ift.tt/292cYc7
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat cdh_attr.rb

#
# Cloudera Manager installation and services
#

default['cdh_install']['install_packages'] = [
                                              'oracle-j2sdk1.7',
                                              'cloudera-manager-daemons',
                                              'cloudera-manager-server'
                                            ]
default['cdh_install']['cm_services'] = [
                                          'cloudera-scm-server'
                                        ]

#
# Repository Configuration
#
# # Setting up repos
# # => http://ift.tt/1SK7O1u
# # => http://ift.tt/292cYc7
# #
#

# description 'Extra Packages for Enterprise Linux'
# mirrorlist 'http://ift.tt/1lrmSD0'
# gpgkey 'http://ift.tt/1ijI09P'

default['yum_repository']['epel']['description'] = 'Extra Packages for Enterprise Linux'
default['yum_repository']['epel']['mirrorlist'] = 'http://ift.tt/1lrmSD0'
default['yum_repository']['epel']['gpgkey'] = 'http://ift.tt/1ijI09P'

# description 'Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 '
# baseurl 'http://ift.tt/292cQsL'
# gpgkey 'http://ift.tt/28TZguZ'

default['yum_repository']['cm']['description'] = 'Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 '
default['yum_repository']['cm']['baseurl'] = 'http://ift.tt/292cQsL'
default['yum_repository']['cm']['gpgkey'] = 'http://ift.tt/28TZguZ'

sysctl_attr File.

This file is to update sysctl.conf. All attributes are from the sysctl cookbook.
sysctl cookbook http://ift.tt/292cO4l
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat sysctl_attr.rb

#
# Setting up custom sysctl configuration
# TODO: We need to make the parameter to be added from attributes.
#
# template '/etc/sysctl.conf' do
#   source 'sysctl.conf.erb'
# end
#
# Setting up sysctl.conf
# => http://ift.tt/28TZuSM
# => http://ift.tt/292cO4l
#

security_sssd_attr File.

This file is to setup (install and configure) sssd on the node.
sysctl_ldap cookbook http://ift.tt/292cRNp
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat security_sssd_attr.rb
#
# SSSD installation and configuration
#
#

#
# Installing and configuring SSSD
# => http://ift.tt/28TZlie
# => http://ift.tt/292cRNp
#

security_krb5_attr File.

This file attributes are to install and configure krb5 for a node.
krb5 cookbook http://ift.tt/292cRx5
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat security_krb5_attr.rb
#
# krb5 installation and configuration
#

#
# Installing and configuring krb5
# => http://ift.tt/28TZwtW
# => http://ift.tt/292cRx5
#

ntp_attr File.

Setting up ntp on a node.
┌─[ahmed][zubair-HP-ProBook][±][master U:5 ✗][~/work/chef-repo/cookbooks/cm_setup/attributes]
└─▪ cat ntp_attr.rb
#
# configuring servers
# => http://ift.tt/292cNgK
# => http://ift.tt/28TZfHs
#

default['ntp']['mode_servers'] = ['0.pool.ntp.org', '1.pool.ntp.org']

default Recipe Details.

Common installations, like disable selinux and iptables.
# Setting up commons
include_recipe 'cm_setup::commons'
Setting up sysctl.conf configuration optimized for Hadoop.
include_recipe 'cm_setup::sysctl_setup'
Installing and configuring ntpd
include_recipe 'cm_setup::ntpd_setup'
Creating user(s) for cloudera-manager.
include_recipe 'cm_setup::users_setup'
Creating /etc/hosts file as required by Hadoop cluster.
include_recipe 'cm_setup::hostfile_setup'
Creating a mysql setup. Setting up cloudera-manager on mysql.
include_recipe 'cm_setup::mysql_setup'
  include_recipe 'cm_setup::mysql_install'
  include_recipe 'cm_setup::mysql_configure'
  include_recipe 'cm_setup::mysql_user_setup
Installing mysql_connector.
include_recipe 'cm_setup::mysql_connector_setup'
Installing cloudera-daemons and agent.
include_recipe 'cm_setup::cloudera_install_setup'
  configuring database using the db script.
  Autostarting `cloudera-scm-server`.
Installation and Configuration of sssd. [ Unit Test complete – Need to do TEST on live environment ]
Installation and Configuration of krb5. [ Unit Test complete – Need to do TEST on live environment ]
Configuration of Cloudera Using Cloudera API. [TODO]

Usage.

Below are the steps to setup and environment to execute this cookbook.

git Clone the project.

git clone http://ift.tt/292cXF1

Update the .kitchen.yml file with below content [ if required – OPTIONAL ]

File can be found in the ${CHEF_COOKBOOK_HOME}/.kitchen.yml.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ cat .kitchen.yml
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

# Uncomment the following verifier to leverage Inspec instead of Busser (the
# default verifier)
# verifier:
#   name: inspec

platforms:
  - name: grtjn/centos-6.5

suites:
  - name: default
    run_list:
      - recipe[cm_setup::default]
    attributes:

Check the for the vagrant box which will be used.

Command
kitchen list
Output
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/starter]
└─▪ kitchen list
Instance                 Driver   Provisioner  Verifier  Transport  Last Action
default-grtjn-centos-65  Vagrant  ChefSolo     Busser    Ssh        

create node.

Command
kitchen create
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:1 ?:3 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen create
-----> Starting Kitchen (v1.8.0)
-----> Creating ...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'grtjn/centos-6.5'...
==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'grtjn/centos-6.5' is up to date...
       ==> default: Setting the name of the VM: kitchen-starter-default-grtjn-centos-65_default_1466270503111_60773
       ==> default: Fixed port collision for 22 => 2222. Now on port 2200.
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 (guest) => 2200 (host) (adapter 1)
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: 127.0.0.1:2200
           default: SSH username: vagrant
           default: SSH auth method: private key
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
           default: The guest additions on this VM do not match the installed version of
           default: VirtualBox! In most cases this is fine, but in rare cases it can
           default: prevent things such as shared folders from working properly. If you see
           default: shared folder errors, please make sure the guest additions within the
           default: virtual machine match the version of VirtualBox you have installed on
           default: your host and reload your VM.
           default:
           default: Guest Additions Version: 4.3.8
           default: VirtualBox Version: 5.0
       ==> default: Setting hostname...
       ==> default: Machine not provisioned because `--no-provision` is specified.
       [SSH] Established
       Vagrant instance  created.
       Finished creating  (0m52.53s).
-----> Kitchen is finished. (0m52.65s)

Login to the node.

Command
kitchen login
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:1 ?:3 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen login
Last login: Sat Jun 18 17:22:13 2016 from 10.0.2.2
[vagrant@default-grtjn-centos-65 ~]$ cat /etc/hosts
127.0.0.1   default-grtjn-centos-65 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[vagrant@default-grtjn-centos-65 ~]$ exit
logout
Connection to 127.0.0.1 closed.

converge – Cookbook with the node.

Command
kitchen converge
Output
┌─[ahmed][zubair-HP-ProBook][±][master ↑1 U:2 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen converge
-----> Starting Kitchen (v1.8.0)
-----> Converging ...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.3.3...
       Removing non-cookbook files before transfer
       Preparing data_bags
       Preparing validation.pem
       Preparing client.rb
-----> Installing Chef Omnibus (install only if missing)
       Downloading http://ift.tt/1yj6n3U to file /tmp/install.sh
       Trying wget...
       Trying curl...
       Download complete.
       el 6 x86_64
       Getting information for chef stable  for el...
       downloading http://ift.tt/292cJh2
         to file /tmp/install.sh.1983/metadata.txt
       trying wget...
       sha1 44e71beed0cc0db2481c3e3d2108ad218c32dade
       sha256 e51559dc7747c03b446f9d1a3cdbb122f274352ba0ed7dd8fdac41e10514b9e2
       url  http://ift.tt/28TZemS
       version  12.11.18
       downloaded metadata file looks valid...
       downloading http://ift.tt/28TZemS
         to file /tmp/install.sh.1983/chef-12.11.18-1.el6.x86_64.rpm
       trying wget...
       trying curl...
       Comparing checksum with sha256sum...

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       You are installing an omnibus package without a version pin.  If you are installing
       on production servers via an automated process this is DANGEROUS and you will
       be upgraded without warning on new releases, even to new major releases.
       Letting the version float is only appropriate in desktop, test, development or
       CI/CD environments.

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       Installing chef
       installing with rpm...
       warning: /tmp/install.sh.1983/chef-12.11.18-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
       Preparing...                                                            (100%########################################### [100%]
          1:chef                                                               (  1%########################################### [100%]
       Thank you for installing Chef!
       Transferring files to 
       Starting Chef Client, version 12.11.18
       Creating a new client identity for default-grtjn-centos-65 using the validator key.
       resolving cookbooks for run list: ["cm_setup::default"]
       Synchronizing Cookbooks:
         - hostsfile (2.4.5)
         - sudo (2.9.0)
         - users (2.0.3)
         - cm_setup (0.1.0)
         - sysctl (0.7.5)
         - mysql (7.1.1)
         - yum (3.11.0)
         - smf (2.2.8)
         - ohai (3.0.1)
         - database (5.1.2)
         - build-essential (6.0.0)
         - rbac (1.0.3)
         - openssl (4.4.0)
         - yum-mysql-community (0.2.0)
         - apt (4.0.0)
         - chef-sugar (3.3.0)
         - mingw (1.2.0)
         - seven_zip (2.0.1)
         - postgresql (4.0.6)
         - compat_resource (12.10.6)
         - windows (1.43.0)
         - chef_handler (1.4.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       [2016-06-18T17:40:28+00:00] WARN: Chef::Provider::AptRepository already exists!  Cannot create deprecation class for LWRP provider apt_repository from cookbook apt
       [2016-06-18T17:40:28+00:00] WARN: AptRepository already exists!  Deprecation class overwrites Custom resource apt_repository from cookbook apt
       [2016-06-18T17:40:28+00:00] WARN: Cloning resource attributes for hostsfile_entry[3.3.3.3] from prior resource (CHEF-3694)
       [2016-06-18T17:40:28+00:00] WARN: Previous hostsfile_entry[3.3.3.3]: /tmp/kitchen/cache/cookbooks/cm_setup/recipes/hostfile_setup.rb:27:in `from_file'
       [2016-06-18T17:40:28+00:00] WARN: Current  hostsfile_entry[3.3.3.3]: /tmp/kitchen/cache/cookbooks/cm_setup/recipes/hostfile_setup.rb:34:in `from_file'
       Converging 37 resources

       ##
       ###################### VERBOSE ########################
       ##

       Recipe: cm_setup::cloudera_install_setup
         * yum_package[oracle-j2sdk1.7] action install (up to date)
         * yum_package[cloudera-manager-daemons] action install (up to date)
         * yum_package[cloudera-manager-server] action install (up to date)
       Recipe: cm_setup::mysql_setup
         * mysql_service[default] action restart
           * service[default :restart mysql-default] action restart
             - restart service service[default :restart mysql-default]

       Running handlers:
       Running handlers complete
       Chef Client finished, 20/106 resources updated in 07 minutes 05 seconds
       Finished converging  (7m25.80s).
Above we have no issues and the cookbook converged successfully.
┌─[ahmed][zubair-HP-ProBook][±][master ✓][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen list
Instance                 Driver   Provisioner  Verifier  Transport  Last Action
default-grtjn-centos-65  Vagrant  ChefZero     Busser    Ssh        Converged

Server Spec Verification using Kitchen.

Command
kitchen verify
Output.
┌─[ahmed][zubair-HP-ProBook][±][master U:1 ✗][~/work/chef-repo/cookbooks/cm_setup]
└─▪ kitchen verify
-----> Starting Kitchen (v1.8.0)
-----> Verifying ...
       Preparing files for transfer
-----> Busser installation detected (busser)
       Installing Busser plugins: busser-serverspec
       Plugin serverspec already installed
       Removing /tmp/verifier/suites/serverspec
       Transferring files to 
-----> Running serverspec test suite
       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/tmp/verifier/gems/gems/rspec-core-3.4.4/lib /opt/chef/embedded/bin/rspec --pattern    /tmp/verifier/suites/serverspec/\*\*/\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec

       cm_setup::default
         File "/etc/yum.repos.d/cloudera-manager.repo"
           should exist
         File "/etc/mysql-default/conf.d/default.cnf"
           should exist
           should be file
           should contain "max_connections = 550"
         File "/etc/sysctl.d/99-chef-attributes.conf"
           should exist
           should be file
           should contain "vm.dirty_ratio"
           should contain "vm.swappiness"
           should contain "vm.nr_hugepages"
         Package "ntp"
           should be installed
         Package "oracle-j2sdk1.7"
           should be installed
         Package "cloudera-manager-daemons"
           should be installed
         Package "cloudera-manager-server"
           should be installed
         File "/etc/hosts"
           should exist
           should be file
           should contain "namenode.ahmed.com"
           should contain "standbynamenode.ahmed.com"
           should contain "resourcemanager.ahmed.com"
         File "/etc/cloudera-scm-server/db.properties"
           should exist
           should be file
           should contain "user=cmadmin"
           should contain "name=cmdb"
         User "cmadmin"
           should exist
           should have home directory "/home/cmadmin"
         User "sysadminuser"
           should exist
           should have home directory "/home/sysadminuser"
         MySQL config parameters
           Mysql config "innodb_flush_log_at_trx_commit"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:55
           Mysql config "socket"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:59
           Mysql config "innodb_flush_method"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:66
           Mysql config "innodb_log_file_size"
             value
               example at /tmp/verifier/suites/serverspec/default_spec.rb:70
         Yumrepo "epel"
           should exist
           should be enabled
         Yumrepo "cloudera-manager"
           should exist
           should be enabled

       Finished in 1.32 seconds (files took 0.52476 seconds to load)
       34 examples, 0 failures

       Finished verifying  (0m6.22s).
-----> Kitchen is finished. (0m6.66s)

Cloudera Manager UI.

Logon to the node and open up a browser and hit.
http://127.0.0.1:7180/
You will see the cloudera manager UI. NOTE: This will take a while for the first time, as clouder will initialize the database for first time use.

from Blogger http://ift.tt/292cVNk
via IFTTT

Categories: Others Tags: ,

Chef Workstation Setup on Windows Machine.

June 9, 2016 Leave a comment
The Chef Development Kit (ChefDK) brings the best-of-breed development tools built by the awesome Chef community to your workstation with just a few clicks. Download your package and start coding Chef in seconds.

Install Instructions

Once the package has been downloaded, double-click the .msi file to run the MSI installer. Use all of the default installation options.
The chef command and the other commands included with the Chef Development Kit should now be available for use.

Download and configure.

Download File
Download File

Install Vagrant.

Install VirtualBox.

Setting up the Environment.

Creating a Blank cookbook using chefdk

Command.
chef generate cookbook my_first_cookbook
Here is the output.
PS C:\Users\zubair.ahmed\centos_vagrant> chef generate cookbook my_first_cookbook
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: code_generator::cookbook
  * directory[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook] action create
    - create new directory C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook
  * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/metadata.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/metadata.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/metadata.rb from none to a4d2ff
    (diff output suppressed by config)

    * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/README.md] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/README.md
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/README.md from none to b84cc5

    (diff output suppressed by config)
  * cookbook_file[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/chefignore] action create
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/chefignore
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/chefignore from none to 15fac5
    (diff output suppressed by config)
  * cookbook_file[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/Berksfile] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/Berksfile
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/Berksfile from none to 9f08dc

    (diff output suppressed by config)

  * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/.kitchen.yml] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/.kitchen.yml
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/.kitchen.yml from none to d53214 
    (diff output suppressed by config)

  * directory[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/default/serverspec] action    create
    - create new directory C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/default/serverspec
  * directory[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/helpers/serverspec] action create
    - create new directory C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/helpers/serverspec
  * cookbook_file[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/helpers/serverspec/spec_helper.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/helpers/serverspec/spec_helper.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/helpers/serverspec/spec_helper.rb from none to d85df4
    (diff output suppressed by config)

  * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/default/serverspec/default_spec.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/default/serverspec/default_spec.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/test/integration/default/serverspec/default_spec.rb from none to 3c5e72
    (diff output suppressed by config)

  * directory[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/unit/recipes] action create
    - create new directory C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/unit/recipes
  * cookbook_file[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/spec_helper.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/spec_helper.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/spec_helper.rb from none to 587075
    (diff output suppressed by config)

  * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/unit/recipes/default_spec.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/unit/recipes/default_spec.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/spec/unit/recipes/default_spec.rb from none to ed26b2
    (diff output suppressed by config)

  * directory[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/recipes] action create
    - create new directory C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/recipes
  * template[C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/recipes/default.rb] action create_if_missing
    - create new file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/recipes/default.rb
    - update content in file C:/Users/zubair.ahmed/centos_vagrant/my_first_cookbook/recipes/default.rb from none to d2c7b9
    (diff output suppressed by config)

Update .kitchen.yml file as below.

We are just updating the platforms from centos 7 to 6.5.
You can keep it as it is if you are looking for centos 7.
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

# Uncomment the following verifier to leverage Inspec instead of Busser (the
# default verifier)
# verifier:
#   name: inspec

platforms:
  - name: centos-6.5

suites:
  - name: default
    run_list:
      - recipe[my_first_cookbook::default]
    attributes:

Creating the VM.

Checking the list of VM which we have in the kitchen.yml file.
kitchen list
Output.
PS C:\Users\zubair.ahmed\centos_vagrant\my_first_cookbook> kitchen.bat list
Instance           Driver   Provisioner  Verifier  Transport  Last Action
default-centos-65  Vagrant  ChefZero     Busser    Ssh        
Creating a VM.
kitchen create default-centos-65
Output.
PS C:\Users\zubair.ahmed\centos_vagrant\my_first_cookbook> kitchen.bat create default-centos-65
-----> Starting Kitchen (v1.7.3)
-----> Creating ...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'bento/centos-6.5'...
==> default: Matching MAC address for NAT networking...
       ==> default: Setting the name of the VM: kitchen-my_first_cookbook-default-centos-65_default_1465204398970_64998
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 (guest) => 2222 (host) (adapter 1)
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: 127.0.0.1:2222
           default: SSH username: vagrant
           default: SSH auth method: private key
           default: Warning: Remote connection disconnect. Retrying...
           default:
           default: Vagrant insecure key detected. Vagrant will automatically replace
           default: this with a newly generated keypair for better security.
           default:
           default: Inserting generated public key within guest...
           default: Removing insecure key from the guest if it's present...
           default: Key inserted! Disconnecting and reconnecting using new SSH key...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
           default: The guest additions on this VM do not match the installed version of
           default: VirtualBox! In most cases this is fine, but in rare cases it can
           default: prevent things such as shared folders from working properly. If you see
           default: shared folder errors, please make sure the guest additions within the
           default: virtual machine match the version of VirtualBox you have installed on
           default: your host and reload your VM.
           default:
           default: Guest Additions Version: 4.3.8
           default: VirtualBox Version: 5.0
       ==> default: Setting hostname...
       ==> default: Machine not provisioned because `--no-provision` is specified.
       [SSH] Established
       Vagrant instance  created.
       Finished creating  (1m58.85s).
-----> Kitchen is finished. (2m7.25s)
Download File

Make some basic changes to the my_first_cookbook.

Update default.rb in recipe as below. File path : ${HOME}/my_first_cookbook/recipes/default.rb
package 'java-1.7.0-openjdk' do
  action :install
end

#
# Creating `group`
group 'tomcat' do
  action :create # This is a default behaviour
end

# Creating user `tomcat`
user 'tomcat' do
  group 'tomcat'
  home '/opt/tomcat'
  shell '/bin/nologin'
  uid '1234'
  action :create
end

# Create directory

directory ('/opt/tomcat') do
  owner 'tomcat'
  group 'tomcat'
end

#
# Downloading Apache tomcat.
#
remote_file ('/tmp/apache-tomcat-8.0.33.tar.gz') do
  source 'http://ift.tt/1tk7123'
end
#
# # TODO: Change this.
#
execute 'tar xvzf /tmp/apache-tomcat-8.0.33.tar.gz -C /opt/tomcat/ --strip-components=1' do
  not_if do ::File.exists?('/opt/tomcat/conf/server.xml') end
end

# TODO: Need to update this ...
execute 'chown -R tomcat:tomcat /opt/tomcat'

directory '/opt/tomcat/conf' do
  mode 0070
end

execute '/opt/tomcat/bin/startup.sh'
Update the test spec. File Path : ${HOME}/my_first_cookbook/test/integration/default/serverspec/default_spec.rb
require 'spec_helper'

describe 'tomcat::default' do

  describe command('curl --noproxy 127.0.0.1 http://127.0.0.1:8080') do
    its(:stdout) { should match /tomcat/ }
  end

  describe file ('/opt/tomcat') do
    it { should exist }
    it { should be_directory }
  end

  describe group ('tomcat') do
    it { should exist }
  end

  describe user ('tomcat') do
    it { should exist }
    it { should belong_to_group 'tomcat' }
    it { should have_home_directory '/opt/tomcat' }
    it { should have_login_shell '/bin/nologin' }
  end

  describe file ('/opt/tomcat/conf') do
    it { should exist }
    it { should be_mode 70 }
  end

end

Converge the cookbook with VM.

This will converge the recipe with the VM.
kitchen.bat converge default-centos-65
Output.
PS C:\Users\zubair.ahmed\centos_vagrant\tomcat> kitchen.bat converge default-centos-65
-----> Starting Kitchen (v1.7.3)
-----> Converging ...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.3.2...
       Removing non-cookbook files before transfer
       Preparing solo.rb
-----> Chef Omnibus installation detected (install only if missing)
       Transferring files to 
       Starting Chef Client, version 12.11.18
       resolving cookbooks for run list: ["tomcat::default"]
       Synchronizing Cookbooks:
         - tomcat (0.1.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Converging 9 resources
       Recipe: tomcat::default
         * yum_package[java-1.7.0-openjdk] action install
           - install version 1.7.0.101-2.6.6.4.el6_8 of package java-1.7.0-openjdk
         * group[tomcat] action create
           - create group tomcat
         * user[tomcat] action create
           - create user tomcat
         * directory[/opt/tomcat] action create (up to date)
         * remote_file[/tmp/apache-tomcat-8.0.33.tar.gz] action create
           - create new file /tmp/apache-tomcat-8.0.33.tar.gz
           - update content in file /tmp/apache-tomcat-8.0.33.tar.gz from none to c77873
           (new content is binary, diff output suppressed)
           - restore selinux security context
         * execute[tar xvzf /tmp/apache-tomcat-8.0.33.tar.gz -C /opt/tomcat/ --strip-components=1] action run
           - execute tar xvzf /tmp/apache-tomcat-8.0.33.tar.gz -C /opt/tomcat/ --strip-components=1
         * execute[chown -R tomcat:tomcat /opt/tomcat] action run
           - execute chown -R tomcat:tomcat /opt/tomcat
         * directory[/opt/tomcat/conf] action create
           - change mode from '0755' to '070'
           - restore selinux security context
         * execute[/opt/tomcat/bin/startup.sh] action run
           - execute /opt/tomcat/bin/startup.sh

       Running handlers:
       Running handlers complete
       Chef Client finished, 8/9 resources updated in 53 seconds
       Finished converging  (0m56.68s).
-----> Kitchen is finished. (1m2.05s)
Download File

Verify the converge.

Verify the converge and check if all the file and services are installed.
kitchen.bat verify default-centos-65
Output.
PS C:\Users\zubair.ahmed\centos_vagrant\tomcat> kitchen.bat verify default-centos-65
-----> Starting Kitchen (v1.7.3)
-----> Setting up ...
       Finished setting up  (0m0.00s).
-----> Verifying ...
       Preparing files for transfer
-----> Installing Busser (busser)
Fetching: thor-0.19.0.gem (100%)
       Successfully installed thor-0.19.0
Fetching: busser-0.7.1.gem (100%)
       Successfully installed busser-0.7.1
       2 gems installed
       Installing Busser plugins: busser-serverspec
       Plugin serverspec installed (version 0.5.9)
-----> Running postinstall for serverspec plugin
       Suite path directory /tmp/verifier/suites does not exist, skipping.
       Transferring files to 
-----> Running serverspec test suite
-----> Installing Serverspec..
Fetching: sfl-2.2.gem (100%)
Fetching: net-telnet-0.1.1.gem (100%)
Fetching: net-ssh-3.1.1.gem (100%)
Fetching: net-scp-1.2.1.gem (100%)
Fetching: specinfra-2.59.0.gem (100%)
Fetching: rspec-support-3.4.1.gem (100%)
Fetching: diff-lcs-1.2.5.gem (100%)
Fetching: rspec-expectations-3.4.0.gem (100%)
Fetching: rspec-core-3.4.4.gem (100%)
Fetching: rspec-its-1.2.0.gem (100%)
Fetching: rspec-mocks-3.4.1.gem (100%)
Fetching: rspec-3.4.0.gem (100%)
Fetching: multi_json-1.12.1.gem (100%)
Fetching: serverspec-2.36.0.gem (100%)
-----> serverspec installed (version 2.36.0)
       /opt/chef/embedded/bin/ruby -I/tmp/verifier/suites/serverspec -I/tmp/verifier/gems/gems/rspec-support-3.4.1/lib:/
tmp/verifier/gems/gems/rspec-core-3.4.4/lib /opt/chef/embedded/bin/rspec --pattern /tmp/verifier/suites/serverspec/\*\*/
\*_spec.rb --color --format documentation --default-path /tmp/verifier/suites/serverspec

       tomcat::default
         Command "curl --noproxy 127.0.0.1 http://127.0.0.1:8080"
           stdout
             should match /tomcat/
         File "/opt/tomcat"
           should exist
           should be directory
         Group "tomcat"
           should exist
         User "tomcat"
           should exist
           should belong to group "tomcat"
           should have home directory "/opt/tomcat"
           should have login shell "/bin/nologin"
         File "/opt/tomcat/conf"
           should exist
           should be mode 70

       Finished in 1.92 seconds (files took 0.38732 seconds to load)
       10 examples, 0 failures

       Finished verifying  (0m26.23s).
-----> Kitchen is finished. (0m31.66s)
Important part in the above output is as below. All the services are fine. We have 10 test examples and 0 failures.
       tomcat::default
         Command "curl --noproxy 127.0.0.1 http://127.0.0.1:8080"
           stdout
             should match /tomcat/
         File "/opt/tomcat"
           should exist
           should be directory
         Group "tomcat"
           should exist
         User "tomcat"
           should exist
           should belong to group "tomcat"
           should have home directory "/opt/tomcat"
           should have login shell "/bin/nologin"
         File "/opt/tomcat/conf"
           should exist
           should be mode 70

       Finished in 1.92 seconds (files took 0.38732 seconds to load)
       10 examples, 0 failures

       Finished verifying  (0m26.23s).
-----> Kitchen is finished. (0m31.66s)
Download File
Your first cookbook is ready.

from Blogger http://ift.tt/1tcYfCh
via IFTTT

Categories: Others Tags: ,

Standalone Chef Server / Workstation / Client Setup on CentOS 6

June 8, 2016 Leave a comment
The standalone installation of Chef server creates a working installation on a single server. This installation is also useful when you are installing Chef server in a virtual machine, for proof-of-concept deployments, or as a part of a development or testing loop.

Before we start.

In our environment we will be creating the below nodes.
192.168.30.132  chefserver.ahmed.com    
192.168.30.142  chefworkstation.ahmed.com    
192.168.30.141  chefnode.ahmed.com 
Setup the /etc/hosts file on all the nodes.
127.0.0.1   localhost
192.168.30.132  chefserver.ahmed.com    chefserver
192.168.30.142  chefworkstation.ahmed.com    chefworkstation
192.168.30.141  chefnode.ahmed.com    chefnode
Install crontabs (if not already installed)
yum install crontabs
Flush all the iptables rules and save them on all Nodes.
iptables -F
service iptables save
Command output.
[root@chefnode Downloads]# iptables -F
[root@chefnode Downloads]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
[root@chefnode Downloads]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@chefnode Downloads]#
Set selinux to Permissive mode on all nodes.
setenforce Permissive
getenforce
Command output.
[root@chefnode Downloads]# setenforce Permissive
[root@chefnode Downloads]# getenforce
Permissive
[root@chefnode Downloads]#
Download the package from http://ift.tt/1AVBlMj.
[root@chefnode Downloads]# ls -l
total 656944
-rwxr--r--. 1 root root  52338858 Jun  8 01:45 chef-12.10.24-1.el6.x86_64.rpm
-rwxr--r--. 1 root root 142488123 Jun  8 01:45 chefdk-0.14.25-1.el6.x86_64.rpm
-rwxr--r--. 1 root root 477870942 Jun  8 01:45 chef-server-core-12.6.0-1.el6.x86_64.rpm
[root@chefnode Downloads]#

Creating chef-server.

Install the package as below.
rpm -ivh chef-server-core-12.6.0-1.el6.x86_64.rpm
First thing we need to do is to reconfigure the server.
chef-server-ctl reconfigure
Create a user for the server.
chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename PATH_TO_FILE_NAME
Command.
chef-server-ctl user-create ahmed Zubair AHMED zubayr.a@gmail.com 'ahmed@123' --filename /etc/chef/ahmed.pem 
Create an organization.
chef-server-ctl org-create short_name 'full_organization_name' --association_user user_name --filename ORGANIZATION-validator.pem 
Command.
chef-server-ctl org-create ahmedinc 'Ahmed, Inc.' --association_user ahmed --filename /etc/chef/ahmed-validator.pem
Setting up and installing chef-manage.
chef-server-ctl install chef-manage
chef-server-ctl reconfigure
chef-manage-ctl reconfigure
Check server status using below command.
chef-server-ctl status
we see all the services running fine.
[root@chefserver Downloads]# chef-server-ctl status
run: bookshelf: (pid 6081) 4002s; run: log: (pid 6121) 4001s
run: nginx: (pid 8725) 2724s; run: log: (pid 6245) 3995s
run: oc_bifrost: (pid 5924) 4008s; run: log: (pid 5959) 4008s
run: oc_id: (pid 5976) 4006s; run: log: (pid 5984) 4006s
run: opscode-erchef: (pid 6192) 3999s; run: log: (pid 6160) 4001s
run: opscode-expander: (pid 6046) 4002s; run: log: (pid 6064) 4002s
run: opscode-solr4: (pid 6007) 4004s; run: log: (pid 6036) 4003s
run: postgresql: (pid 5901) 4009s; run: log: (pid 5914) 4009s
run: rabbitmq: (pid 5814) 4010s; run: log: (pid 5807) 4010s
run: redis_lb: (pid 8133) 2805s; run: log: (pid 6241) 3996s
[root@chefserver Downloads]#

Installing chefdk on workstation node.

More information here

Install chefdk-0.14.25-1.el6.x86_64.rpm on the node.

[root@chefworkstation Downloads]# rpm -ivh chefdk-0.14.25-1.el6.x86_64.rpm
warning: chefdk-0.14.25-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID
Preparing...                ########################################### [100%]
   1:chefdk                 ########################################### [100% ]
Thank you for installing Chef Development Kit!
Check for selinux.
[root@chefworkstation Downloads]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Set selinux to Permissive .
[root@chefworkstation Downloads]# setenforce Permissive
[root@chefworkstation Downloads]# getenforce
Permissive
Check installation using chef verify.
[root@chefworkstation chefdk]# chef verify
Running verification for component 'berkshelf'
Running verification for component 'test-kitchen'
Running verification for component 'tk-policyfile-provisioner'
Running verification for component 'chef-client'
Running verification for component 'chef-dk'
Running verification for component 'chef-provisioning'
Running verification for component 'chefspec'
Running verification for component 'generated-cookbooks-pass-chefspec'
Running verification for component 'rubocop'
Running verification for component 'fauxhai'
Running verification for component 'knife-spork'
Running verification for component 'kitchen-vagrant'
Running verification for component 'package installation'
Running verification for component 'openssl'
Running verification for component 'inspec'
Running verification for component 'delivery-cli'
Running verification for component 'git'
Running verification for component 'opscode-pushy-client'
Running verification for component 'chef-sugar'
Running verification for component 'knife-supermarket'
..............................................
---------------------------------------------
Verification of component 'fauxhai' succeeded.
Verification of component 'kitchen-vagrant' succeeded.
Verification of component 'openssl' succeeded.
Verification of component 'delivery-cli' succeeded.
Verification of component 'test-kitchen' succeeded.
Verification of component 'rubocop' succeeded.
Verification of component 'inspec' succeeded.
Verification of component 'opscode-pushy-client' succeeded.
Verification of component 'knife-supermarket' succeeded.
Verification of component 'berkshelf' succeeded.
Verification of component 'knife-spork' succeeded.
Verification of component 'git' succeeded.
Verification of component 'tk-policyfile-provisioner' succeeded.
Verification of component 'chefspec' succeeded.
Verification of component 'chef-sugar' succeeded.
Verification of component 'chef-client' succeeded.
Verification of component 'chef-dk' succeeded.
Verification of component 'package installation' succeeded.
Verification of component 'chef-provisioning' succeeded.
Verification of component 'generated-cookbooks-pass-chefspec' succeeded.
Setting up ruby.
[root@chefworkstation ahmed]# eval "$(chef shell-init bash)"
[root@chefworkstation ahmed]# which ruby
/opt/chefdk/embedded/bin/ruby
[root@chefworkstation ahmed]#
To make these changes persist, execute below command to update .bash_profile.
echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile

Getting the starter kit.

Goto the chef-server > Administration Tab > Select organization > Click Starter Kit on the left pane.
Admin Tab
files
Select starter-kit
files
Reset keys and download
files
Copy the starter kit (chef-repo) from the server and unzip and save it in the home directory on WORKSTATION NODE.
[root@chefworkstation ahmed]# pwd
/home/ahmed
[root@chefworkstation ahmed]# ls -l
total 40
drwxr-xr-x. 5 root  root  4096 Jun  8 03:27 chef-repo
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Desktop
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Documents
drwxr-xr-x. 2 ahmed ahmed 4096 Jun  8 02:55 Downloads
drwxrwxr-x. 6 ahmed ahmed 4096 Feb  3 01:39 github
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Music
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Pictures
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Public
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Templates
drwxr-xr-x. 2 ahmed ahmed 4096 May  7  2015 Videos
[root@chefworkstation ahmed]# tree chef-repo/
chef-repo/
├── cookbooks
│   ├── chefignore
│   ├── cm_setup
│   │   ├── Berksfile
│   │   ├── Berksfile.lock
│   │   ├── chefignore
│   │   ├── metadata.rb
│   │   ├── README.md
│   │   ├── recipes
│   │   │   └── default.rb
│   │   ├── spec
│   │   │   ├── spec_helper.rb
│   │   │   └── unit
│   │   │       └── recipes
│   │   │           └── default_spec.rb
│   │   ├── templates
│   │   │   └── default
│   │   │       ├── cloudera_manager.repo.erb
│   │   │       ├── my.cnf.erb
│   │   │       └── sysctl.conf.erb
│   │   └── test
│   │       └── integration
│   │           ├── default
│   │           │   └── serverspec
│   │           │       └── default_spec.rb
│   │           └── helpers
│   │               └── serverspec
│   │                   └── spec_helper.rb
│   └── starter
│       ├── attributes
│       │   └── default.rb
│       ├── files
│       │   └── default
│       │       └── sample.txt
│       ├── metadata.rb
│       ├── recipes
│       │   └── default.rb
│       └── templates
│           └── default
│               └── sample.erb
├── README.md
└── roles
    └── starter.rb

22 directories, 21 files
[root@chefworkstation ahmed]#

Fetching the SSL key from the chef-server.

[root@chefworkstation chef-repo]# knife ssl fetch
WARNING: Certificates from chefserver.ahmed.com will be fetched and placed in your trusted_cert
directory (/home/ahmed/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for chefserver.ahmed.com in /home/ahmed/chef-repo/.chef/trusted_certs/chefserver_ahmed_com.crt
Check client currently available. Currently we see only the work station.
[root@chefworkstation chef-repo]# knife client list
ahmedinc-validator

Bootstrapping a Node.

More information here.

Setting up a client using the knife bootstrap command.

[root@chefworkstation chef-repo]#  knife bootstrap chefnode.ahmed.com -x ahmed -P ahmed --sudo
Doing old-style registration with the validation key at /home/ahmed/chef-repo/.chef/ahmedinc-validator.pem...
Delete your validation key in order to use your user credentials instead

Connecting to chefnode.ahmed.com
chefnode.ahmed.com This is BASH 4.1     - DISPLAY on
chefnode.ahmed.com
chefnode.ahmed.com Wed Jun  8 03:05:01 PDT 2016
chefnode.ahmed.com -----> Existing Chef installation detected
chefnode.ahmed.com Starting the first Chef Client run...
chefnode.ahmed.com Starting Chef Client, version 12.10.24
chefnode.ahmed.com Creating a new client identity for chefnode.ahmed.com using the validator key.
chefnode.ahmed.com resolving cookbooks for run list: []
chefnode.ahmed.com Synchronizing Cookbooks:
chefnode.ahmed.com Installing Cookbook Gems:
chefnode.ahmed.com Compiling Cookbooks...
chefnode.ahmed.com [2016-06-08T03:05:08-07:00] WARN: Node chefnode.ahmed.com has an empty run list.
chefnode.ahmed.com Converging 0 resources
chefnode.ahmed.com
chefnode.ahmed.com Running handlers:
chefnode.ahmed.com Running handlers complete
chefnode.ahmed.com Chef Client finished, 0/0 resources updated in 05 seconds
chefnode.ahmed.com Hasta la vista, baby
We can check the node using the below command.
[root@chefworkstation chef-repo]# knife client list                          
ahmedinc-validator
chefnode.ahmed.com
Here is how we can see the new node on the chef-server.
cookbook on chef-server
Lets update the starter cookbook and upload to server and add the cookbook to the node.
Update /home/ahmed/chef-repo/cookbooks/starter/recipes/default.rb with below contents.
log "Welcome to Chef, #{node["starter_name"]}!" do
  level :info
end

file '/etc/my_first_file' do
        content 'This is my first file creation using chef server'
end

file '/etc/my_second_file' do
        content 'My Second file'
end
Here is the contents from the command output.
[root@chefworkstation chef-repo]# pwd                                         
/home/ahmed/chef-repo
[root@chefworkstation chef-repo]# cat cookbooks/starter/recipes/default.rb
# This is a Chef recipe file. It can be used to specify resources which will
# apply configuration to a server.

log "Welcome to Chef, #{node["starter_name"]}!" do
  level :info
end

file '/etc/my_first_file' do
        content 'This is my first file creation using chef server'
end

file '/etc/my_second_file' do
        content 'My Second file'
end
Now lets upload the cookbook to the server.
[root@chefworkstation chef-repo]# knife upload cookbooks/starter
Created cookbooks/starter
[root@chefworkstation chef-repo]#
Starter cookbook on the chef-server
cookbook on chef-server
Contents of starter Cookbook
cookbook on chef-server
Now lets configure the node to get the cookbook assigned.
Edit Node Configuration
files
Select the cookbook and add to Current Run List
files
files

Sync client with server using chef-client command.

Once we have assigned we can logon to the ched-node and execute chef-client. This will communicate to the chef-server and get the cookbook and syncronize the server.
[root@chefnode Downloads]# chef-client
Starting Chef Client, version 12.10.24
resolving cookbooks for run list: ["starter"]
Synchronizing Cookbooks:
  - starter (1.0.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 2 resources
Recipe: starter::default
  * log[Welcome to Chef, Sam Doe!] action write

  * file[/etc/my_first_file] action create
    - create new file /etc/my_first_file
    - update content in file /etc/my_first_file from none to ad169c
    --- /etc/my_first_file      2016-06-08 03:37:02.461008058 -0700
    +++ /etc/.chef-my_first_file20160608-3200-18m1pty   2016-06-08 03:37:02.460008058 -0700
    @@ -1 +1,2 @@
    +This is my first file creation using chef server
    - restore selinux security context

Running handlers:
Running handlers complete
Chef Client finished, 2/2 resources updated in 05 seconds

Getting repo from git and sync with chef-server.

On the workstation node.
[root@chefworkstation cookbooks]# pwd
/home/ahmed/chef-repo/cookbooks
Get repo from git and upload to chef server.
[root@chefworkstation cookbooks]# git clone http://ift.tt/25M3fvU
Initialized empty Git repository in /home/ahmed/chef-repo/cookbooks/cm_setup/.git/
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (35/35), done.
remote: Total 62 (delta 5), reused 61 (delta 4), pack-reused 0
Unpacking objects: 100% (62/62), done.
[root@chefworkstation cookbooks]# cd ..
[root@chefworkstation chef-repo]# ls
cookbooks  README.md  roles
Upload to server.
[root@chefworkstation chef-repo]# knife upload cookbooks/cm_setup
Created cookbooks/cm_setup
[root@chefworkstation chef-repo]#
As seen on the server.
files

from Blogger http://ift.tt/25M3V4k
via IFTTT

Categories: Others Tags: ,