Archive

Author Archive

Parcel Not Distributing Cloudera CDH.

March 27, 2017 Leave a comment
We were deploying one of the cluster on our lab environment which is used by everyone.
So the lab has it own share of stale information on it.
During installation we notice that the distribution is not working. There could be couple of reasons.
This was the second time we are having this issue.
  1. Check /etc/hosts file if we have all the server names added correctly
  2. Second reason would be due to the fact that, one of the installation was terminated midway, leaving a stale config which set the status to ACTIVATING for CDH parcel. So when we try to install, parcel was not distributing.
  3. Again there could be similar issue if we do not have enough space on the node for /opt/cloudera.

Solution:

  • Deactivating parcel and retry.
    curl -u username:password -X POST http://adminnode:7180/api/v14/clusters//parcels/products/CDH/versions/5.10.0-1.cdh5.10.0.p0.41/commands/deactivate
    
  • Check for space and increase space for /opt/cloudera/
  • Most of the time should see the issue on the logs, server-logs: /var/log/cloudera-scm-server/cloudera-scm-server.log or agent-logs: /var/log/cloudera-scm-agent/cloudera-scm-agent.log

‚Äč

from Blogger http://ift.tt/2o70m9m
via IFTTT

Categories: Others Tags: ,

Creating /etc/hosts file in Chef.

March 26, 2017 Leave a comment
We had a cluster environment which we needed to update the /etc/hosts file. Which would help communicate between the server over a private network. Our servers have multiple interfaces and we need them to communicate between each other using the private network.
Goal for us is to create a /etc/hosts with all the nodes within the cluster with their private IP addresses.

Chef Setup (Assumptions).

  • We have multiple cluster.
  • Each cluster has an environment set.
  • We have a multiple interfaces on each node (But this solutions should work for single interface as well).

    Steps we do to get information from each node.

  1. We take 3 attributes to look for in the cluster.
    1. Each server with the cluster have specific fqdn.
    2. Private IP (string search).
    3. Interface to look for on the node.
  2. all_nodes_in_cluster will get all the nodes with that fqdn (This can be change based on requirement to node, tags, roles, hostname).
  3. For every node we look for the specific interface and get the private IP.

    Before we start.

    Before we start we need to check for our search criteria using knife search more details here
    Below if an example to look for server fqdn containing string env-lab-1.
    knife search node "fqdn:*env-lab-1*"
    

    About the Recipe.

The interface information on hash is located as below
 node_in_cluster['network']['interfaces']['bond.007']['addresses']
This has a Dictionary with multiple values, we are specifically looking for IP.
 if private_interface[0].include? node['private_ip_search_filter']
Above we are looking for the interface which matches out search filter. Required information is in private_interface[0]
Here is how we would write the information in our /etc/hosts file, IP,FQDN,HOSTNAME.
 puts "#{private_interface[0]} #{node_in_cluster['fqdn']} #{node_in_cluster['hostname']}"
Here is complete ruby code which does the same thing as in the erb template file.
 all_nodes_in_cluster.each do |node_in_cluster|
   node_in_cluster['network']['interfaces'][int_to_look]['addresses'].each do |private_interface|
     if private_interface[0].include? node['private_ip_search_filter']
       puts "#{private_interface[0]} #{node_in_cluster['fqdn']} #{node_in_cluster['hostname']}"
     end
   end
 end

Attribute File.

 # Example :  We take the search Criteria to generate /etc/hosts
 default['env_search_filter'] = "fqdn:*lab-env-1*"
 default['private_ip_search_filter'] = "192.168.149"
 default['interface_to_look'] = 'bond.007'

Recipe

 # Search Criteria
 all_nodes_in_cluster = search(:node, node['env_search_filter'])
 int_to_look = node['interface_to_look']

 template '/etc/hosts' do
   source 'etc_hosts_file.erb'
   mode '0755'
   owner 'root'
   group 'root'
   variables({
     all_nodes_in_cluster: all_nodes_in_cluster,
     int_to_look: int_to_look,
     private_ip_search_filter: node['private_ip_search_filter']
   })
 end

Template File etc_hosts_file.erb.

 127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
 <% @all_nodes_in_cluster.each do |node_in_cluster| -%>
   <% node_in_cluster['network']['interfaces'][@int_to_look]['addresses'].each do |private_interface| -%>
     <% if private_interface[0].include? @private_ip_search_filter -%>
 <%= private_interface[0] %>     <%= node_in_cluster['fqdn'] %>      <%= node_in_cluster['hostname'] %>  # Serial Number: <%= node_in_cluster['dmi']['system']['serial_number'] %> ( <%= node_in_cluster['dmi']['system']['manufacturer'] %> ) <%= node_in_cluster['dmi']['system']['product_name'] %>
     <% end -%>
   <% end -%>
 <% end -%>
 ::1     localhost localhost.localdomain localhost6 localhost6.localdomain6

Disclaimer.

  1. This does not look like an optimized solution, but something which worked for me.
  2. search method which will be run every 30mins, will query to get all information for all the nodes, which I think would be time/bandwidth consuming operation if we have a very large cluster. (A single node information was about 10000 lines of ruby Hash for our nodes).
  3. If any one has a better way to do it, please post it in the comments below. Thanks ūüôā

‚Äč

from Blogger http://ift.tt/2nnNYn6
via IFTTT

Categories: Others Tags: ,

Enable Kerberos Using Cloudera API.

March 25, 2017 Leave a comment
Python API for cloudera is really nice, apart from getting the cluster setup, we can also do configuration and automation. We use a lot of automation using Chef/Ansible, but cloudera API give more control over the cluster.
One of the awesome features of cloudera API is to setup kerberos for the cluster, which otherwise done manually (command line) is very tricky task (loads of this to go wrong).
For our kerberos setup we will need to complete below steps.
  1. Cloudera Manager Configuration.
  2. Stop Cluster.
  3. Stop Cloudera Manager service.
  4. Service configuration for all the services which needs updates like HDFS, HBASE, KAFKA, ZOOKEEPER, HUE.
  5. Creating KT_RENEWER for HUE service.
  6. Start Cloudera Manager Services.
  7. Start Cluster.

Steps To kerberize Cloudera Cluster

Step 1: Configure cloudera with LDAP information.
 #
 # Deploy LDAP configuration for CM
 #
 self.deploy_cm_ldap_configuration()
Step 2: Import Admin credentials, using the username / password, which has permission to create/delete AD service accounts.
 #
 # Creating Admin Credentials
 #
 self.cm_import_admin_credentials()
Step 3: Stopping cloudera cluster services.
 #
 # Stopping Cluster
 #
 self.gets_cm_cluster.stop().wait()
 logging.debug("Waiting for CLUSTER to stop completely !!!")
 time.sleep(5)
Step 4: Stopping cloudera management services.
 #
 # Stopping CM services
 #
 self.get_mgmt_services.stop().wait()
 logging.debug("Waiting for CM to stop completely !!!")
 time.sleep(5)
Step 5: Update Zookeeper configuration.
 #
 # Deploy Kerberos Config Zookeeper
 #
 logging.debug("Deploy Service Configuration!!!")
 zk_service = Zookeeper(self.cluster, self.config, self.cloudera_manager)
 zk_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 6: Update HDFS configuration.
 #
 # Deploy Kerberos Config Hdfs
 #
 logging.debug("Deploy Service Configuration!!!")
 hdfs_service = Hdfs(self.cluster, self.config, self.cloudera_manager)
 hdfs_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 6: Update HBASE configuration.
 #
 # Deploy Kerberos Config Hbase
 #
 logging.debug("Deploy Service Configuration!!!")
 hbase_service = Hbase(self.cluster, self.config, self.cloudera_manager)
 hbase_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 7: Update KAFKA configuration.
 #
 # Deploy Kerberos Config Kafka
 #
 logging.debug("Deploy Service Configuration!!!")
 kafka_service = Kafka(self.cluster, self.config, self.cloudera_manager)
 kafka_service.update_configuration()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 8: Update HUE configuration.
 #
 # Deploy Kerberos Config Hue
 #
 logging.debug("Deploy Service Configuration!!!")
 hue_service = Hue(self.cluster, self.config, self.cloudera_manager)
 hue_service.update_configuration()
 hue_service.add_service_kt_renewer_to_cluster()
 logging.debug("Deploy Service Configuration COMPLETE!!!")
Step 9: Generating Credentials.
 #
 # Generated kerberos credentials in AD.
 #
 self.cm_generate_credentials()
 time.sleep(5)
Step 10: Deploy Client Configuration.
 #
 # Deploy Client Configuration
 #
 logging.info("Deploying Client Config...")
 self.deploy_client_configuration()
 logging.debug("Waiting for CLUSTER to deploy completely !!!")
 time.sleep(5)
Step 11: Starting CM Services.
 #
 # Starting CM services.
 #
 self.get_mgmt_services.start().wait()
 logging.debug("Waiting for CM to start completely !!!")
 time.sleep(5)
Step 12: Starting cluster services.
 #
 # Restart Cluster based on stale configuration and redeploy config if required.
 #
 self.gets_cm_cluster.start().wait()
 logging.info("Cluster Kerberos Deployment Complete.")

Important configuration information.

LDAP setup.
 'cm_ldap': {
     'KDC_HOST': 'adserver1.service.ahmedinc.com',
     'SECURITY_REALM': 'SERVICE.AHMEDINC.COM',
     'KRB_ENC_TYPES': 'rc4-hmac',
     'KDC_ACCOUNT_CREATION_HOST_OVERRIDE': 'adserver2.service.ahmedinc.com',
     'AD_KDC_DOMAIN': 'OU=accounts,OU=test-lab-ou,OU=bigdata,DC=service,DC=ahmedinc,DC=com',
     'AD_DELETE_ON_REGENERATE': True,
     'AD_ACCOUNT_PREFIX': 'Lab1Test'
   }
LDAP credentials for user with create, delete, modify permissions.
 'cm_kdc_import_credentials': {
     'kdc_password': 'Ahm3d@123',
     'kdc_username': 'service-acc-lab@SERVICE.AHMEDINC.COM'
 }
Configuration to connect to Cloudera Manager.
 'cm': {
     'username': 'admin',
     'tls': False,
     'host': 'server-admin-node.ahmedinc.com',
     'api-version': 13,
     'password': 'admin',
     'port': 7180
 }
Cluster configuration.
 'cluster': {
     'version': 'CDH5',
     'hosts': [
       'server-admin-node.ahmedinc.com',
       'server-edge-node.ahmedinc.com',
       'server-worker-node.ahmedinc.com'
     ],
     'name': 'AutomatedHadoopCluster',
     'fullVersion': '5.8.3'
 }
Service Configuration Hdfs.
 'HDFS': {
   'config': {
     'hadoop_security_authentication': 'kerberos',
     'trusted_realms': 'USERACC.AHMEDINC.COM',
     'hadoop_security_authorization': True
   },
   'roles': [
     {
       'config': {
         'dfs_datanode_http_port': 1006,
         'dfs_datanode_data_dir_perm': 700,
         'dfs_datanode_port': 1004
       },
       'hosts': [
         'server-admin-node.ahmedinc.com',
         'server-edge-node.ahmedinc.com',
         'server-worker-node.ahmedinc.com'
       ],
       'group': 'DATANODE'
     }
   ]
 }
Service Configuration Hbase.
 'HBASE': {
   'config': {
     'hbase_thriftserver_security_authentication': 'auth',
     'hbase_security_authorization': True,
     'hbase_security_authentication': 'kerberos'
   }
 }
Service Configuration Zookeeper.
 'ZOOKEEPER': {
   'config': {
     'enableSecurity': True
   }
 }
Service Configuration Kafka.
 'KAFKA': {
   'config': {
     'kerberos.auth.enable': True
   }
 }
Service Configuration Hue.
 'HUE': {
   'roles': [
     {
       'group': 'KT_RENEWER',
       'hosts': [
         'server-admin-node.ahmedinc.com'
       ]
     }
   ]
 }
NOTE : In a kerberos setup, when we do a update_config, generate_missing_credentials command is triggered.
  • we need to wait for this command to complete, before we start the next update_config or else generate_credential command will FAIL.
  • get_commands will return a List of ApiCommand responses. we iterate through all the command to complete. [currently we will receive only one command]
  • get_commands
  • Code snippet.
    current_running_commands = self.cm.get_commands()
    logging.debug("Current Running Commands: " + str(current_running_commands))
    if not current_running_commands:
      logging.info("Currently no command is running.")
    elif current_running_commands:
      for cmd in current_running_commands:
          if cmd.wait().success:
              logging.info("Command Completed. moving on..")
    else:
      logging.error("We are in ELSE, something went wrong. :(")
      sys.exit(1)
    
    logging.info("Configuration Updated Successfully..")
    time.sleep(5)
    
    Update Configuration for Cloudera Manager LDAP.
    def deploy_cm_ldap_configuration(self):
      """
          Updating LDAP configuratio no CM server.
          This will require a `sudo servive cloudera-scm-server restart`, we are doing a restart in the end of the script.
      :return:
      """
      cm_api_handle = ApiResource(config['cm']['host'],
                                            config['cm']['port'],
                                            config['cm']['username'],
                                            config['cm']['password'],
                                            config['cm']['tls'],
                                            version=config['cm']['api-version'])
      cluster = cm_api_handle.get_cluster(config['cluster']['name'])
      cloudera_manager = cm_api_handle.get_cloudera_manager()
      cloudera_manager.update_config(self.config['cm_ldap'])
    
    Example Configuration update for Zookeeper.
    def update_configuration(self):
      """
          Update service configurations
      :return:
      """
      cm_api_handle = ApiResource(config['cm']['host'],
                                            config['cm']['port'],
                                            config['cm']['username'],
                                            config['cm']['password'],
                                            config['cm']['tls'],
                                            version=config['cm']['api-version'])
      cluster = cm_api_handle.get_cluster(config['cluster']['name'])
      get_service = cluster.get_service('ZOOKEEPER')
      get_service.update_config(config['ZOOKEEPER']['config'])
      logging.info("Service Configuration Updated.")
    
      #   In a kerberos setup, when we do a `update_config`
      #      `generate_missing_credentials` command is triggered.
      #      we need to wait for this command to complete,
      #      before we start the next `update_config`
      #      or else `generate_credential` command will FAIL.
    
      cm = cm_api_handle.get_cloudera_manager()
      current_running_commands = cm.get_commands()
      logging.debug("Current Running Commands: " + str(current_running_commands))
      if not current_running_commands:
          logging.info("Currently no command is running.")
      elif current_running_commands:
          for cmd in current_running_commands:
              if cmd.wait().success:
                  logging.info("Command Completed. moving on..")
      else:
          logging.error("We are in ELSE, something went wrong. :(")
          sys.exit(1)
    
      logging.info("Configuration Updated Successfully..")
      time.sleep(5)
    
    Creating KT_RENEWER code snippet.
 def create_kt_renewer_service(self):
     
     
     cm_api_handle = ApiResource(config['cm']['host'],
                                           config['cm']['port'],
                                           config['cm']['username'],
                                           config['cm']['password'],
                                           config['cm']['tls'],
                                           version=config['cm']['api-version'])
     cluster = cm_api_handle.get_cluster(config['cluster']['name'])
     get_service = cluster.get_service('HUE')
     
     
     for role in self.config['HUE']['roles']:
         role_group = self.get_service.get_role_config_group("{0}-{1}-BASE".format('HUE', role['group']))
         # Update the group's configuration.
         # [https://cloudera.github.io/cm_api/epydoc/5.10.0/cm_api.endpoints.role_config_groups.ApiRoleConfigGroup-class.html#update_config]
         role_group.update_config(role.get('config', {}))
         self.create_roles(get_service, role, role['group']) 
Creating Roles.
 def create_roles(self, service, role, group):
     """
     Create individual roles for all the hosts under a specific role group
     
     :param role: Role configuration from yaml
     :param group: Role group name
     """
     role_id = 0
     for host in role.get('hosts', []):
         role_id += 1
         role_name = '{0}-{1}-{2}'.format('HUE', group, role_id)
         logging.info("Creating Role name as: " + str(role_name))
         try:
             service.get_role(role_name)
         except ApiException:
             service.create_role(role_name, group, host) 
Example Configuration update for Zookeeper.

from Blogger http://ift.tt/2n2oPvG

via IFTTT

Categories: Others Tags: ,

Setting Up HDFS Services Using Cloudera API [Part 3]

March 24, 2017 Leave a comment
This is the second follow up post. In the earlier post
  1. Create a cluster.
  2. Install HDFS service to our cluster.
    Creating a Cluster and setting up parcel is part of earlier post.

Install HDFS Service.

HDFS service is installed in stages.
  1. Create a HDFS service (if not exist).
  2. Update configuration for our newly create HDFS service.
  3. Create HDFS roles (NAMENODE, SECONDARYNAMENODE, DATANODE, GATEWAY) on the Cluster.
  4. Format Namenode.
  5. Start HDFS service.
  6. Create Temporary /tmp directory in HDFS

    Create a HDFS service.

    This is simple create a service if it does not exist.
    def create_service(cluster):
     try:
         zk_service = cluster.get_service('HDFS')
         logging.debug("Service {0} already present on the cluster".format('HDFS'))
     except ApiException:
         #
         # Create service if it the first time.
         #
         zk_service = cluster.create_service('HDFS', 'HDFS')
         logging.info("Created New Service: HDFS")
    
     return zk_service
    

    Update configuration for HDFS.

    This information is picked up from the configuration yaml file.
    yaml file.
    HDFS:
     config:
       dfs_replication: 3
       dfs_permissions: false
       dfs_block_local_path_access_user: impala,hbase,mapred,spark
     roles:
       - group: NAMENODE
         hosts:
           - mycmhost.ahmed.com
         config:
           dfs_name_dir_list: /data/1/dfs/nn,/data/2/dfs/nn
           dfs_namenode_handler_count: 30
       - group: SECONDARYNAMENODE
         hosts:
           - mycmhost.ahmed.com
         config:
           fs_checkpoint_dir_list: /data/1/dfs/snn,/data/2/dfs/snn
    
       - group: DATANODE
         hosts:
           - mycmhost.ahmed.com
         config:
           dfs_data_dir_list: /data/1/dfs/dn,/data/2/dfs/dn
           dfs_datanode_handler_count: 30
           #dfs_datanode_du_reserved: 1073741824
           dfs_datanode_failed_volumes_tolerated: 0
           dfs_datanode_data_dir_perm: 755
       - group: GATEWAY
         hosts:
           - mycmhost.ahmed.com
         config:
           dfs_client_use_trash: true
    
    Code snippet.
    def service_update_configuration(zk_service):
     """
         Update service configurations
     :return:
     """
     zk_service.update_config(config['services']['HDFS']['config'])
     logging.info("Service Configuration Updated.")
    

    Create HDFS roles (NAMENODE, SECONDARYNAMENODE, DATANODE, GATEWAY) on the Cluster.

    To create all the roles.
  • Each role needs to be unique on each host.
  • We create a unique role_name for each node.
    Each role is unique based on below set of strings. (service_name, group, role_id)
    role_name = '{0}-{1}-{2}'.format(service_name, group, role_id)
    
    Here is the code snippet.
    def hdfs_create_cluster_services(config, service, service_name):
      """
          Creating Cluster services
      :return:
      """
    
      #
      # Get the role config for the group
      # Update group configuration and create roles.
      #
      for role in config['services'][service_name]['roles']:
          role_group = service.get_role_config_group("{0}-{1}-BASE".format(service_name, role['group']))
          #
          # Update the group's configuration.
          # [http://ift.tt/2o1ijoO]
          #
          role_group.update_config(role.get('config', {}))
          #
          # Create roles now.
          #
          hdfs_create_roles(service, service_name, role, role['group'])
    
    def hdfs_create_roles(service, service_name, role, group):
      """
      Create individual roles for all the hosts under a specific role group
    
      :param role: Role configuration from yaml
      :param group: Role group name
      """
      role_id = 0
      for host in role.get('hosts', []):
          role_id += 1
          role_name = '{0}-{1}-{2}'.format(service_name, group, role_id)
          logging.info("Creating Role name as: " + str(role_name))
          try:
              service.get_role(role_name)
          except ApiException:
              service.create_role(role_name, group, host)
    

Format Namenode.

First time when we create a HDFS environment we need to format namenode, this init the HDFS cluster.
format_hdfs method returns as ApiCommand which we can track progress of execution.
 def format_namenode(hdfs_service, namenode):
     try:
         #
         # Formatting HDFS - this will have no affect the second time it runs.
         # Format NameNode instances of an HDFS service.
         #
         # http://ift.tt/2nc7voC
         #

         cmd = hdfs_service.format_hdfs(namenode)[0]
         logging.debug("Command Response: " + str(cmd))
         if not cmd.wait(300).success:
             print "WARNING: Failed to format HDFS, attempting to continue with the setup"
     except ApiException:
         logging.info("HDFS cannot be formatted. May be already in use.")

Start HDFS service

We do this using the service.start() method. This method return ApiCommand which we can track the progress and wait for the service to start using cmd.wait().success
More details about the Api here
Our service should be up and running.

Finally Creating /tmp directory on HDFS.

When we create a HDFS cluster we create a /tmp directory, HDFS /tmp directory is used as a temporary storage during mapreduce operation.
Mapreduce artifacts, intermediate data will be kept under this directory. If we delete the /tmp contents then any MR jobs currently running will loose its current intermediate data.
Any MR run after the /tmp is clear will still work without any issues.
Creating /tmp is done using create_hdfs_tmp method which returns ApiCommand response.

Code Location

‚Äč

from Blogger http://ift.tt/2nhFBct
via IFTTT

Categories: Others Tags: ,

Setting Up Zookeeper Services Using Cloudera API [Part 2]

March 23, 2017 Leave a comment
This is the second follow up post. In the earlier post Setting Up Cloudera Manager Services Using Cloudera API [Part 1] we install the cloudera management services. Now we will be installing Zookeeper service to the cluster.
But first we need to to couple of things before we install Zookeeper.
  1. Create a cluster.
  2. Download, Distribute, Activate CDH Parcels.
  3. Install Zookeeper service to our cluster.

Creating a Cluster

First we create a cluster if it does not exists. Config details below.
def init_cluster(cm_api_handle):
    try:
        cluster = cm_api_handle.get_cluster(config['cluster']['name'])
        return cluster
    except ApiException:
        cluster = cm_api_handle.create_cluster(config['cluster']['name'],
                                                config['cluster']['version'],
                                                config['cluster']['fullVersion'])
If it is the first time then we need to add all the hosts to the cluster (including the admin node), this information is coming from the configuration yaml file.
# Basic cluster information
cluster:
  name: AutomatedHadoopCluster
  version: CDH5
  fullVersion: 5.8.3
  hosts:
     - mycmhost.ahmed.com
In the above yaml snippet hosts will have all the hosts in the cluster, since we are testing on a VM with just one host we have that host added there.

    cluster_hosts = []
    #
    # Picking up all the nodes from the yaml configuration.
    #
    for host_in_cluster in cluster.list_hosts():
        cluster_hosts.append(host_in_cluster)

    hosts = []

    #
    # Create a host list, make sure we dont have duplicates.
    #
    for host in config['cluster']['hosts']:
        if host not in cluster_hosts:
            hosts.append(host)

    #
    # Adding all hosts to the cluster.
    #
    cluster.add_hosts(hosts)
    return cluster
Once we have the cluster ready then we can install the parcels to the cluster.

Download, Distribute, Activate CDH Parcels

For Parcels there are three major methods which initiate the parcels distribution.
  1. parcel.start_download()
  2. parcel.start_distribution()
  3. parcel.activate()
These are the three methods which do all the work. When the start_download command is running we need to keep track of the progress, this is done using the get_parcels method.
get_parcelshttp://ift.tt/2nD3tbG
#
# When we execute and parcel download/distribute/activate command
# we can track the progress using the `get_parcel` method.
# This return a JSON described here : http://ift.tt/2nD3tbG
# We can check progress by checking `stage`
#
#   AVAILABLE_REMOTELY: Stable stage - the parcel can be downloaded to the server.
#   DOWNLOADING: Transient stage - the parcel is in the process of being downloaded to the server.
#   DOWNLOADED: Stable stage - the parcel is downloaded and ready to be distributed or removed from the server.
#   DISTRIBUTING: Transient stage - the parcel is being sent to all the hosts in the cluster.
#   DISTRIBUTED: Stable stage - the parcel is on all the hosts in the cluster. The parcel can now be activated, or removed from all the hosts.
#   UNDISTRIBUTING: Transient stage - the parcel is being removed from all the hosts in the cluster>
#   ACTIVATING: Transient stage - the parcel is being activated on the hosts in the cluster. New in API v7
#   ACTIVATED: Steady stage - the parcel is set to active on every host in the cluster. If desired, a parcel can be deactivated from this stage.
#
We track the progress of each stage using the snippet below.
def check_current_state(cluster, product, version, states):
    logging.info("Checking Status for Parcel.")
    while True:
        parcel = cluster.get_parcel(product, version)
        logging.info("Parcel Current Stage: " + str(parcel.stage))
        if parcel.stage in states:
            break
        if parcel.state.errors:
            raise Exception(str(parcel.state.errors))

        logging.info("%s progress: %s / %s" % (states[0], parcel.state.progress,
                                               parcel.state.totalProgress))
        time.sleep(15)
Rest of the parcel execute is straight forward.

Install Zookeeper Service.

Zookeeper service is installed in stages.
  1. Create a service (if not exist)
  2. Update configuration for our newly create Zookeeper service.
  3. Create Zookeeper role (SERVER) on the Cluster.
  4. Initalize Zookeeper using the init_zookeeper() command.
  5. Start Zookeeper service.

Create a service.

This is simple create a service if it does not exist.
def zk_create_service(cluster):
    try:
        zk_service = cluster.get_service('ZOOKEEPER')
        logging.debug("Service {0} already present on the cluster".format(self.name))
    except ApiException:
        #
        # Create service if it the first time.
        #
        zk_service = cluster.create_service('ZOOKEEPER', 'ZOOKEEPER')
        logging.info("Created New Service: ZOOKEEPER")

    return zk_service

Update configuration for Zookeeper.

This information is picked up from the configuration yaml file.
yaml file.
  ZOOKEEPER:
    config:
      zookeeper_datadir_autocreate: true
Code snippet.
def zk_update_configuration(zk_service):
    """
        Update service configurations
    :return:
    """
    zk_service.update_config(config['services']['ZOOKEEPER']['config'])
    logging.info("Service Configuration Updated.")

Create Zookeeper role (SERVER) on the Cluster.

This is the important part, here we create Zookeeper roles SERVER (each instance of zookeeper on each server is referred as a SERVER role).
Role names should be unique, we combine the service_name, role, zookeeper_id to form a unique identifier. Example : Here it would be ZOOKEEPER-SERVER-1
Here is the code snippet.
zookeeper_host_id = 0

    #
    # Configure all the host.
    #
    for zookeeper_host in config['services']['ZOOKEEPER']['roles'][0]['hosts']:
        zookeeper_host_id += 1
        zookeeper_role_config = config['services']['ZOOKEEPER']['roles'][0]['config']
        role_name = "{0}-{1}-{2}".format('ZOOKEEPER', 'SERVER', zookeeper_host_id)
Next once we create the role we update the configuration which we get from the yaml file.
Yaml file.
roles:
  - group: SERVER
    hosts:
      - mycmhost.ahmed.com
    config:
      quorumPort: 2888
      electionPort: 3888
      dataLogDir: /var/lib/zookeeper
      dataDir: /var/lib/zookeeper
Code snippet.
#
# Configuring Zookeeper server ID
#
zookeeper_role_config['serverId'] = zookeeper_host_id

#
# Update configuration
#
role.update_config(zookeeper_role_config)
Now we are set to start the Zookeeper.

Initalize Zookeeper using the init_zookeeper() command.

Before we start Zookeeper we need to init the service, which create id for each service running on each server. ps: it creates the my_id for each service in /var/lib/zookeeper location. This will help each Zookeeper identify itself as unique.
If there were 3 Zookeeper servers (which is minimum recommended) we would have something like below.
  1. Role ZOOKEEPER-SERVER-1, Server 1, Zookeeper ID my_id 1
  2. Role ZOOKEEPER-SERVER-2, Server 2, Zookeeper ID my_id 2
  3. Role ZOOKEEPER-SERVER-3, Server 3, Zookeeper ID my_id 3
And so on.
Once we have the service initialized we are ready to start the service.

Start Zookeeper service

We do this using the zk_service.start() method. This method return ApiCommand which we can track the progress and wait for the service to start using cmd.wait().success
More details about the Api here
Our service should be up and running.

Yaml File

Code File.

Executing Code

‚Äč

from Blogger http://ift.tt/2nD1hAV
via IFTTT

Categories: Others Tags: ,

Setting Up Cloudera Manager Services Using Cloudera API [Part 1]

March 22, 2017 Leave a comment
Cloudera API is a very convenient way to setup a cluster and do more.

Here are some of the cool things you can do with [Cloudera Manager via the API](http://ift.tt/2mmdiLd):

– Deploy an entire Hadoop cluster programmatically. Cloudera Manager supports HDFS, MapReduce, YARN, ZooKeeper, HBase, Hive, Oozie, Hue, Flume, Impala, Solr, Sqoop, Spark and Accumulo.
– Configure various Hadoop services and get config validation.
– Take admin actions on services and roles, such as start, stop, restart, failover, etc. Also available are the more advanced workflows, such as setting up high availability and decommissioning.
– Monitor your services and hosts, with intelligent service health checks and metrics.
– Monitor user jobs and other cluster activities.
– Retrieve timeseries metric data.
– Search for events in the Hadoop system.
– Administer Cloudera Manager itself.
– Download the entire deployment description of your Hadoop cluster in a json file.

Additionally, with the appropriate licenses, the API lets you:

– Perform rolling restart and rolling upgrade.
– Audit user activities and accesses in Hadoop.
– Perform backup and cross data-center replication for HDFS and Hive.
– Retrieve per-user HDFS usage report and per-user MapReduce resource usage report.

### Prerequisites [IMPORTANT]

**Assuming initial setup is complete. [Cloudera Manager Setup Using Chef](http://ift.tt/2njFAVW)**

1. Mysql is install and all databases are created.
2. Cloudera Manager server is installed and configured.
3. Cloudera Manager repo is setup (optional if you are using internal repo else update yaml)

### Before we start.

**NOTE: We will be using Mysql for database and will assume that the database is already installed and all the necessary databases are created with users.**

1. We need to understand the services and setup configuration for each service.
2. Setup database configuration.

### Currently we will be doing below operation in this post.

– Install Hosts.
– Setup `Cloudera Manager Services` using API.
    – Activity Monitor
    – Alert Publisher
    – Event Server
    – Host Monitor
    – Reports Manager
    – Service Monitor

### Steps for the current task.

1. Set License (or set trial for the setup)
2. Install Hosts
3. Configure and setup all the services above.
3. Start Services on cloudera manager.

### Activity Monitor

Cloudera Manager’s activity monitoring capability monitors the MapReduce, Pig, Hive, Oozie, and streaming jobs, Impala queries, and YARN applications running or that have run on your cluster. When the individual jobs are part of larger workflows (using Oozie, Hive, or Pig), these jobs are aggregated into MapReduce jobs that can be monitored as a whole, as well as by the component jobs. [Courtesy Cloudera Website](http://ift.tt/2o1mXmQ)

The following sections describe how to view and monitor activities that run on your cluster.

– [Monitoring MapReduce Jobs](http://ift.tt/2njHHJz)
– [Monitoring Impala Queries](http://ift.tt/2o1fuUY)
– [Monitoring YARN Applications](http://ift.tt/2njAUzB)
– [Monitoring Spark Applications](http://ift.tt/2o1wVod)

#### Configuration Setup Information for Activity Monitor.

Complete details about the configuration. [Cloudera Activity Monitor](http://ift.tt/2njAPvH)

“` json
{
  ‘config’: {
    ‘firehose_database_host’: ‘server-admin-node.ahmedinc.com:3306’,
    ‘firehose_database_password’: ‘amon_password’,
    ‘firehose_database_user’: ‘amon’,
    ‘firehose_database_type’: ‘mysql’,
    ‘firehose_database_name’: ‘amon’
  },
  ‘hosts’: [
    ‘server-admin-node.ahmedinc.com’
  ],
  ‘group’: ‘ACTIVITYMONITOR’
}
“`

For activity monitor we need to set the database and will leave other configuration as default which cloudera manager will take care. Activity monitor was earlier called as `firehose`. So the configuration information still has `firehose` in it.

– Group Name `ACTIVITYMONITOR` (This the information which cloudera manager uses to know which service it needs to create)
– DB Host `admin-node` or the node which is hosting the `mysql` database.

### Reports Manager

The Reports page lets you create reports about the usage of HDFS in your cluster‚ÄĒdata size and file count by user, group, or directory. It also lets you report on the MapReduce activity in your cluster, by user.

“` json
‘config’: {
    ‘headlamp_database_name’: ‘rman’,
    ‘headlamp_database_user’: ‘rman’,
    ‘headlamp_database_type’: ‘mysql’,
    ‘headlamp_database_password’: ‘rman_password’,
    ‘headlamp_database_host’: ‘server-admin-node.ahmedinc.com:3306’
},
    ‘hosts’: [
    ‘server-admin-node.ahmedinc.com’
    ],
    ‘group’: ‘REPORTSMANAGER’
“`

– Group Name `REPORTSMANAGER` (This the information which cloudera manager uses to know which service it needs to create)

### Rest of the service Alert Publiser, Event Server, Host Monitor, Service Monitor

– **Alert Publisher** can be to send alert notifications by email or by SNMP trap to a trap receiver.
– **Event Server** aggregates relevant events and makes them available for alerting and for searching. This way, you have a view into the history of all relevant events that occur cluster-wide.
– **Host Monitoring** features let you manage and monitor the status of the hosts in your clusters.
– **Service Monitoring** feature monitors dozens of service health and performance metrics about the services and role instances running on your cluster:
    – Presents health and performance data in a variety of formats including interactive charts
    – Monitors metrics against configurable thresholds
    – Generates events related to system and service health and critical log entries and makes them available for searching and alerting
    – Maintains a complete record of service-related actions and configuration changes

Group Names : `ALERTPUBLISHER``EVENTSERVER``HOSTMONITOR``SERVICEMONITOR`

## Creating Script For Deploying Cloudera Management Services. 

Tasks we need to work on.

1. Create a configuration JSON file. (We will be using the same JSON above)
2. Setup license for the cluster (we will be setting trial version)
3. Install Hosts.
4. Initialize Cluster for the First time.
5. Deploy Management Roles (Update service configuration and create roles for each service).

### Creating JSON file.

Host Installation

“` json
‘cm_host_installation’: {
    ‘host_cm_repo_gpg_key_custom_url’: ‘http://ift.tt/2o1gfxp&#8217;,
    ‘host_java_install_strategy’: ‘AUTO’,
    ‘host_cm_repo_url’: ‘http://ift.tt/2njHnug&#8217;,
    ‘host_unlimited_jce_policy’: True,
    ‘host_password’: ‘Bigdata@123’,
    ‘host_username’: ‘cmadmin’,
    ‘ssh_port’: 22
}
“`

Cloudera Manager Credentials and Remote Repo setup.

“` json
‘cm’: {
    ‘username’: ‘admin’,
    ‘tls’: False,
    ‘host’: ‘server-admin-node.ahmedinc.com’,
    ‘api-version’: 13,
    ‘remote_parcel_repo_urls’: ‘http://ift.tt/2o1Emvw&#8217;,
    ‘password’: ‘admin’,
    ‘port’: 7180
}
“`

Cluster Information.

“` json
‘cluster’: {
    ‘version’: ‘CDH5’,
    ‘hosts’: [
      ‘server-admin-node.ahmedinc.com’,
      ‘server-edge-node.ahmedinc.com’,
      ‘server-worker-node.ahmedinc.com’
    ],
    ‘name’: ‘AutomatedHadoopCluster’,
    ‘fullVersion’: ‘5.8.3’
}
“`

Management Service.

“` json
‘MGMT’: {
      ‘roles’: [
        {
          ‘config’: {
            ‘firehose_database_host’: ‘server-admin-node.ahmedinc.com:3306’,
            ‘firehose_database_password’: ‘amon_password’,
            ‘firehose_database_user’: ‘amon’,
            ‘firehose_database_type’: ‘mysql’,
            ‘firehose_database_name’: ‘amon’
          },
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ],
          ‘group’: ‘ACTIVITYMONITOR’
        },
        {
          ‘group’: ‘ALERTPUBLISHER’,
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ]
        },
        {
          ‘group’: ‘EVENTSERVER’,
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ]
        },
        {
          ‘group’: ‘HOSTMONITOR’,
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ]
        },
        {
          ‘group’: ‘SERVICEMONITOR’,
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ]
        },
        {
          ‘config’: {
            ‘headlamp_database_name’: ‘rman’,
            ‘headlamp_database_user’: ‘rman’,
            ‘headlamp_database_type’: ‘mysql’,
            ‘headlamp_database_password’: ‘rman_password’,
            ‘headlamp_database_host’: ‘server-admin-node.ahmedinc.com:3306’
          },
          ‘hosts’: [
            ‘server-admin-node.ahmedinc.com’
          ],
          ‘group’: ‘REPORTSMANAGER’
        }
      ]
    }
“`

### License Update.

First we get license for the cluster, if we receive an exception the we set trial version.

“` python
def enable_license_for_cm(cloudera_manager)
    try:
        # Check for current License.
        cloudera_license = cloudera_manager.get_license()
    except ApiException:
        # If we recieve an exception, then this is first time we are setting the license
        cloudera_manager.begin_trial()
“`

### Installing Hosts.

Once we have set the lic installed, we need to setup the hosts to have all the services running. `host_install` command will install all the services like `agent``daemon``java` and `jce_policy` files. This will also configure the hosts to contact the admin-node for heartbeat.

Below is a code snippet to install hosts.

“` python
def host_installation(cloudera_manager, config):
    “””
        Host installation.
        http://ift.tt/2njKnXE
    “””
    logging.info(“Installing HOSTs.”)
    cmd = cloudera_manager.host_install(config[‘cm_host_installation’][‘host_username’],
                                   config[‘cluster’][‘hosts’],
                                   ssh_port=config[‘cm_host_installation’][‘ssh_port’],
                                   password=config[‘cm_host_installation’][‘host_password’],
                                   parallel_install_count=10,
                                   cm_repo_url=config[‘cm_host_installation’][‘host_cm_repo_url’],
                                   gpg_key_custom_url=config[‘cm_host_installation’][‘host_cm_repo_gpg_key_custom_url’],
                                   java_install_strategy=config[‘cm_host_installation’][‘host_java_install_strategy’],
                                   unlimited_jce=config[‘cm_host_installation’][‘host_unlimited_jce_policy’])

    #
    # Check command to complete.
    #
    if not cmd.wait().success:
        logging.info(“Command `host_install` Failed. {0}”.format(cmd.resultMessage))
        if (cmd.resultMessage is not None and
                    ‘There is already a pending command on this entity’ in cmd.resultMessage):
            raise Exception(“HOST INSTALLATION FAILED.”)
“`

### Deploy Management Services.

First we need to create a management services if it does not exsist.

“` python
mgmt_service = cloudera_manager.create_mgmt_service(ApiServiceSetupInfo()) 
“`

Adding all services based on roles.

“` python 
for role in config[‘services’][‘MGMT’][‘roles’]:
    if not len(mgmt_service.get_roles_by_type(role[‘group’])) > 0:
        logging.info(“Creating role for {0}”.format(role[‘group’]))
        mgmt_service.create_role(‘{0}-1’.format(role[‘group’]), role[‘group’], role[‘hosts’][0])
“`

Update configuration

“` python
for role in config[‘services’][‘MGMT’][‘roles’]:
    role_group = mgmt_service.get_role_config_group(‘mgmt-{0}-BASE’.format(role[‘group’]))
    logging.info(role_group)
    #
    # Update the group’s configuration.
    # [http://ift.tt/2o1ijoO]
    #
    role_group.update_config(role.get(‘config’, {}))
“`

Now we start the service.

“` python
#
# Start mgmt services.
#
mgmt_service.start().wait()
“`

### Yaml File

http://ift.tt/2njFBcs

### Code File

http://ift.tt/2o1nyVt

### Executing Code.

[Video – Setting Up Cloudera Manager Services Using Cloudera API](https://youtu.be/kH1DhfQuD5M)

### Useful Links.

– [Ansible Hadoop Playbook](http://ift.tt/2njI1rR)
– [Cloudera API Example](http://ift.tt/1ug59pN)
– [Cloudera API](http://ift.tt/2njLJBE)
– [Cloudera Epy Document](http://ift.tt/2n1tZsE)
– [Cloudera API Getting Started](http://ift.tt/2njEd9E)
– [Cloudera API Properties](http://ift.tt/2o1tpdF)
– [Cloudera Manager Server Properties](http://ift.tt/2njSLWV)
– [Cloudera Manager Service Properties](http://ift.tt/2njAPvH)

from Blogger http://ift.tt/2o1kMjg
via IFTTT

Categories: Others Tags: ,

Getting Started with Cloudera API

March 21, 2017 Leave a comment
This is a basic steps to get connected with cloudera manager.
Here are some of the cool things you can do with Cloudera Manager via the API:
  • Deploy an entire Hadoop cluster programmatically. Cloudera Manager supports HDFS, MapReduce, YARN, ZooKeeper, HBase, Hive, Oozie, Hue, Flume, Impala, Solr, Sqoop, Spark and Accumulo.
  • Configure various Hadoop services and get config validation.
  • Take admin actions on services and roles, such as start, stop, restart, failover, etc. Also available are the more advanced workflows, such as setting up high availability and decommissioning.
  • Monitor your services and hosts, with intelligent service health checks and metrics.
  • Monitor user jobs and other cluster activities.
  • Retrieve timeseries metric data.
  • Search for events in the Hadoop system.
  • Administer Cloudera Manager itself.
  • Download the entire deployment description of your Hadoop cluster in a json file.
    Additionally, with the appropriate licenses, the API lets you:
  • Perform rolling restart and rolling upgrade.
  • Audit user activities and accesses in Hadoop.
  • Perform backup and cross data-center replication for HDFS and Hive.
  • Retrieve per-user HDFS usage report and per-user MapReduce resource usage report.

API Installations.

Getting Connected To Cloudera Manager.

First we get a API handle to use to connect to Cloudera Manager Services and Cluster Services. config is coming from a yaml file.
 @property
 def cm_api_handle(self):

     """
         This method is to create a handle to CM.
     :return: cm_api_handle
     """
     if self._cm_api_handle is None:
         self._cm_api_handle = ApiResource(self.config['cm']['host'],
                                           self.config['cm']['port'],
                                           self.config['cm']['username'],
                                           self.config['cm']['password'],
                                           self.config['cm']['tls'],
                                           version=self.config['cm']['api-version'])
     return self._cm_api_handle
A simple way to write it would be as below. (I am using version=13 here)
 cm_api_handle = api = ApiResource(cm_host, username="admin", password="admin", version=13)
Now we can use this handle to connect to CM or Cluster. Now lets look at what we can do once we have the cloudera_manager object.
We can do all these method calls on this. CM Class
 cloudera_manager = cm_api_handle.get_cloudera_manager()
 cloudera_manager.get_license()

 cm_api_response = cloudera_manager.get_services()
Here API response cm_api_response would be APIService
Similarly we get cluster methods for the cluster but in a List format as there are many services on the cluster.
 cloudera_cluster = cm_api_handle.get_cluster("CLUSTER_NAME")
 cluster_api_response = cloudera_cluster.get_all_services()
Again the response would be a ApiService.

Example Code to get started.

First lets create a yaml file.
 # Cloudera Manager config
 cm:
   host: 127.0.0.1
   port: 7180
   username: admin
   password: admin
   tls: false
   version: 13

 # Basic cluster information
 cluster:
   name: AutomatedHadoopCluster
   version: CDH5
   fullVersion: 5.8.3
   hosts:
      - 127.0.0.1
Next we create script to process that data.
 import yaml
 import sys
 from cm_api.api_client import ApiResource, ApiException


 def fail(msg):
     print (msg)
     sys.exit(1)

 if __name__ == '__main__':

     try:
         with open('cloudera.yaml', 'r') as cluster_yaml:
             config = yaml.load(cluster_yaml)

         api_handle = ApiResource(config['cm']['host'],
                                  config['cm']['port'],
                                  config['cm']['username'],
                                  config['cm']['password'],
                                  config['cm']['tls'],
                                  version=config['cm']['version'])

         # Checking CM services
         cloudera_manager = api_handle.get_cloudera_manager()
         cm_api_response = cloudera_manager.get_service()

         print "\nCLOUDERA MANAGER SERVICES\n----------------------------"
         print "Complete ApiService: " + str(cm_api_response)
         print "Check URL for details : http://ift.tt/2n1dYCL"
         print "name: " + str(cm_api_response.name)
         print "type: " + str(cm_api_response.type)
         print "serviceUrl: " + str(cm_api_response.serviceUrl)
         print "roleInstancesUrl: " + str(cm_api_response.roleInstancesUrl)
         print "displayName: " + str(cm_api_response.displayName)

         # Checking Cluster services
         cm_cluster = api_handle.get_cluster(config['cluster']['name'])
         cluster_api_response = cm_cluster.get_all_services()
         print "\n\nCLUSTER SERVICES\n----------------------------"
         for api_service_list in cluster_api_response:
             print "Complete ApiService: " + str(api_service_list)
             print "Check URL for details : http://ift.tt/2n1dYCL"
             print "name: " + str(api_service_list.name)
             print "type: " + str(api_service_list.type)
             print "serviceUrl: " + str(api_service_list.serviceUrl)
             print "roleInstancesUrl: " + str(api_service_list.roleInstancesUrl)
             print "displayName: " + str(api_service_list.displayName)

     except IOError as e:
         fail("Error creating cluster {}".format(e))
Output
 CLOUDERA MANAGER SERVICES
 ----------------------------
 Complete ApiService: : mgmt (cluster: None)
 Check URL for details : http://ift.tt/2n1dYCL
 name: mgmt
 type: MGMT
 serviceUrl: http://ift.tt/2n1dfll:7180/cmf/serviceRedirect/mgmt
 roleInstancesUrl: http://ift.tt/2n1dfll:7180/cmf/serviceRedirect/mgmt/instances
 displayName: Cloudera Management Service


 CLUSTER SERVICES
 ----------------------------
 Complete ApiService: : ZOOKEEPER (cluster: AutomatedHadoopCluster)
 Check URL for details : http://ift.tt/2n1dYCL
 name: ZOOKEEPER
 type: ZOOKEEPER
 serviceUrl: http://ift.tt/2n1dfll:7180/cmf/serviceRedirect/ZOOKEEPER
 roleInstancesUrl: http://ift.tt/2n1dfll:7180/cmf/serviceRedirect/ZOOKEEPER/instances
 displayName: ZOOKEEPER
This is the basics, we will be build on top of this in coming blog posts.
‚Äč

from Blogger http://ift.tt/2mlZikz
via IFTTT

Categories: Others Tags: ,