Archive

Archive for December, 2014

NGINX – Load Balancing Web Servers.

December 31, 2014 Leave a comment
Setting up NGINX for HTTP load balancing.
Nginx server is running as below.
nginx-server : nginx.server.com //nginx
Below are the 3 server which run a web application which needs to be load balanced.
Server1 : node1.application.com //web_server
Server2 : node2.application.com //web_server
Server3 : node3.application.com //web_server
Install NGINX on RHEL/Centos6
Step 1 : Create a Repo. in [vim /etc/yum.repos.d/nginx.repo]
CENTOS6
[nginx]
name=nginx repo
gpgcheck=0
enabled=1

RHEL:
[nginx]
name=nginx repo
gpgcheck=0
enabled=1
Step 2 : Installation 
[ahmed@ngnix ~]$ sudo yum install nginx
Step 3 : Configuration: sudo vim /etc/nginx/nginx.conf
#——BEGIN CONFIG————
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  ‘$remote_addr – $remote_user [$time_local] “$request” ‘
                      ‘$status $body_bytes_sent “$http_referer” ‘
                      ‘”$http_user_agent” “$http_x_forwarded_for”‘;

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;
    #include /etc/nginx/conf.d/*.conf;
    
    upstream myappredirect 
    {
        server node1.application.com;
        server node2.application.com;
        server node3.application.com;
    }
    server 
    {
    listen 80;
        location / {
            proxy_pass http://myappredirect;
        }    
    }
}
#——END CONFIG———–
Now we are ready for testing.
[ahmed@ngnix ~]$ sudo service nginx start
Now we can go the browser and hit the IP address of nginx server.
As per the current setup (Default) it will do a Round-Robin to send request to the servers.
“All inbound request to NGINX[nginx.server.com]” -> NGINX will load_balance and distribute traffic to Servers.
–>[NGINX]—->[HOST1]
  \–>[HOST2]
  \–>[HOST3]
Useful Links:
Categories: HOWTOs

HBase on HDFS Single Node

December 30, 2014 Leave a comment
Setting up Local HBase on Top of HDFS.
First lets setup the Hbase Configuration Files.
————–
hbase-site.xml
————–

 
    hbase.rootdir
    hdfs://localhost:9000/hbase
 
 
 hbase.zookeeper.property.clientPort
 2181
 
 
    hbase.cluster.distributed
    true
 
 
      hbase.zookeeper.quorum
      localhost
   
If  you are running zookeeper seperatly then make the line below as ‘false’.
————–
hbase-env.sh
————–
export HBASE_MANAGES_ZK=false
Next creating a zoo.cfg for the zookeeper.
————
zoo.cfg
————

 # The number of milliseconds of each tick
 tickTime=2000
 # The number of ticks that the initial
 # synchronization phase can take
 initLimit=20
 # The number of ticks that can pass between
 # sending a request and getting an acknowledgement
 syncLimit=10
 # the directory where the snapshot is stored.
 dataDir=/opt/mapr/zkdata
 # the port at which the clients will connect
 clientPort=2181
 # max number of client connections
 maxClientCnxns=100
 maxSessionTimeout=300000
Now lets create the HDFS configuration files.
————-
core-site.xml
————-

hdfs://localhost:9000


————-
hdfs.site.xml
————-
dfs.replication
1

dfs.namenode.name.dir
file:///root/hadoop-2.5.1/yarn_data/hdfs/namenode

dfs.datanode.data.dir
file:///root/hadoop-2.5.1/yarn_data/hdfs/datanode
————-
mapred-site.xml
————-
yarn
Add ‘localhost’ in slaves files.
————–
yarn-site.xml
————–

yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
Now That we are ready lets start the services.
ahmed@master:~ ]# ./hdfs namenode -format

This will start DFS (NameNode, SecondaryNameNode, DataNode)
ahmed@master:~ ]# ./start-dfs.sh
This will start Yarn service (Resource Manager and Node Manager)
ahmed@master:~ ]# ./start-yarn.sh
Here is the command output.
ahmed@master:~ ]# jps
43655 Jps
12018 Bootstrap
31585 NameNode
32114 SecondaryNameNode
31798 DataNode
32494 NodeManager
Lets Do a HDFS Test.
ahmed@master:~]# hadoop jar /root/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.1-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 100
Next Lets start HBase services.
First Start the zookeeper.
ahmed@master:~]# ./hbase-daemon.sh –config ../conf/zoo.cfg start zookeeper
Next start RegionServer
ahmed@master:~]# ./hbase-daemon.sh start regionserver
Then we start the master
ahmed@master:~]# ./hbase-daemon.sh start master
ahmed@master:~] # jps
43655 Jps
12018 Bootstrap
40171 HQuorumPeer
40425 HRegionServer
31585 NameNode
32114 SecondaryNameNode
41509 HMaster
31798 DataNode
32494 NodeManager
32277 ResourceManager
Now Lets do a basic test. 
ahmed@master:~] # hbase shell
2014-12-30 20:04:15,116 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter ‘help’ for list of supported commands.
Type “exit” to leave the HBase Shell
Version 0.98.1-hadoop2, r1583035, Sat Mar 29 17:19:25 PDT 2014

hbase(main):001:0> list
TABLE
test
1 row(s) in 2.3090 seconds

=> [“test”]
hbase(main):002:0> scan ‘test’
ROW                                  COLUMN+CELL
 row                                 column=test_fam:, timestamp=1419947982152, value=NewValue
1 row(s) in 0.4890 seconds

hbase(main):003:0> put ‘test’, ‘row2’, ‘test_fam’, ‘SecondValue’
0 row(s) in 0.1110 seconds

hbase(main):004:0> scan ‘test’
ROW                                  COLUMN+CELL
 row                                 column=test_fam:, timestamp=1419947982152, value=NewValue
 row2                                column=test_fam:, timestamp=1419950094363, value=SecondValue
2 row(s) in 0.0260 seconds

hbase(main):005:0>
More information on this can be found below links here.
Categories: HOWTOs

Enabling HBase Authorization – HBase 0.96

December 30, 2014 Leave a comment
Add this below tag to hbase-site.xml in all the Master and Region Server.
 
    hbase.security.authorization
    true
 
 
    hbase.coprocessor.master.classes
    org.apache.hadoop.hbase.security.access.AccessController
 
 
    hbase.coprocessor.region.classes
    org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController
 
 
    hbase.rpc.engine
    org.apache.hadoop.hbase.ipc.SecureRpcEngine
 
[ahmed@master home]# sudo -u hbase hbase shell
hbase(main):002:0> grant

ERROR: wrong number of arguments (0 for 2)

Here is some help for this command:
Grant users specific rights.
Syntax : grant [
[ []]

permissions is either zero or more letters from the set “RWXCA”.
READ('R'), WRITE('W'), EXEC('X'), CREATE('C'), ADMIN('A')

For example:

    hbase> grant 'bobsmith', 'RWXCA'
    hbase> grant 'bobsmith', 'RW', 't1', 'f1', 'col1'

hbase(main):003:0> grant  'ahmed', 'RWCA'
0 row(s) in 3.5870 seconds
 
hbase(main):004:0> exit

More Details in Links Below:
 

Categories: HOWTOs

Mounting DVD/CD on Linux (Centos 6.5)

December 26, 2014 Leave a comment
Check where we have DVD/CD mount points.
[ahmed@zabbixmin ~]$ ll /dev | grep sr0
lrwxrwxrwx 1 root  root           3 Dec 26 16:46 cdrom -> sr0
lrwxrwxrwx 1 root  root           3 Dec 26 16:46 cdrw -> sr0
lrwxrwxrwx 1 root  root           3 Dec 26 16:46 dvd -> sr0
lrwxrwxrwx 1 root  root           3 Dec 26 16:46 dvdrw -> sr0
lrwxrwxrwx 1 root  root           3 Dec 26 16:46 scd0 -> sr0
brw-rw—- 1 root  cdrom    11,   0 Dec 26 16:46 sr0
Mount using auto select mode, from /dev/sr0 to /media/
-r : Mount the file system read-only. A synonym is -o ro.
-t : Type
[ahmed@zabbixmin ~]$ sudo mount -r -t auto /dev/sr0 /media/
[ahmed@zabbixmin ~]$ cd /media/
[ahmed@zabbixmin media]$ ls
manifest.txt     VMwareTools-9.6.1-1378637.tar.gz  vmware-tools-upgrader-64
run_upgrader.sh  vmware-tools-upgrader-32

[ahmed@zabbixmin media]$
Categories: HOWTOs

‘yum’ Download ‘rpms’ without Installing to Specific

December 25, 2014 Leave a comment
Install the package including “downloadonly” plugin:
(RHEL5/CentOS5)
# yum install yum-downloadonly
(RHEL6/CentOS6)
# yum install yum-plugin-downloadonly
Run yum command with “–downloadonly” option as follows:
# yum install –downloadonly –downloaddir=
Confirm the RPM files are available in the specified download directory.
Before using the plugin, check /etc/yum/pluginconf.d/downloadonly.conf to confirm that this plugin is “enabled=1
Categories: HOWTOs

Zabbix Installation 2.4 – CentOS 6.5

December 25, 2014 Leave a comment
Adding Repos and Install Server Configuration.
[ahmed@server ~]# yum install mysql-server mysql
[ahmed@server ~]# yum install zabbix-server-mysql zabbix-web-mysql
[ahmed@server ~]# yum install zabbix-agent
Create Database.
[ahmed@server ~]# mysql -uroot
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> grant all privileges on zabbix.* to zabbix@localhost identified by 'zabbix';
mysql> exit
Creating Schema.
[ahmed@server ~]# cd /usr/share/doc/zabbix-server-mysql-2.4.0/create
[ahmed@server ~]# mysql -uroot zabbix < schema.sql
[ahmed@server ~]# mysql -uroot zabbix < images.sql
[ahmed@server ~]# mysql -uroot zabbix < data.sql
Setting Configuration.
[ahmed@server ~]# vi /etc/zabbix/zabbix_server.conf
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=zabbix
Start Server.
[ahmed@server ~]# service zabbix-server start
Apache Configuration. Change timezone.
[ahmed@server ~]# sudo vim /etc/httpd/conf.d/zabbix.conf
php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
php_value date.timezone Asia/Kolkata
Start httpd.
[ahmed@server ~]# service httpd restart
Categories: HOWTOs

Adding services to auto start after reboot.

December 25, 2014 Leave a comment
1. List All Services, Using ‘–list‘ parameter will displayed all services and their current status in each run-level configuration.
[ahmed@server ~]# chkconfig –list

myservermgesrv  0:off   1:off   2:on    3:on    4:on    5:on    6:off
abc-servc       0:off   1:off   2:off   3:on    4:off   5:on    6:off
xyz:ahmed       0:off   1:off   2:off   3:on    4:off   5:on    6:off
2. Adding mysqld, zabbix-server, zabbix-agent, httpd to chkconfig
[ahmed@server ~]# chkconfig –level 235 mysqld on
[ahmed@server ~]# chkconfig –level 235 zabbix-server on
[ahmed@server ~]# chkconfig –level 235 zabbix-agent on
[ahmed@server ~]# chkconfig –level 235 httpd on
3. To Disable any run-level use the off parameter.
[ahmed@server ~]# chkconfig –level 2 mysqld off
Categories: HOWTOs